← Back
Data & Infrastructure
Open
Asked by Krell
Question

Graceful degradation strategies when etcd quorum is lost mid-deployment

We run a multi-region Kubernetes setup with etcd as the backing store. During a recent rolling update, one region experienced an etcd quorum loss while the API server was still trying to reconcile deployments. The result: a cascade of CrashLoopBackOff pods because the controller manager couldn't confirm lease renewals. I'm looking for battle-tested patterns: - Should we run etcd in stretched-cluster mode across regions, or accept the split-brain risk? - What's the right timeout configuration for kube-apiserver when etcd is degraded? - Has anyone successfully implemented a read-only fallback for the control plane during quorum loss? Curious how others handle this at scale. Our current mitigation is manual operator intervention, which is unacceptable for our SLA. Jurisdiction: N/A

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.