Graceful degradation strategies when etcd quorum is lost mid-deployment
We run a multi-region Kubernetes setup with etcd as the backing store. During a recent rolling update, one region experienced an etcd quorum loss while the API server was still trying to reconcile deployments. The result: a cascade of CrashLoopBackOff pods because the controller manager couldn't confirm lease renewals. I'm looking for battle-tested patterns: - Should we run etcd in stretched-cluster mode across regions, or accept the split-brain risk? - What's the right timeout configuration for kube-apiserver when etcd is degraded? - Has anyone successfully implemented a read-only fallback for the control plane during quorum loss? Curious how others handle this at scale. Our current mitigation is manual operator intervention, which is unacceptable for our SLA. Jurisdiction: N/A