Last updated on October 27, 2026
Updated October 2026. I first wrote this after losing two of three control plane nodes on a real cluster running Kubernetes 1.32. For this update I rebuilt the whole scenario in a lab with kubeadm 1.36.5 and etcd 3.6.8: three control planes, two switched off, back to one working node, back to three. The recovery is shorter than the original: no etcd download, no hand-started etcd process, and no editing --initial-cluster on every node afterwards. That last part was not needed at all, kubeadm join takes care of it.
What breaks when two of three control planes die
etcd needs a majority to accept any write: two of three members. With one member left it still runs, but it cannot elect a leader, so every request to the API server times out. Pods on the workers keep running and serving traffic, but nothing can change: no deploys, no rescheduling, no kubectl. The cluster data is still intact on the surviving node. The job is to tell that one etcd member to stop waiting for the dead ones.
Everything below runs on the surviving control plane, cp1 (192.168.1.11). cp2 and cp3 are gone.
1. Confirm it is quorum
kubectl get nodesThis just hangs and times out. Next stop is etcd itself. kubeadm runs it as a static pod, and the etcd image already contains etcdctl, so there is nothing to download. You do need crictl. The kubeadm 1.36 packages no longer depend on cri-tools, so on a fresh node it may simply be missing:
sudo apt install cri-tools
sudo crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock
E=$(sudo crictl ps --name '^etcd$' -q)
sudo crictl exec $E etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint healthhttps://127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy clustermember list without quorum times out as well, because by default it asks the leader. Ask the local member instead:
sudo crictl exec $E etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
member list --consistency=s -w table+------------------+---------+------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------+---------------------------+---------------------------+------------+
| 5decca2a1a730c26 | started | cp2 | https://192.168.1.12:2380 | https://192.168.1.12:2379 | false |
| ba0103d7d2416dcb | started | cp1 | https://192.168.1.11:2380 | https://192.168.1.11:2379 | false |
| d43b616bab1176c3 | started | cp3 | https://192.168.1.13:2380 | https://192.168.1.13:2379 | false |
+------------------+---------+------+---------------------------+---------------------------+------------+Three members, one reachable. That is the situation this post fixes. If all three were still reachable you would have a different problem, and none of the steps below apply.
2. Stop etcd and back up its data
Moving the static pod manifest out of /etc/kubernetes/manifests makes kubelet stop etcd. Copy the data directory while nothing writes to it:
sudo mkdir -p /root/etcd-recovery
sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd-recovery/etcd.yaml
sudo crictl ps --name '^etcd$' # wait until this is empty
sudo cp -a /var/lib/etcd /root/etcd-recovery/etcd-data-$(date +%F-%H%M)If anything below goes wrong, this copy plus the original manifest gets you back to where you started.
3. Start etcd as a one-member cluster
--force-new-cluster makes etcd drop every other member from its configuration and continue alone with the data it has. Add it to a copy of the manifest right after --data-dir:
sudo cp /root/etcd-recovery/etcd.yaml /root/etcd-recovery/etcd.yaml.orig
sudo sed -i 's| - --data-dir=/var/lib/etcd|&\n - --force-new-cluster|' /root/etcd-recovery/etcd.yaml
sudo diff /root/etcd-recovery/etcd.yaml.orig /root/etcd-recovery/etcd.yaml18a19
> - --force-new-clustersudo cp /root/etcd-recovery/etcd.yaml /etc/kubernetes/manifests/etcd.yaml
E=$(sudo crictl ps --name '^etcd$' -q) # repeat until it returns an ID
sudo crictl exec $E etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
member list -w table+------------------+---------+------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------+---------------------------+---------------------------+------------+
| ba0103d7d2416dcb | started | cp1 | https://192.168.1.11:2380 | https://192.168.1.11:2379 | false |
+------------------+---------+------+---------------------------+---------------------------+------------+One member, same ID as before, and endpoint health now reports is healthy: successfully committed proposal.
4. Remove the flag again, right away
sudo cp /root/etcd-recovery/etcd.yaml.orig /etc/kubernetes/manifests/etcd.yamlkubelet restarts etcd without the flag, and the member list stays at one member. Do not skip this. --force-new-cluster is applied on every start, not once. I left it in on purpose after rebuilding HA, and the next etcd restart on cp1 threw both new members out again.
5. Check the API and the data, remove the dead nodes
kubectl get --raw=/readyz
kubectl get nodes
kubectl delete node cp2 cp3The API answers again, and everything created before the outage is still there. In the lab I made a ConfigMap before switching cp2 and cp3 off, and it was there afterwards with its data. Deleting the two node objects also cleans up their etcd and API server mirror pods after a short while.
6. Rebuild high availability
A node that was “lost” has a habit of coming back once someone fixes the power or the hypervisor. When cp2 booted again, its etcd went into a restart loop with:
"msg":"server error","error":"the member has been permanently removed from the cluster"
"msg":"data-dir used by this member must be removed"Harmless for the cluster, but the node is useless until it is reset. Do that on every old control plane you want to reuse. It also wipes the stale etcd data:
sudo kubeadm reset -fThen, on cp1, upload the control plane certificates again (the key from the original kubeadm init expired after two hours) and create a join command:
sudo kubeadm init phase upload-certs --upload-certs
sudo kubeadm token create --print-join-commandOn cp2 and cp3, run the printed command with the control plane flags added:
sudo kubeadm join k8s-api:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane --certificate-key <key from upload-certs>+------------------+---------+------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------+---------------------------+---------------------------+------------+
| 88304ab288813d39 | started | cp3 | https://192.168.1.13:2380 | https://192.168.1.13:2379 | false |
| 91bbcd2d1a44e8c2 | started | cp2 | https://192.168.1.12:2380 | https://192.168.1.12:2379 | false |
| ba0103d7d2416dcb | started | cp1 | https://192.168.1.11:2380 | https://192.168.1.11:2379 | false |
+------------------+---------+------+---------------------------+---------------------------+------------+Three members again, and no etcd manifest was edited by hand. The original version of this post had a long section about updating --initial-cluster on every node. That flag is only read when a member starts with an empty data directory, so changing it on a running member does nothing, and kubeadm join already adds the new members properly.
Gotchas
- A kube-vip address can disappear together with quorum. With
vip_leaderelectionenabled, kube-vip picks the VIP owner through a Lease in the API, and without the API nobody owns it. On a real cluster I had a working API server on the surviving node and a control plane VIP that answered nowhere. Talk to the node’s own address (kubectl --server https://192.168.1.11:6443, which kubeadm certificates cover) or keep the control plane VIP on keepalived, which does not depend on the cluster. - No crictl on kubeadm 1.36.
apt install cri-toolsand point it at the containerd socket, or usectr -n k8s.io. member listhangs without quorum. Add--consistency=s.--force-new-clusteris not one-shot. Remove it from the manifest as soon as the single member is up.- Reset returning nodes. A node with old etcd data cannot rejoin and only produces restart loops.
kubeadm reset -ffirst, then join. - This only works because cp1 still had its data. If the last member’s disk is gone too, you need an etcd snapshot (
etcdctl snapshot save) and the restore procedure from the etcd docs. Take those snapshots on a schedule, not after the outage.
Takeaways
- Losing two of three control planes stops all changes but not running workloads. The data survives on the last member.
- Back up
/var/lib/etcd, start etcd with--force-new-clusterthrough its static pod manifest, then remove the flag. - Delete the dead node objects,
kubeadm resetanything that comes back, and rejoin with a fresh certificate key. No manual etcd membership edits needed. - Keep the control plane VIP independent of the cluster, and keep etcd snapshots somewhere else.