A Failover Test That Passed and Proved Nothing
Stopping the API server on the node holding the virtual IP produced a clean pass in zero seconds. The address never moved, which is exactly why the result was worthless.
The production Kubernetes control plane sits behind a virtual IP managed by kube-vip in ARP mode. One node holds the address; if it dies, another should claim it within seconds. That is the entire point of the component, so before putting anything real on the cluster I tested it.
The test passed. The result was worthless.
The first attempt
The obvious way to test a control-plane failover is to stop the control plane. I stopped the RKE2 server process on the node currently holding the VIP, then polled the API through the virtual address until it answered again.
holder BEFORE: 10.110.0.41
API through VIP: ok
recovered in ~0s
holder AFTER : 10.110.0.41Zero seconds of downtime looks like an excellent result until you read the last line. The address never moved. The same node held it before and after, so nothing failed over and the test measured nothing.
kube-vip runs as a DaemonSet pod with its own leadership lease. Stopping the API server did not stop that pod, so it kept renewing its claim on the address. I had tested that the API restarts quickly, not that the VIP moves.
This is the same trap as deleting a pod that a DaemonSet recreates in two seconds and concluding the workload is resilient. The system repaired the thing I broke, not the thing I was asking about.
The second attempt
To force a real leadership transfer I had to remove the component holding the lease, on the specific node holding it. Not a node reboot, not an API restart: delete the kube-vip pod on the current holder and watch where the address lands.
holder BEFORE: 10.110.0.41
VIP MOVED to : 10.110.0.43 after ~3s
API through VIP: ok
final holders: 10.110.0.43 (count=1)Three seconds, the API stayed reachable through the virtual address, and exactly one node claimed it at the end. That last number is the one I care about most. A count of zero is an outage; a count of two or more is split brain, where two machines answer for the same address and clients get whichever one ARP happens to favour.
The holder count is now a monitored signal for the same reason. It is the failure mode that does not announce itself: everything appears to work until two nodes disagree about state.
What I took from it
An inconclusive test is worth less than no test, because no test leaves you appropriately uncertain while an inconclusive one leaves you confident and wrong. Before running a resilience test now, I write down which specific component must lose its role, and what observable value has to change if the test actually exercised anything. If I cannot name that value in advance, I am not testing, I am hoping.
Found this helpful?
I write about infrastructure, backend development, and DevOps. Follow along as I continue building.