The Backup That Restored an Empty File

The nightly job backed up three Kubernetes clusters. It ran on schedule, exited zero, and wrote its snapshot into restic without complaint. The monitoring signal for backup freshness was green. It had been green for weeks.

Then I actually restored one, and the single file that makes a cluster rebuild possible came back as zero bytes.

What the backup was supposed to contain

Rebuilding an RKE2 cluster from a snapshot needs three things: the etcd snapshot itself, the admin kubeconfig, and the node token. The node token is the piece that matters most. Without it, existing nodes cannot rejoin the restored control plane, so you are not restoring a cluster, you are building a new one and migrating workloads by hand.

tar cf /tmp/rebuild.tar \
  /var/lib/rancher/rke2/server/node-token \
  /etc/rancher/rke2/rke2.yaml

Why it was empty

On an RKE2 server, node-token is a symbolic link to a file named token in the same directory. Plain tar archives the link, not the file it points at. Restoring into a fresh directory produced a dangling symlink, which reads as zero bytes.

This is the second time I have seen this exact shape. An earlier set of Talos cluster secrets was backed up as zero-byte artifacts the same way. Recognising the pattern is the only reason I checked the file size at all.

The verifier was also lying

There was already a restore-verification script. It restored the archive into a scratch directory and printed a report. On the run that exposed the bug, its output contained both of these lines:

node-token restored: 0 bytes TOO-SMALL
RESTORE VERIFIED

It measured the problem correctly and then reported success anyway, because the size check printed a warning instead of setting a non-zero exit. A checker that does not fail on its own failure is worse than no checker, because it converts an unknown into a false assurance.

The fix

Two changes. Archive with dereference so the link is followed, and assert a minimum real size so an empty result fails the job rather than warning about it.

# -h dereferences symlinks instead of archiving the link
tar -chf /tmp/rebuild.tar \
  /var/lib/rancher/rke2/server/node-token \
  /var/lib/rancher/rke2/server/token \
  /etc/rancher/rke2/rke2.yaml

# and the verifier must fail, not warn
size=$(stat -c%s "$RESTORED/node-token")
if [ "$size" -lt 50 ]; then
  echo "FAIL: node-token is $size bytes"
  exit 1
fi

All three clusters now restore a node-token of 109 real bytes, verified by restoring it rather than by trusting the job's exit code. The same bug was live in the dev cluster's backup script since the day it was written, and was fixed there too.

What I took from it

A backup you have never restored is not a backup, it is a belief. The useful version of that rule is narrower than it sounds: it is not enough to restore and eyeball the output, because I had a script doing exactly that and it told me everything was fine. The assertion has to be mechanical, and failing the assertion has to fail the job.