Kubernetes backups and disaster recovery: what actually restores
Everyone has backups. Far fewer people have restores. A backup job that runs every night and reports success tells you almost nothing, because the only question that matters is whether the backup can bring the system back. I learned this by testing it: on my own two-node Kubernetes cluster I run Velero backups every day, and I proved they work by deleting a namespace and restoring it. The pods came back Running. Until that drill, the backups were a hope, not a plan.
The first thing to understand is what a Kubernetes backup actually has to capture. A cluster is not a single database. It is etcd state, application manifests, secrets, persistent volume data, and the custom resources that operators create. Backing up etcd alone gives you the state but not the data on disk. Snapshotting volumes alone gives you the data but not the objects that point at it. Velero covers the middle ground: it backs up the Kubernetes objects and, with the right configuration, the persistent volumes behind them. My backups go to Cloudflare R2, off the cluster and off the cloud account that runs it, with a 30-day retention. A backup that lives on the same infrastructure it protects is not a backup.
The second thing is the restore drill, and it has to be a real one. A drill is not reading the Velero logs. It is picking a namespace, deleting it, restoring it from the backup, and checking that the application actually works — not just that the pods exist, but that they serve traffic and read their data. I run this kind of check because the gaps only show up during a restore. The common ones are secrets that were never in the backup because they were created by hand, custom resources that the backup skipped, and volume snapshots that were never taken because the storage class was not configured for them. Every one of those is a five-minute fix before an incident and a multi-hour problem during one.
Retention and schedule deserve the same honesty. Daily backups with a 30-day window mean I can lose a day of work at most, and I can go back a month. That is a deliberate trade: more frequent backups cost more storage and more time, and the right answer depends on how much data you can afford to lose. Whatever you choose, write it down. A recovery plan that only exists in someone’s head is not a plan.
The last piece is the runbook. When the cluster is down, nobody should be improvising. The runbook says where the backups live, which command restores them, in what order the namespaces come back, and how you verify each one. It is short, it is tested, and it is boring. Boring is the goal. Disaster recovery is not a heroic act. It is a checklist you have already run, on a calm afternoon, when nothing was on fire.