Disaster recovery plan for node1

What has to survive, how much loss and downtime is acceptable, and how each piece gets back. The mechanics of the nightly job are in Backing up the cluster; the rebuild steps are in infra/backup/RESTORE.md.

History: what the 2026-10-31 audit found

The nightly job had been running and succeeding for weeks, yet nothing had ever left node1 (the upload half of the backup flow was never installed), postgresql-ha had never been dumped (backup.sh listed only clusters that were not deployed, and skipped them silently), and the volume tarballs were of empty, retired directories. Fixed the same day — steps 1–4 below; the first off-site run is node1/*/20261031-131322.

1. Inventory and tiers

System	Holds	Tier	Backup (target)
postgresql-ha (Patroni/Spilo, 3 replicas, one SSD)	app, hrm, metabase, postgres — ~95 MB	1	WAL-G continuous archive + daily base backup → S3; nightly pg_dumpall as a second path
Sealed-Secrets master key	decrypts every committed SealedSecret	1	nightly, critical half → S3 STANDARD_IA
microk8s CA (ca.key)	signs every kubeconfig, incl. the only ArgoCD break-glass	1	nightly, critical half
Kubernetes Secrets, dqlite datastore	out-of-band secrets (argocd-oidc-secret, …), cluster objects	1	nightly, critical half
age identity, AWS keys	open the backups / write them	1	Proton Pass, plus one offline copy — never on node1
PV data: 9router, x-hrm, monitoring/grafana	app files, Grafana users and UI-made dashboards	2	nightly, bulk half → S3
Netplan, NetBird peer state	Wi-Fi PSK, overlay identity (a new IP breaks every NPM proxy host)	2	nightly, critical half
Git repos (tiktuzki-gitops, charts, values)	desired state of everything	2	GitHub + local clones
Prometheus TSDB, ARC runner cache, Ollama models	re-derivable	3	none, by decision

2. Targets

Tier	RPO	RTO	Why it is achievable
1 — Postgres	5 min	1 h — measured 43 s to writable (drill 2026-10-31, ~100 MB)	archive_timeout = 300s ships WAL at least every 5 min; restore = new cluster cloned from the S3 base backup + WAL
1 — cluster secrets and keys	24 h	4 h	critical half is instant-retrieval; rebuild per RESTORE.md
2 — PV data	24 h	4 h	bulk half, uploaded daily (it is megabytes)
3	—	best effort	rebuilt from Git and upstream

These are targets until the first drill measures them (section 6).

3. Disasters planned for

Scenario	Likelihood here	Covered by
SSD fails — node1 has one disk for OS, dqlite, all three Postgres copies and the backups	highest	everything in S3; the 3 Postgres replicas do not help
Bad migration / DROP / app bug corrupts data	medium	WAL-G point-in-time restore to just before it (14+ days)
node1 compromised, attacker wipes data and tries the backups	low	node1's keys cannot delete locked objects (Object Lock, governance, 30 days) and cannot read archives (age public key only)
node1 lost (theft, fire, PSU)	low	S3 + Git + Proton Pass are all off-site
age identity lost	low, catastrophic	Proton Pass and one offline copy
AWS account compromised	low	root MFA; governance retention needs an explicit bypass permission no user has; occasional pull-backup.sh copy on the laptop

4. Architecture

node1                                     AWS ap-southeast-1
─────
02:30 cluster-backup.timer                s3://tiktuzki-cluster-backup
  backup.sh   → /srv/k8s-volumes/backups    Object Lock governance 30 d, versioning, SSE-S3,
                (staging, keep 7)           public access blocked, TLS only
  cluster-backup upload  ── age ──────►     node1/critical/…tar.zst.age   STANDARD_IA, expire 400 d
                                            node1/bulk/…tar.zst.age       STANDARD_IA, expire 90 d
Spilo (every postgresql-ha pod)           s3://tiktuzki-pg-wal
                                            versioning, deleted versions kept 30 d, SSE-S3, TLS only
  WAL-G: each WAL segment ── libsodium ►    spilo/postgresql-ha/wal/…
         daily base backup 01:00      ►     spilo/postgresql-ha/basebackups/…

Two buckets, because WAL-G cannot write to a locked one. A bucket with default Object Lock retention rejects any upload without Content-MD5 or a CRC checksum; cluster-backup (aws-sdk-s3 1.119) sends CRC32, Spilo's WAL-G (aws-sdk-go v1) sends neither. The WAL bucket gets the same guarantee another way: versioning, and a WAL-G user with DeleteObject but no DeleteObjectVersion or lifecycle/versioning permissions — a compromised node1 can hide objects behind delete markers, never destroy a version younger than 30 days.

Two IAM users, one per writer:

User	Allowed on	Actions	Cannot
node1-cluster-backup	tiktuzki-cluster-backup/node1/*	PutObject, AbortMultipartUpload, GetObject (upload verification)	list, delete, write anywhere else
node1-walg	tiktuzki-pg-wal	PutObject, GetObject, ListBucket, DeleteObject	purge versions, change versioning or lifecycle, touch the archive bucket

Both are created by infra/backup/aws-setup.sh, which writes their access keys straight into Proton Pass (pass://athena/node1-backup-s3, pass://athena/node1-walg-s3) and then proves every allow and deny with the new keys.

Encryption is client-side for both paths, so a leaked bucket is ciphertext: age for the nightly archives (node1 holds only the public key), libsodium (WALG_LIBSODIUM_KEY, a sealed secret) for WAL-G.

Cost: under 1 GB in total; well under US$1/month.

Changes from the original backup-flow.md design: the bulk half goes to STANDARD_IA daily instead of DEEP_ARCHIVE weekly. It is megabytes, so Deep Archive's 180-day minimum and 12–48 h thaw buy nothing and cost a day of RPO.

5. Implementation

#	Step	Who	Restarts anything?
1 ✅	infra/backup/aws-setup.sh --yes: both buckets, lifecycle rules, two IAM users, keys into Proton Pass, policy checks	you (AWS admin credentials)	no
2 ✅	infra/backup/age-setup.sh: keypair straight into pass://athena/node1-backup-age, round-trip tested; + one offline copy	you	no
3 ✅	Fix backup.sh: dump postgresql-ha by cluster-name=postgresql-ha,role=master, verify the completion marker, retry ×3 (API stalls cut kubectl exec); volumes 9router, x-hrm, monitoring/grafana; flag missing required DBs in INCOMPLETE.txt	Claude	no
4 ✅	infra/backup/install-node1.sh (on the laptop): backup.sh, cluster-backup v0.1.31, s3.env streamed from Proton Pass, upload drop-in; first run + decrypt check	you (sudo on node1 once)	no
5 ✅	WAL-G (charts/postgresql-ha walg:): sealed postgresql-ha-walg from Proton Pass (node1-walg-s3, node1-walg-libsodium); archive_timeout 300 s; base backup 01:00 UTC, keep 14. Rolled 2026-10-31 13:22 UTC, one switchover	Claude	rolled once (~2 min)
6 ✅	infra/monitoring/alerts-backup.yaml: upload stale/missing, run incomplete, collect stale, WAL-G base backup stale, archiving failing, pg_wal piling up — from a node-exporter textfile written by backup.sh + mark-upload.sh, and live pg_wal_size_bytes	Claude	node-exporter only
7 ✅	infra/backup/drill-pitr.sh --run: markers before/after a target time on production, clone pgha-drill from S3 to that time, verify markers + per-table row counts, tear down; prints the drill-log row	Claude runs, you watch	no (one marker table, one WAL switch)

6. Drills

When	Drill	Pass when
After step 7, then quarterly	Postgres PITR: clone postgresql-ha from S3 into a throwaway namespace to a chosen timestamp	row counts match the source at that time; time to ready ≤ 1 h
After step 4, then quarterly	Critical half: download the latest, decrypt with the Proton Pass identity on the laptop, read the sealed-secrets key	decrypts; key matches the live one
Yearly	Full rebuild tabletop of RESTORE.md against a fresh VM	every step runs as written; gaps fixed in the doc

Log each run here:

Date	Drill	Measured RPO / RTO	Broke	Follow-up
2026-10-31	Postgres PITR: pgha-drill cloned from base_000003FA00000008000000BB + archived WAL to T = 14:09:19 UTC	RTO 43 s to writable; RPO = archive_timeout 300 s; marker 2 s before T restored, marker 2 s after T absent; same databases; identical row counts in all 179 tables	first 3 runs broke on the drill, the 4th on the restore path itself: post_init runs after a clone and fails on CREATE ROLE app — a real production restore would have looped. Also: no data volume without persistence; readiness probe too early	all fixed in the chart / script; next drill in a quarter
2026-10-31	Critical half: first upload node1/critical/20261031-131322 downloaded and decrypted on the laptop with the Proton Pass identity; listing shows MANIFEST, master key, secrets, node state	— (not a timed restore)	installer's listing filter printed secret file names (fixed)	Postgres PITR drill after step 5

7. Restore paths, in brief

• Postgres to a point in time — the chart's clone: block (charts/postgresql-ha/values.yaml): on an empty cluster, Spilo fetches the newest WAL-G base backup at or before targetTime and replays archived WAL to it. Two ways to use it:
  • Side by side (bad migration, need yesterday's rows): exactly what infra/backup/drill-pitr.sh --run --keep does — release pgha-drill next to production, then pg_dump what you need out of it and tear it down.
  • Replace production (cluster lost): wipe (chart README, "Wiping and re-bootstrapping"), set clone.enabled: true, clone.s3Prefix: s3://tiktuzki-pg-wal/spilo/postgresql-ha/wal/18, clone.region, clone.targetTime in values-dev.yaml, sync; once it is up, set clone.enabled: false in the next commit.
• Postgres, simplest — pg_dumpall file from last night's bulk half → psql into any Postgres 18.
• Whole node — RESTORE.md: critical half first (sealed-secrets key, CA, secrets), Argo re-syncs from Git, then Postgres from WAL-G, then PV tarballs.