Skip to main content

OpenBao

OpenBao (the open-source Vault fork) on node1, at http://100.66.50.60:8200. It's reachable over NetBird only.

Argo appapps/base/openbao.yaml: the official chart openbao/openbao 0.30.7 (OpenBao v2.7.2), plus infra/openbao/
Namespaceopenbao
Storageraft, one node, static local PV /srv/k8s-volumes/openbao (5Gi, Retain)
Sealstatic key auto-unseal. The key is in SealedSecret openbao/openbao-unseal
In-clusterhttp://openbao-active.openbao.svc:8200
Over NetBirdhttp://100.66.50.60:8200 (UI at /ui): Traefik entryPoint openbao, hostPort 8200, infra/openbao/ingressroute.yaml. No DNS needed
Optional hostnamehttps://bao.tiktuzki.com: Traefik websecure, cert-manager letsencrypt-dns. Needs DNS or an /etc/hosts line → 100.66.50.60
Injector / CSIoff. Turn them on when the first app reads its secrets from OpenBao

Why a static seal​

node1 restarts every pod several times a week. Each time its Wi-Fi reconnects, microk8s re-issues the API server certificate and kills containerd. A Shamir-sealed OpenBao would come back sealed every time and serve nothing until someone typed in key shares.

The static seal reads a 32-byte AES key from a mounted Secret and unseals by itself. The cost: anyone who can read Secrets in openbao, or who holds the sealed-secrets master key, can unseal. The cluster is the root of trust. To move to a stronger seal (a cloud KMS or Transit), do a seal migration. Editing the config isn't enough.

Deploy​

0. Before you start​

  • letsencrypt-dns ClusterIssuer is READY True. It's shared with Argo CD; see Argo CD behind the VPN, steps 1–2.
  • Your NetBird peer is allowed TCP 443 to node1.

1. Volume directory (on node1)​

sudo ./infra/storage/create-volume-dirs.sh # creates /srv/k8s-volumes/openbao as 100:1000, 0700

2. Unseal key: generate, seal, keep a copy​

The key never prints, and it never stays on disk:

umask 077; K=$(mktemp)
openssl rand -out "$K" 32
kubectl create secret generic openbao-unseal -n openbao --from-file=unseal.key="$K" \
--dry-run=client -o yaml \
| kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller \
--format yaml > infra/openbao/openbao-unseal-sealedsecret.yaml
base64 < "$K" | pbcopy # paste into Proton Pass as pass://athena/node1-openbao-unseal/secret
rm -P "$K"; pbcopy < /dev/null

⚠️ The Proton Pass copy is not optional. Without this key, the raft data and every snapshot are unreadable. infra/backup/backup.sh also captures it, because it dumps every Secret, but don't rely on that backup alone.

3. Commit and apply​

Commit infra/openbao/openbao-unseal-sealedsecret.yaml, then:

kubectl apply -f apps/base/openbao.yaml
kubectl -n openbao get pod openbao-0 -w # Running, 0/1 ready: it isn't initialised yet

4. Initialise (once)​

With an auto seal, init returns recovery keys and a root token, not unseal keys. Recovery keys authorise operations like generate-root. They can't decrypt anything. Send the output straight to a file in Proton Pass and never into a terminal you're recording or a chat:

kubectl -n openbao exec -ti openbao-0 -- bao operator init -recovery-shares=3 -recovery-threshold=2
# → Proton Pass item openbao-init: root token + 3 recovery keys
kubectl -n openbao get pod openbao-0 # 1/1 Ready: initialised and unsealed
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Seal Type|Sealed|Initialized'
# Seal Type static · Initialized true · Sealed false

Then delete the pod once and check it comes back unsealed by itself. That's the whole point of the static seal:

kubectl -n openbao delete pod openbao-0 && kubectl -n openbao wait pod openbao-0 --for=condition=Ready --timeout=120s

5. Access​

With NetBird up, and no DNS needed:

curl -s http://100.66.50.60:8200/v1/sys/health | jq '{initialized, sealed}'
export BAO_ADDR=http://100.66.50.60:8200
bao login # root token, only to set up a real auth method; then `bao token revoke -self`

Plain HTTP is safe here only because of the tunnel. Traffic to 100.66.50.60 travels inside NetBird's WireGuard tunnel, and infra/host-firewall drops port 8200 arriving on node1's LAN interfaces. Re-run harden-firewall.sh after changing its port list. Without that, anyone on the home Wi-Fi could reach OpenBao in clear text on 192.168.1.5:8200.

For HTTPS, use the hostname instead. Either add a Cloudflare record A bao 100.66.50.60 (DNS only), or add 100.66.50.60 bao.tiktuzki.com to /etc/hosts on each machine.

Don't use the root token day to day. Set up an auth method (OIDC with Google, or kubernetes for in-cluster apps), then revoke the root token. bao operator generate-root with the recovery keys mints a new one when you need it.

Google login​

People sign in with Google through the oidc auth method. They get the admin policy if their email is on the allow-list in infra/openbao/configure-google-login.sh, the same two addresses as Argo CD and Grafana. Tokens last 1h and renew up to 12h.

1. Google OAuth client. OpenBao shares the tik-k8s client with Grafana (Google Cloud console → APIs & Services → Credentials → tik-k8s). Add these Authorized redirect URIs next to Grafana's:

https://bao.tiktuzki.com/ui/vault/auth/oidc/oidc/callback
http://localhost:8250/oidc/callback

Its client ID and secret are in pass://athena/tik-k8s-google-auth/{client_id,secret}, which google-login.env.tpl references. If the consent screen is in Testing mode, both emails must be on its test-user list.

2. Configure OpenBao (re-runnable). Check that the paths in google-login.env.tpl match your Pass items first:

brew install openbao
pass-cli run --env-file infra/openbao/google-login.env.tpl -- infra/openbao/configure-google-login.sh

3. Sign in.

  • CLI: BAO_ADDR=http://100.66.50.60:8200 bao login -method=oidc opens Google in the browser and returns to localhost:8250.
  • UI: https://bao.tiktuzki.com/ui → method OIDC → Sign in with OIDC Provider. The name must resolve to 100.66.50.60 (DNS or /etc/hosts).

⚠️ The UI at http://100.66.50.60:8200/ui can't do Google login. Google rejects plain-HTTP and bare-IP redirect URIs (localhost is the only exception), and the UI builds its redirect from the address in the browser bar.

4. Revoke the root token once a Google login works:

pass-cli run --env-file <(echo 'BAO_TOKEN=pass://athena/openbao-init/root_token') -- \
env BAO_ADDR=http://100.66.50.60:8200 bao token revoke -self

Then mark it revoked in Pass. When you need root again, use bao operator generate-root -init with 2 of the 3 recovery keys.

Backup​

The nightly cluster-backup run (infra/backup/backup.sh, section 1b) takes a raft snapshot to secrets/openbao-raft.snap. That's the critical half, uploaded to S3 every night with the openbao-unseal Secret it needs. A tarball of the raft directory could be torn, so it isn't used.

No stored credential. backup.sh mints a 10-minute JWT for ServiceAccount openbao/openbao-backup (audience openbao) and logs in at auth/kubernetes/role/backup. That gives a 5-minute token with policy snapshot, which can read sys/storage/raft/snapshot and nothing else. The JWT travels on stdin, and the token is revoked after use. Set up by infra/openbao/configure-backup-auth.sh (re-runnable, needs an admin login).

Failures are loud. After 3 failed attempts the run is marked INCOMPLETE, and ClusterBackupIncomplete fires. cluster_backup_openbao_snapshot_bytes is the snapshot size (0 = missing).

Restore: infra/backup/RESTORE.md, step 5b. For a manual snapshot at any time:

BAO_ADDR=http://100.66.50.60:8200 bao operator raft snapshot save openbao-$(date +%F).snap

Rotating the unseal key​

  1. Generate a new key as in step 2, but seal it under a second key name with kubeseal --merge-into: unseal-<date>.key.
  2. In apps/base/openbao.yaml, set current_key / current_key_id to the new key, and previous_key / previous_key_id to the old one. Sync. OpenBao re-wraps its root key.
  3. Once it has restarted cleanly, remove previous_* and the old key name.