OpenBao
OpenBao (the open-source Vault fork) on node1, at http://100.66.50.60:8200. It's reachable
over NetBird only.
| Argo app | apps/base/openbao.yaml: the official chart openbao/openbao 0.30.7 (OpenBao v2.7.2), plus infra/openbao/ |
| Namespace | openbao |
| Storage | raft, one node, static local PV /srv/k8s-volumes/openbao (5Gi, Retain) |
| Seal | static key auto-unseal. The key is in SealedSecret openbao/openbao-unseal |
| In-cluster | http://openbao-active.openbao.svc:8200 |
| Over NetBird | http://100.66.50.60:8200 (UI at /ui): Traefik entryPoint openbao, hostPort 8200, infra/openbao/ingressroute.yaml. No DNS needed |
| Optional hostname | https://bao.tiktuzki.com: Traefik websecure, cert-manager letsencrypt-dns. Needs DNS or an /etc/hosts line → 100.66.50.60 |
| Injector / CSI | off. Turn them on when the first app reads its secrets from OpenBao |
Why a static seal
node1 restarts every pod several times a week. Each time its Wi-Fi reconnects, microk8s re-issues the API server certificate and kills containerd. A Shamir-sealed OpenBao would come back sealed every time and serve nothing until someone typed in key shares.
The static seal reads a 32-byte AES key from a mounted Secret and unseals by itself. The
cost: anyone who can read Secrets in openbao, or who holds the sealed-secrets master key,
can unseal. The cluster is the root of trust. To move to a stronger seal (a cloud KMS or
Transit), do a seal migration. Editing the config isn't enough.
Deploy
0. Before you start
letsencrypt-dnsClusterIssuer isREADY True. It's shared with Argo CD; see Argo CD behind the VPN, steps 1–2.- Your NetBird peer is allowed TCP 443 to node1.
1. Volume directory (on node1)
sudo ./infra/storage/create-volume-dirs.sh # creates /srv/k8s-volumes/openbao as 100:1000, 0700
2. Unseal key: generate, seal, keep a copy
The key never prints, and it never stays on disk:
umask 077; K=$(mktemp)
openssl rand -out "$K" 32
kubectl create secret generic openbao-unseal -n openbao --from-file=unseal.key="$K" \
--dry-run=client -o yaml \
| kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller \
--format yaml > infra/openbao/openbao-unseal-sealedsecret.yaml
base64 < "$K" | pbcopy # paste into Proton Pass as pass://athena/node1-openbao-unseal/secret
rm -P "$K"; pbcopy < /dev/null
⚠️ The Proton Pass copy is not optional. Without this key, the raft data and every
snapshot are unreadable. infra/backup/backup.sh also captures it, because it dumps every
Secret, but don't rely on that backup alone.
3. Commit and apply
Commit infra/openbao/openbao-unseal-sealedsecret.yaml, then:
kubectl apply -f apps/base/openbao.yaml
kubectl -n openbao get pod openbao-0 -w # Running, 0/1 ready: it isn't initialised yet
4. Initialise (once)
With an auto seal, init returns recovery keys and a root token, not unseal keys.
Recovery keys authorise operations like generate-root. They can't decrypt anything. Send the
output straight to a file in Proton Pass and never into a terminal you're recording or a chat:
kubectl -n openbao exec -ti openbao-0 -- bao operator init -recovery-shares=3 -recovery-threshold=2
# → Proton Pass item openbao-init: root token + 3 recovery keys
kubectl -n openbao get pod openbao-0 # 1/1 Ready: initialised and unsealed
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Seal Type|Sealed|Initialized'
# Seal Type static · Initialized true · Sealed false
Then delete the pod once and check it comes back unsealed by itself. That's the whole point of the static seal:
kubectl -n openbao delete pod openbao-0 && kubectl -n openbao wait pod openbao-0 --for=condition=Ready --timeout=120s
5. Access
With NetBird up, and no DNS needed:
curl -s http://100.66.50.60:8200/v1/sys/health | jq '{initialized, sealed}'
export BAO_ADDR=http://100.66.50.60:8200
bao login # root token, only to set up a real auth method; then `bao token revoke -self`
Plain HTTP is safe here only because of the tunnel. Traffic to 100.66.50.60 travels inside
NetBird's WireGuard tunnel, and infra/host-firewall drops port 8200 arriving on node1's LAN
interfaces. Re-run harden-firewall.sh after changing its port list. Without that, anyone on the
home Wi-Fi could reach OpenBao in clear text on 192.168.1.5:8200.
For HTTPS, use the hostname instead. Either add a Cloudflare record A bao 100.66.50.60
(DNS only), or add 100.66.50.60 bao.tiktuzki.com to /etc/hosts on each machine.
Don't use the root token day to day. Set up an auth method (OIDC with Google, or
kubernetes for in-cluster apps), then revoke the root token. bao operator generate-root
with the recovery keys mints a new one when you need it.
Google login
People sign in with Google through the oidc auth method. They get the admin policy if
their email is on the allow-list in infra/openbao/configure-google-login.sh, the same two
addresses as Argo CD and Grafana. Tokens last 1h and renew up to 12h.
1. Google OAuth client. OpenBao shares the tik-k8s client with Grafana (Google Cloud
console → APIs & Services → Credentials → tik-k8s). Add these Authorized redirect URIs
next to Grafana's:
https://bao.tiktuzki.com/ui/vault/auth/oidc/oidc/callback
http://localhost:8250/oidc/callback
Its client ID and secret are in pass://athena/tik-k8s-google-auth/{client_id,secret}, which google-login.env.tpl references.
If the consent screen is in Testing mode, both emails must be on its test-user list.
2. Configure OpenBao (re-runnable). Check that the paths in google-login.env.tpl match
your Pass items first:
brew install openbao
pass-cli run --env-file infra/openbao/google-login.env.tpl -- infra/openbao/configure-google-login.sh
3. Sign in.
- CLI:
BAO_ADDR=http://100.66.50.60:8200 bao login -method=oidcopens Google in the browser and returns tolocalhost:8250. - UI:
https://bao.tiktuzki.com/ui→ method OIDC → Sign in with OIDC Provider. The name must resolve to100.66.50.60(DNS or/etc/hosts).
⚠️ The UI at http://100.66.50.60:8200/ui can't do Google login. Google rejects plain-HTTP
and bare-IP redirect URIs (localhost is the only exception), and the UI builds its redirect
from the address in the browser bar.
4. Revoke the root token once a Google login works:
pass-cli run --env-file <(echo 'BAO_TOKEN=pass://athena/openbao-init/root_token') -- \
env BAO_ADDR=http://100.66.50.60:8200 bao token revoke -self
Then mark it revoked in Pass. When you need root again, use
bao operator generate-root -init with 2 of the 3 recovery keys.
Backup
The nightly cluster-backup run (infra/backup/backup.sh, section 1b) takes a raft snapshot to
secrets/openbao-raft.snap. That's the critical half, uploaded to S3 every night with the
openbao-unseal Secret it needs. A tarball of the raft directory could be torn, so it isn't
used.
No stored credential. backup.sh mints a 10-minute JWT for ServiceAccount openbao/openbao-backup
(audience openbao) and logs in at auth/kubernetes/role/backup. That gives a 5-minute token with
policy snapshot, which can read sys/storage/raft/snapshot and nothing else. The JWT travels on
stdin, and the token is revoked after use. Set up by infra/openbao/configure-backup-auth.sh
(re-runnable, needs an admin login).
Failures are loud. After 3 failed attempts the run is marked INCOMPLETE, and
ClusterBackupIncomplete fires. cluster_backup_openbao_snapshot_bytes is the snapshot size
(0 = missing).
Restore: infra/backup/RESTORE.md, step 5b. For a manual snapshot at any time:
BAO_ADDR=http://100.66.50.60:8200 bao operator raft snapshot save openbao-$(date +%F).snap
Rotating the unseal key
- Generate a new key as in step 2, but seal it under a second key name with
kubeseal --merge-into:unseal-<date>.key. - In
apps/base/openbao.yaml, setcurrent_key/current_key_idto the new key, andprevious_key/previous_key_idto the old one. Sync. OpenBao re-wraps its root key. - Once it has restarted cleanly, remove
previous_*and the old key name.