Chuyển tới nội dung chính
Chưa có bản tiếng Việt
Trang này hiện chỉ có bằng tiếng Anh. Các bài trong mục Production patterns đã được dịch.

OpenBao

OpenBao (the open-source Vault fork) on node1, at http://100.66.50.60:8200. It's reachable over NetBird only.

Argo appapps/base/openbao.yaml: the official chart openbao/openbao 0.30.7 (OpenBao v2.7.2), plus infra/openbao/
Namespaceopenbao
Storageraft, one node, static local PV /srv/k8s-volumes/openbao (5Gi, Retain)
Sealstatic key auto-unseal. The key is in SealedSecret openbao/openbao-unseal
In-clusterhttp://openbao-active.openbao.svc:8200
Over NetBirdhttp://100.66.50.60:8200 (UI at /ui): Traefik entryPoint openbao, hostPort 8200, infra/openbao/ingressroute.yaml. No DNS needed
Optional hostnamehttps://bao.tiktuzki.com: Traefik websecure, cert-manager letsencrypt-dns. Needs DNS or an /etc/hosts line → 100.66.50.60
Injector / CSIoff. Turn them on when the first app reads its secrets from OpenBao

Why a static seal​

node1 restarts every pod several times a week. Each time its Wi-Fi reconnects, microk8s re-issues the API server certificate and kills containerd. A Shamir-sealed OpenBao would come back sealed every time and serve nothing until someone typed in key shares.

The static seal reads a 32-byte AES key from a mounted Secret and unseals by itself. The cost: anyone who can read Secrets in openbao, or who holds the sealed-secrets master key, can unseal. The cluster is the root of trust. To move to a stronger seal (a cloud KMS or Transit), do a seal migration. Editing the config isn't enough.

Deploy​

0. Before you start​

  • letsencrypt-dns ClusterIssuer is READY True. It's shared with Argo CD; see Argo CD behind the VPN, steps 1–2.
  • Your NetBird peer is allowed TCP 443 to node1.

1. Volume directory (on node1)​

sudo ./infra/storage/create-volume-dirs.sh # creates /srv/k8s-volumes/openbao as 100:1000, 0700

2. Unseal key: generate, seal, keep a copy​

The key never prints, and it never stays on disk:

umask 077; K=$(mktemp)
openssl rand -out "$K" 32
kubectl create secret generic openbao-unseal -n openbao --from-file=unseal.key="$K" \
--dry-run=client -o yaml \
| kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller \
--format yaml > infra/openbao/openbao-unseal-sealedsecret.yaml
base64 < "$K" | pbcopy # paste into Proton Pass as pass://athena/node1-openbao-unseal/secret
rm -P "$K"; pbcopy < /dev/null

⚠️ The Proton Pass copy is not optional. Without this key, the raft data and every snapshot are unreadable. infra/backup/backup.sh also captures it, because it dumps every Secret, but don't rely on that backup alone.

3. Commit and apply​

Commit infra/openbao/openbao-unseal-sealedsecret.yaml, then:

kubectl apply -f apps/base/openbao.yaml
kubectl -n openbao get pod openbao-0 -w # Running, 0/1 ready: it isn't initialised yet

4. Initialise (once)​

With an auto seal, init returns recovery keys and a root token, not unseal keys. Recovery keys authorise operations like generate-root. They can't decrypt anything. Send the output straight to a file in Proton Pass and never into a terminal you're recording or a chat:

kubectl -n openbao exec -ti openbao-0 -- bao operator init -recovery-shares=3 -recovery-threshold=2
# → Proton Pass item openbao-init: root token + 3 recovery keys
kubectl -n openbao get pod openbao-0 # 1/1 Ready: initialised and unsealed
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Seal Type|Sealed|Initialized'
# Seal Type static · Initialized true · Sealed false

Then delete the pod once and check it comes back unsealed by itself. That's the whole point of the static seal:

kubectl -n openbao delete pod openbao-0 && kubectl -n openbao wait pod openbao-0 --for=condition=Ready --timeout=120s

5. Access​

With NetBird up, and no DNS needed:

curl -s http://100.66.50.60:8200/v1/sys/health | jq '{initialized, sealed}'
export BAO_ADDR=http://100.66.50.60:8200
bao login # root token, only to set up a real auth method; then `bao token revoke -self`

Plain HTTP is safe here only because of the tunnel. Traffic to 100.66.50.60 travels inside NetBird's WireGuard tunnel, and infra/host-firewall drops port 8200 arriving on node1's LAN interfaces. Re-run harden-firewall.sh after changing its port list. Without that, anyone on the home Wi-Fi could reach OpenBao in clear text on 192.168.1.5:8200.

For HTTPS, use the hostname instead. Either add a Cloudflare record A bao 100.66.50.60 (DNS only), or add 100.66.50.60 bao.tiktuzki.com to /etc/hosts on each machine.

Don't use the root token day to day. Set up an auth method (OIDC with Google, or kubernetes for in-cluster apps), then revoke the root token. bao operator generate-root with the recovery keys mints a new one when you need it.

Google login​

People sign in with Google through the oidc auth method. They get the admin policy if their email is on the allow-list in infra/openbao/configure-google-login.sh, the same two addresses as Argo CD and Grafana. Tokens last 1h and renew up to 12h.

1. Google OAuth client. OpenBao shares the tik-k8s client with Grafana (Google Cloud console → APIs & Services → Credentials → tik-k8s). Add these Authorized redirect URIs next to Grafana's:

https://bao.tiktuzki.com/ui/vault/auth/oidc/oidc/callback
http://localhost:8250/oidc/callback

Its client ID and secret are in pass://athena/tik-k8s-google-auth/{client_id,secret}, which google-login.env.tpl references. If the consent screen is in Testing mode, both emails must be on its test-user list.

2. Configure OpenBao (re-runnable). Check that the paths in google-login.env.tpl match your Pass items first:

brew install openbao
pass-cli run --env-file infra/openbao/google-login.env.tpl -- infra/openbao/configure-google-login.sh

3. Sign in.

  • CLI: BAO_ADDR=http://100.66.50.60:8200 bao login -method=oidc opens Google in the browser and returns to localhost:8250.
  • UI: https://bao.tiktuzki.com/ui → method OIDC → Sign in with OIDC Provider. The name must resolve to 100.66.50.60 (DNS or /etc/hosts).

⚠️ The UI at http://100.66.50.60:8200/ui can't do Google login. Google rejects plain-HTTP and bare-IP redirect URIs (localhost is the only exception), and the UI builds its redirect from the address in the browser bar.

4. Revoke the root token once a Google login works:

pass-cli run --env-file <(echo 'BAO_TOKEN=pass://athena/openbao-init/root_token') -- \
env BAO_ADDR=http://100.66.50.60:8200 bao token revoke -self

Then mark it revoked in Pass. When you need root again, use bao operator generate-root -init with 2 of the 3 recovery keys.

Backup​

The nightly cluster-backup run (infra/backup/backup.sh, section 1b) takes a raft snapshot to secrets/openbao-raft.snap. That's the critical half, uploaded to S3 every night with the openbao-unseal Secret it needs. A tarball of the raft directory could be torn, so it isn't used.

No stored credential. backup.sh mints a 10-minute JWT for ServiceAccount openbao/openbao-backup (audience openbao) and logs in at auth/kubernetes/role/backup. That gives a 5-minute token with policy snapshot, which can read sys/storage/raft/snapshot and nothing else. The JWT travels on stdin, and the token is revoked after use. Set up by infra/openbao/configure-backup-auth.sh (re-runnable, needs an admin login).

Failures are loud. After 3 failed attempts the run is marked INCOMPLETE, and ClusterBackupIncomplete fires. cluster_backup_openbao_snapshot_bytes is the snapshot size (0 = missing).

Restore: infra/backup/RESTORE.md, step 5b. For a manual snapshot at any time:

BAO_ADDR=http://100.66.50.60:8200 bao operator raft snapshot save openbao-$(date +%F).snap

Rotating the unseal key​

  1. Generate a new key as in step 2, but seal it under a second key name with kubeseal --merge-into: unseal-<date>.key.
  2. In apps/base/openbao.yaml, set current_key / current_key_id to the new key, and previous_key / previous_key_id to the old one. Sync. OpenBao re-wraps its root key.
  3. Once it has restarted cleanly, remove previous_* and the old key name.