Skip to main content

Argo CD behind the VPN

Takes https://argocd.tiktuzki.com off the public internet. After this, only a machine on the NetBird network (or node1's LAN) can open the console, the API, or the argocd CLI. The hostname, Google SSO and its redirect URI stay exactly as they are.

before: browser → Cloudflare → NPM (VPS) → node1:80 (Traefik web) → argocd-server
after: browser ─(NetBird)──────────────→ node1:443 (Traefik websecure) → argocd-server

Three things make it private, and each one alone would not be enough:

LayerWhat it does
DNSargocd.tiktuzki.com → 100.66.50.60, node1's NetBird address, DNS-only. Anyone can resolve it; only NetBird peers can route to it
Networknode1 has no inbound path from the internet. The only public door is NPM on the VPS
Routingthe Argo CD router listens on websecure (:443) only. NPM forwards to :80, so even a re-created NPM proxy host gets a 404

TLS can no longer come from NPM or Cloudflare, so cert-manager issues a Let's Encrypt certificate with a DNS-01 challenge through the Cloudflare API, which works for a host Let's Encrypt cannot reach.

Files: infra/cert-manager/cluster-issuer-letsencrypt-dns.yaml, bootstraps/argocd/ingress-vpn.yaml.

0. Before you start​

  • NetBird policy. Every person who needs Argo CD must have a policy allowing their peer → node1 on TCP 443 (and 16443 if they also use kubectl).
  • node1 can resolve and reach the internet. cert-manager calls acme-v02.api.letsencrypt.org and api.cloudflare.com. If kubectl -n cert-manager logs deploy/cert-manager shows DNS timeouts, set extraArgs: [--dns01-recursive-nameservers-only, --dns01-recursive-nameservers=1.1.1.1:53] in apps/base/cert-manager.yaml.
  • Break-glass stays the same and needs no DNS: kubectl -n devops port-forward svc/argocd-server 8080:443.

1. Cloudflare API token, sealed​

Cloudflare → My Profile → API Tokens → Create Token → Edit zone DNS, scoped to Zone: tiktuzki.com only. Store it as pass://athena/cloudflare-dns-token/token, then seal it without the value ever touching a file or your shell history:

pass-cli run --env-file <(echo 'CF_DNS_TOKEN=pass://athena/cloudflare-dns-token/token') -- \
sh -c 'kubectl create secret generic cloudflare-api-token -n cert-manager \
--from-literal=api-token="$CF_DNS_TOKEN" --dry-run=client -o yaml' \
| kubeseal --controller-namespace sealed-secrets --format yaml \
> infra/cert-manager/cloudflare-api-token-sealedsecret.yaml

2. Issuer, ingress, certificate​

kubectl apply -f infra/cert-manager/
kubectl apply -f bootstraps/argocd/ingress-vpn.yaml

kubectl get clusterissuer letsencrypt-dns # READY True
kubectl -n devops get certificate argocd-tiktuzki-com-tls -w # READY True, usually 1–3 min

Stuck at False: kubectl -n devops describe challenge names the step that failed. The usual causes are a token without DNS edit rights, or node1 failing to resolve (step 0).

The public path still works at this point. The old ingress serves :80 and the new one serves :443.

3. Test over NetBird before touching DNS​

curl -s --resolve argocd.tiktuzki.com:443:100.66.50.60 https://argocd.tiktuzki.com/healthz
# ok (no -k: the certificate must verify)

4. Point DNS at node1​

Cloudflare → DNS → Add record: A argocd 100.66.50.60, Proxy status: DNS only (grey cloud), TTL Auto. An explicit record wins over the * wildcard.

dig +short argocd.tiktuzki.com @lindsey.ns.cloudflare.com # 100.66.50.60

Open https://argocd.tiktuzki.com with NetBird up and sign in with Google. The CLI also works: argocd login argocd.tiktuzki.com --sso --grpc-web.

5. Close the public path​

  1. NPM → Hosts → Proxy Hosts → delete argocd.tiktuzki.com.

  2. Delete the old :80 ingress and remove it from bootstraps/argocd/install.yaml:

    kubectl -n devops delete ingress argocd-server-ingress
  3. Verify from outside the VPN (netbird down, or a phone on mobile data):

    curl -s -m 5 https://argocd.tiktuzki.com/ ; echo "exit $?" # times out — 100.66.x is unroutable
    curl -sk -H 'Host: argocd.tiktuzki.com' https://<VPS public IP>/ -o /dev/null -w '%{http_code}\n' # not Argo CD
  4. Update the argocd section of ACCESS.md (public → NetBird only) and the developer guide.

Rollback​

Delete the argocd DNS record (the wildcard sends the name back to NPM), re-create the NPM proxy host, and re-apply the old ingress from bootstraps/argocd/install.yaml in git history.

What changes for users and tools​

PeopleNetBird must be connected, including at home on the LAN: DNS now points at the overlay address
GitHub webhooks to Argo CDcannot reach it. Argo CD falls back to polling Git every ~3 min
In-cluster callers (ARC runners, CI)unaffected — they use argocd-server.devops.svc
100.66.50.60 in public DNSvisible to anyone, but a CGNAT address that routes nowhere off the overlay