Git knows the name, OpenBao knows the value

For a long time my secrets lived in two bad places.

In Kubernetes they were made by hand: kubectl create secret generic ..., typed once, never written down anywhere else. My cluster runs on Talos, everything else in it is declared in Git and applied by Argo CD, and yet a rebuild would have meant rediscovering thirty-odd Secrets from memory and old terminal history.

In my Ansible repository it was worse. Passwords for databases, mail, a few services, and some private keys sat in variables and files, in plain text, in Git.

Secrets were the one exception to “everything is code”, and exceptions are where things go wrong. So this week I moved all of them into OpenBao. It was a big job: a new VM, a three-node vault in the cluster, a second operator, config as code, thirty Secrets in twelve namespaces, and an entire Ansible repository. This is the long version: the design, how the secrets moved without a single pod restart, and everything that bit me.

The idea in one sentence

Git says which secrets exist. OpenBao holds the values.

OpenBao is the open source fork of HashiCorp Vault, run under the Linux Foundation. It’s a vault: an encrypted store for secrets, with access policies, tokens that expire, and an audit log of every single read. I picked it over Vault itself for the licence, because its Helm chart ships a snapshot agent out of the box, and because the tooling I wanted to use with it supports both.

The bridge into Kubernetes is the External Secrets Operator, ESO for short. You write an ExternalSecret resource in Git that says “make a Kubernetes Secret called X from this path in OpenBao”, and ESO keeps the two in sync. Apps don’t know OpenBao exists. They keep reading an ordinary Kubernetes Secret with the same name as before. That was a hard requirement: I didn’t want to touch thirty applications to change where their passwords come from.

For Ansible there’s no operator in between. Playbooks look the values up from OpenBao at run time.

What I wanted out of it

Before any YAML, I wrote down what “done” should mean:

• Git declares every secret. Its name, its namespace, where the value lives. Not the value.
• Every read is audited. Who read what, when.
• Secrets can be rotated in one place. Change it in OpenBao, and within the hour the cluster has the new value.
• Apps don’t change. Same Secret names, same keys.
• No manual unseal, ever. The lab reboots at night for updates.
• Nothing restarts during the migration.

That list decided most of the design.

Four plans instead of one big weekend

Like the monitoring migration, this started as a design document and was split into plans small enough to finish and verify one at a time:

Each plan ended with the system working and verified, so stopping halfway would never have left the lab broken. That turned out to matter: plan 3’s last wave, with the password manager and AWX in it, was explicitly marked “only with Robin awake”.

The shape of it

bao00  (small VM, outside the cluster)
  OpenBao, one node, unseals itself from a key on its own disk
  transit engine: the key that unseals the cluster
        ▲
        │  TLS, a token that may only encrypt/decrypt with that one key
        │
Kubernetes cluster
  OpenBao, 3 pods on 3 different nodes, Raft storage
  snapshots to S3 every 6 hours
        ▲
        │  Kubernetes auth, one role per namespace
        │
  External Secrets Operator  ──►  Kubernetes Secrets  ──►  apps

AWX / Ansible  ──►  OpenBao (AppRole)  ──►  lookups at run time

Why there’s a second OpenBao on a VM

The strange-looking part is the extra OpenBao outside the cluster. It’s there because of unsealing.

A vault starts sealed. Its storage is encrypted, and until someone provides the key it can’t read anything, not even its own configuration. The classic answer is Shamir’s secret sharing: split the key into five parts, give them to five people, and have three of them type their part after every restart. In a company that’s a ceremony. In a homelab that reboots nodes at three in the morning for updates, it’s an outage.

The other classic answer is a cloud key service. That would make my lab depend on a cloud account for something as basic as starting up.

So a tiny OpenBao on its own VM does exactly one job. Its transit engine holds a key that the cluster’s OpenBao uses to unseal itself. The cluster has a token that can do nothing except encrypt and decrypt with that one key. The VM in turn unseals itself from a key file on its own disk, readable only by the OpenBao user.

It’s worth being honest about what this buys. It doesn’t make the VM unbreakable. What it does is split the problem: someone with the cluster alone can’t unseal the vault, and someone with the VM alone has no data. They need both. The VM runs on the datacenter node, next to most of the cluster, so it doesn’t depend on my home internet connection.

And the failure mode is gentle. If the VM goes down, the cluster vault keeps running, because it’s already unsealed. Only a pod that restarts during the outage has to wait until the VM is back.

The part that isn’t gentle: if the VM is lost for good, the cluster vault can never be unsealed again. Recovery keys can’t unseal a vault that uses auto-unseal. So the VM gets three layers of backup: VM backups on the backup server, a daily snapshot of its own data copied to S3, and its seal key and recovery key stored in the password manager.

Three pods in the cluster

The cluster side is the official OpenBao Helm chart:

• Three replicas with Raft storage. Raft is OpenBao’s built-in replicated storage, so there’s no separate database to run. One pod is the leader, two are standbys, and a hard anti-affinity rule puts them on three different worker nodes.
• A disruption budget of one. Node drains during updates take one pod at a time.
• Encrypted volumes. Data and audit logs live on Longhorn’s encrypted storage class.
• Upgrades are manual on purpose. The StatefulSet uses OnDelete: snapshot first, delete the standbys one at a time, then the leader, checking the Raft peer list between each step. The Argo CD application is manual sync with no pruning. This is the one component where I want to press the button myself.
• No sidecar injector. ESO does the delivery, so the chart’s injector and CSI driver stay switched off.

A network policy that says who may talk

A Cilium network policy allows traffic to OpenBao’s port only from the OpenBao pods themselves (for Raft), the External Secrets pods, the internal ingress for the web interface, the monitoring namespace, and the snapshot job. Nothing else in the cluster can even open a connection to it. The web interface is only on the internal network, never on the public BunkerWeb edge.

Certificates, and the CA I couldn’t use

OpenBao wants TLS everywhere: between its own pods, from ESO, and to the unseal VM. The design said: sign all of it with my homelab’s root CA.

That’s where the first surprise came from. The root CA’s private key is protected by a passphrase, and nobody remembers it. Nothing can be signed with it. Rather than stall the project, the design changed:

• cert-manager runs its own small CA for OpenBao. A self-signed issuer creates a CA certificate, and that CA issues the pods’ certificates, with a 90-day lifetime and renewal 30 days before expiry. No private key has to be created by hand or stored anywhere outside the cluster.
• The unseal VM gets a self-signed certificate that the cluster pins. Instead of trusting a CA, the cluster trusts exactly that one certificate, from a ConfigMap. If the certificate on the VM ever changes unexpectedly, an alert fires.

One detail to know: OpenBao pods keep their certificate in memory until they restart. cert-manager renews the file, but the pods only pick it up on the next rolling restart. An alert watches the certificate the pods actually serve, not the one on disk, and warns at 14 days.

Rotating the old homelab CA properly is now on the list, as its own project.

Configuration as code

The OpenBao setup itself, which secret engines exist, which auth methods, which policies and roles, is OpenTofu code in my infrastructure repository. It manages:

• the key-value store where the secrets live;
• Kubernetes authentication, with one role per namespace;
• a policy per namespace that can read only that namespace’s path;
• a role for the snapshot job, which can take Raft snapshots and nothing else;
• username-and-password login for me, with tokens that live one hour and at most eight.

The pipeline follows the same rules as the rest of that repository. Any branch gets a lint. On main, a plan runs automatically and applying it is a manual button I press after reading the plan. The token the pipeline uses is a protected CI variable, only available on the protected branch and in that one environment, and it can manage policies and auth methods but has no access to a single secret value. A weekly scheduled pipeline renews it so it never expires quietly.

With that in place, the root token was revoked. There’s no standing all-powerful token anywhere. When one is needed, it’s generated from the recovery keys, used, and revoked again.

One namespace, one key ring

The part I spent the most thought on is who may read what.

The easy setup is a single ClusterSecretStore: one store for the whole cluster, used by every namespace. It’s what most tutorials show. It also means one identity that can read everything, and a Secret in any namespace could ask for any path.

Instead, every namespace gets its own small store, in the same folder as the app’s other manifests. A service account called openbao-reader in that namespace logs in to OpenBao with a role that can only read that namespace’s path. ESO is allowed to create a token for that one service account in that one namespace and nothing else:

# ESO may create tokens for this one ServiceAccount, in this namespace only.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-openbao-reader-token
  namespace: homepage
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["openbao-reader"]
    verbs: ["create"]
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: homepage
spec:
  provider:
    vault:
      server: https://openbao-active.openbao.svc:8200
      path: kv
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso-homepage            # policy: read kv/homepage/* only
          serviceAccountRef:
            name: openbao-reader

The detail that makes this work is a chart setting. By default, the ESO Helm chart grants the operator the right to create tokens for any service account in the cluster. That’s the real hole: with it, a misconfigured store could log in as any identity. Setting rbac.serviceAccountTokenCreate: false removes it, and each namespace grants it back for its own reader only.

Before moving a single real secret, the pilot tested exactly that: a store in one namespace trying to read another namespace’s path. It was refused by the policy, by the network, and by authentication. Only then did the migration start.

The honest limitation: the ESO controller itself still has cluster-wide read and write on Secrets, because it has to write them. Token minting is scoped; the controller isn’t. That’s inherent to how any operator like this works.

Two small practical things. The store embeds the public CA certificate inline, because a namespaced store can only reference Secrets in its own namespace, and the CA lives in the OpenBao namespace. And testing permissions with kubectl auth can-i --as was useless in my cluster: through Omni’s proxy it answers “yes” to everything. Real tests had to be real requests.

Moving secrets without restarting anything

The migration itself was the part I was most nervous about. Around thirty Secrets in twelve namespaces, including the password manager, GitLab registry credentials, Argo CD’s image updater credentials, the monitoring stack, AWX, and the customer portal, whose single Secret has 45 keys.

The two settings that made it boring

The trick is one pair of settings on every ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: homepage-secrets
  namespace: homepage
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: openbao
    kind: SecretStore
  target:
    name: homepage-secrets        # the Secret that already exists
    creationPolicy: Orphan        # take it over in place
    deletionPolicy: Retain        # never delete it
  dataFrom:
    - extract:
        key: homepage/homepage-secrets

With creationPolicy: Orphan, ESO doesn’t create a new Secret. It adopts the existing one and writes into it. Same object, same UID. Because the values in OpenBao are identical to what’s already there, nothing notices. No pod restarted during the whole migration.

deletionPolicy: Retain is the safety net the other way round: if I ever delete the ExternalSecret, the Secret it managed stays where it is. A typo in Git can’t take down an app’s credentials.

How each wave went

Every wave followed the same steps:

1. Inventory. List every Secret in the namespaces of the wave, who uses it, and which Argo CD application owns the namespace. Read-only.
2. Roles first. Add the namespace’s OpenBao role and policy in OpenTofu, plan, apply.
3. Stores second. Commit the namespace’s SecretStore and check it reports ready.
4. Seed. Copy each live Secret into OpenBao with a small script that reads it straight from the cluster and writes it into OpenBao, without ever printing a value.
5. Adopt. Commit the ExternalSecrets.
6. Verify. Compare the Secret’s content hash before and after. Not the values. The hashes.

Wave 1 was the pilot: nine Secrets in the dashboard and monitoring namespaces. Wave 2 was the bulk: the wiki, Coder, CrowdSec, BunkerWeb’s bouncer key, the registry credentials in four namespaces, and the customer portal. Wave 3 was the password manager and AWX, done one Secret at a time with a manual sync for each.

A few apps are on manual sync in Argo CD, BunkerWeb and Coder among them. For those, only the new ExternalSecrets were synced selectively, so a wave never dragged in an unrelated change to the edge proxy.

Secrets that used to be in Git

Two Secrets in the monitoring stack were declared in Git as manifests. Just deleting those files would have made Argo CD prune the live Secrets before ESO had adopted them. So they got an extra commit first: the annotation argocd.argoproj.io/sync-options: Prune=false, which tells Argo CD to leave the live object alone when its manifest disappears. Then the manifests were removed, and ESO took over.

What deliberately stays outside

Not every Secret belongs in OpenBao:

• Bootstrap secrets. Anything that must exist before OpenBao can run: its own unseal token, its snapshot bucket credentials, Longhorn’s encryption key and Argo CD’s own secrets. A vault can’t unlock itself with a key that’s inside it.
• Generated secrets. Operators and charts that create their own Secrets, like AWX’s internal credentials and cert-manager’s certificates, keep doing that. Adopting them would make two controllers fight over one object.

Three unused duplicates turned up in the inventory, copies of Coder’s login secrets in the default namespace that nothing referenced. They were deleted instead of migrated.

Ansible: lookups instead of values

The second half was my Ansible repository, which AWX runs. The goal: the repository holds no secret values at all.

Lookups

Every plaintext value became a lookup from OpenBao, using the community.hashi_vault collection, which works against OpenBao unchanged:

# before
mysql_root_password: "..."

# after
mysql_root_password: >-
  {{ lookup('community.hashi_vault.vault_kv2_get', 'awx-homelab/db',
            engine_mount_point='kv').secret.mysql_root_password }}

The values live under one path per scope: databases, mail, the proxies, and so on. Seventeen variables in group vars, host vars and role defaults changed this way.

Proving nothing changed

This is the part I’d copy into any migration like it. Before each commit, a small verify script ran every lookup and compared the result to the old plaintext value, by hash, so neither the old nor the new value ever showed up in a terminal or a log. 17 out of 17 matched. Only then was the plaintext removed.

The private keys that used to live in the repository moved too, and so did the password AWX itself uses and the SSH key it connects to hosts with. Files that playbooks need on disk, like a certificate for the proxies, are now written from a lookup at run time instead of being copied from the repository.

AWX’s identity

AWX logs in to OpenBao with an AppRole, a machine identity made of a role ID and a secret ID. A custom credential type in AWX holds both and is attached to every job template, so every job can read the Ansible path in OpenBao and nothing else.

For runs from my own shell, the same lookups work with environment variables pointing at OpenBao and a short-lived token from my own login.

What went wrong

The certificate authority story is above. Here’s the rest, in the order it happened.

I lost the root token on day one. After the initial setup, the cluster’s root token was no longer valid, and the standard recovery command refused to work. Newer OpenBao versions switch off the unauthenticated endpoint for generating a root token by default, for good reasons. The fix was to turn it back on for exactly as long as the recovery took, generate a new root token with the recovery keys, and revert the change in the very next commit. Both commits are in Git with a clear message, which I like: even the emergency is documented. The rule I kept during the recovery: the encoded token and the one-time password that decodes it never appear in the same place.

Audit devices couldn’t be enabled the normal way. The usual command to enable an audit device returns an error in OpenBao’s current version. They are declared in the server configuration instead: one writing to a file, one to standard output so it lands in the pod logs and gets shipped with everything else. I think that’s better anyway. Audit logging is now part of the config in Git, not a runtime setting someone could switch off.

A token almost landed in a log. The first version of the seed script passed the OpenBao token as an argument to kubectl exec. Command arguments end up in the Kubernetes API server’s audit log. It now goes over stdin. Small, but exactly the kind of leak a migration like this is supposed to prevent.

The unseal token has an expiry date. The token the cluster uses to reach the unseal VM is renewed automatically every 24 hours. If the VM is unreachable for longer than that, or the cluster is restored from a snapshot old enough, the token has expired and the vault can’t unseal even after the VM is back. The recovery is documented step by step in the runbook, including how to mint a new token on the VM. I decided to keep the 24-hour period rather than make the token live longer.

A pod that starts while the VM is down crash-loops. I expected it to wait politely, sealed. It exits and backs off until the VM is reachable again. Harmless, and deleting the pod skips the back-off, but good to know before you see it in the middle of the night.

An alert rule that couldn’t evaluate. The first audit-failure alert combined several metric names in one selector, and Prometheus dropped the names in the result, which made two series look identical. The rule errored instead of alerting. ESO’s own metrics had a similar trap: the namespace of a failing ExternalSecret is in a label called exported_namespace, not namespace, because the scrape adds its own.

The Ansible execution environment hadn’t built since April. Adding the vault collection to AWX’s container image meant rebuilding it, and the build had been silently broken for months over a certificate problem in its container-in-container setup. Fixing secrets management started with fixing a pipeline nobody had looked at.

AWX refuses some environment variables. The obvious way to hand the OpenBao address to Ansible is ANSIBLE_HASHI_VAULT_* environment variables from the custom credential. AWX doesn’t allow credentials to inject any ANSIBLE_* variable. The credential injects extra vars instead, which the lookups read the same way.

Policy changes need their own permissions. Adding the AppRole for AWX failed on the first apply with a 403: the configuration pipeline’s own policy wasn’t allowed to manage AppRole roles yet. It took a small change to that policy, applied first, and then the real change. Least privilege works; it also means you meet it.

Monitoring it

A vault that stops working quietly is worse than no vault, because everything keeps running on cached Secrets until the day something restarts. So it’s monitored from every side, feeding into the same Alertmanager as everything else:

There’s a Grafana dashboard next to it, and the VM has the same host monitoring as every other machine.

Backups

The cluster vault snapshots its Raft storage to S3 every six hours and keeps two weeks. The unseal VM snapshots daily, to its own folder in the same bucket. Both have alerts when a snapshot is late. A backup you haven’t restored is a hope, so the plan’s acceptance test restores a snapshot into a scratch namespace and reads a known key back.

If you want to build the same

The order I’d do it in, knowing what I know now:

1. Decide how the vault unseals. Transit on a separate machine, a cloud key service, or manual. Everything else follows from this.
2. Sort out TLS before you start. Check you can actually sign with the CA you’re planning to use.
3. Deploy the vault with three replicas and Raft, anti-affinity, a disruption budget and manual upgrades.
4. Put its configuration in code with a plan-and-apply pipeline, then revoke the root token.
5. Install ESO with cluster-wide token creation turned off.
6. Pilot one namespace and try to break the isolation before anything real depends on it.
7. Migrate in waves with Orphan and Retain, verifying hashes, the riskiest apps last.
8. Then the configuration management, with lookups and a verify script.
9. Alerts and backups from day one, not after.

What I’d tell you before you start

• Design unsealing first. It decides what happens after every reboot, and what you lose if one machine dies.
• Per-namespace stores, and turn off ESO’s cluster-wide token minting. It’s one chart value and it’s the whole security model.
• Adopt Secrets in place with Orphan and Retain. It turns a scary migration into a series of boring commits.
• Never print a secret to prove it matches. Compare hashes.
• Keep bootstrap secrets outside the vault. Write down which ones they are and why.
• Moving a secret out of Git doesn’t remove it from Git history. Anything that was ever committed should be treated as known, and rotated.

What’s next

Rotating everything that ever lived in Git, and replacing the old homelab CA. After that, the part of a vault I haven’t used yet: letting OpenBao issue short-lived database credentials on demand, instead of storing long-lived passwords at all.