The Problem

GitOps has one famous exception: secrets. Everything else in my clusters is a file in git, but credentials can’t be, so every GitOps repo grows a secrets strategy.

I currently run two different ones, on purpose:

• Homelab: Sealed Secrets, plus a few gitignored imperative secrets
• A team platform I built: External Secrets Operator backed by a self-hosted Vaultwarden organization

This isn’t indecision. It’s the same decision made twice with different inputs. The variable that flips the answer isn’t security or scale of workloads. It’s the number of humans.

Sealed Secrets: Exactly Enough for a Party of One

The homelab model: encrypt a secret with the cluster controller’s public key, commit the sealed blob, and only the in-cluster controller can decrypt it.

kubeseal --controller-name sealed-secrets < mysql-secret.yaml > mysql-sealedsecret.yaml

Nothing sensitive in git, no external dependencies, nothing extra to run. For a solo operator this is close to perfect, and I want to defend it before I list its costs, because the costs are real but bounded:

The key dance. Every sealed secret is encrypted for one controller keypair: lose the key, lose every secret. So the private key gets backed up, and every cluster rebuild starts with the restore incantation:

# Backup (store main.key somewhere very safe, NOT in the repo)
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > main.key

# Restore on a rebuilt cluster
kubectl delete secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key=active
kubectl apply -f main.key
kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

Per-cluster sealing. A secret sealed for talos-metal is useless on talos-02; promoting an app means re-sealing. My repo carries main-talos-metal.key and main-talos-02.key backups, plural.

The plaintext shadow repo. To re-seal you need the original values, so gitignored *-secret.yaml files lurk next to their sealed twins. A few low-churn secrets never even got sealed; they’re applied imperatively at bootstrap and live only in the gitignore.

Annoying? Mildly. But I rebuild clusters a few times a year, I’m the only person who touches any of this, and the whole mental model fits in one shell history. For one operator, the simplicity is worth more than the friction costs. The homelab is staying on Sealed Secrets. (I did test-drive External Secrets against Bitwarden’s cloud Secrets Manager on my legacy cluster. It worked, and it also confirmed there was no problem here for it to solve with one human on the team.)

What Broke the Model: Other People

The platform I built at work is a different animal. Multi-tenant, per-client and per-user instances, and every instance carries credentials: API keys, tokens, connection strings. Two things happened fast:

1. Secrets scaled with tenants, not with apps. Onboarding a client isn’t one secret; it’s a handful, times every user, times every environment.
2. Teammates needed to add and rotate secrets. The sealed-secrets workflow has a brutal onboarding curve for that. “Install kubeseal, fetch the right controller cert for the right cluster, seal against it, commit YAML to the infra repo” is a lot to ask of someone who just wants to update an API key. SOPS has the same problem with extra GPG homework.

The tooling wasn’t the bottleneck; the people-to-tooling interface was. What the team needed was: open a familiar UI, paste the new value, done. And the team already lived in a password manager all day. That’s what pointed at Vaultwarden: the same Bitwarden apps and browser extension everyone already understood, self-hosted, with an organization as the shared secrets store.

External Secrets + Vaultwarden

External Secrets Operator (ESO) inverts the flow: secrets live in the password manager as the source of truth, and the cluster pulls them.

Vaultwarden organization   ← humans add/rotate here, in the Bitwarden UI they already know
    → bitwarden-cli deployment (bw serve REST API, in-cluster)
        → ClusterSecretStore (ESO webhook provider + jsonPath)
            → ExternalSecret (in git: just a pointer, nothing sensitive)
                → plain Kubernetes Secret (created/refreshed by ESO)

ESO has no native Vaultwarden provider, so the glue is the webhook provider pointed at an in-cluster deployment of the official bw CLI running bw serve, a little REST API in front of the vault. Two ClusterSecretStores extract values with jsonPath, one for login items and one for custom fields:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: bitwarden-login
spec:
  provider:
    webhook:
      url: "http://bitwarden-cli.external-secrets.svc.cluster.local:8087/object/item/{{ .remoteRef.key }}"
      result:
        jsonPath: "$.data.login.{{ .remoteRef.property }}"
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: bitwarden-fields
spec:
  provider:
    webhook:
      url: "http://bitwarden-cli.external-secrets.svc.cluster.local:8087/object/item/{{ .remoteRef.key }}"
      result:
        jsonPath: "$.data.fields[[email protected]==\"{{ .remoteRef.property }}\"].value"

And what lives in git changes character completely. An ExternalSecret is a reference, identical for every cluster, with nothing to seal and nothing to rotate in git:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: client-api-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: bitwarden-login
  target:
    name: client-api-credentials
  data:
    - secretKey: password
      remoteRef:
        key: <vault-item-uuid>
        property: password

Rotation became: change the value in the Bitwarden UI, wait for the refresh interval (or force it). No kubeseal, no repo access, no Kubernetes knowledge required. That’s the entire reason this stack exists.

The Pod That Knows Everything

Be honest about what bw serve is: an HTTP endpoint that answers with plaintext secrets from an unlocked vault session. That pod is the most sensitive workload on the cluster, and it gets treated accordingly:

• NetworkPolicy so only ESO can talk to port 8087. On a default-deny cluster (see the Hermes post), nothing else can even complete a TCP handshake with it
• Single replica, Recreate strategy. The CLI holds one vault session; two instances fighting over sync state helps nobody
• One bootstrap secret. The vault host and API credentials that unlock the session are the one secret that can’t come from the secrets manager, applied once per cluster

Migrating: the Controller Becomes the Export Tool

Getting existing secrets into Vaultwarden had a pleasing trick to it. The sealed blobs in git can’t be exported, but the live cluster has already decrypted all of them. So the migration script walks the running cluster’s Secrets (courtesy of the sealed-secrets controller having done its job), and imports each one into Vaultwarden via the bw CLI as the new source of truth. The old system’s output became the new system’s input; the sealed-secrets controller spent its final days as an export utility.

The Actual Decision Table

The takeaway I’d offer anyone choosing: count the humans who will touch secrets, not the secrets. One human: Sealed Secrets’ friction is a rounding error and its simplicity is a gift. More than one: the first time a teammate asks “how do I update the API key?”, you’ll want the answer to be a URL they already have bookmarked, not a wiki page about kubeseal.

Traps

Outstanding Work

1. Homelab stays put, but document the key locations. The sealed-secret key backups are the single most important files I own and their storage story is “I remember where they are”
2. ESO for shared homelab secrets, maybe. If family members ever manage their own app credentials, that’s the “second human” trigger
3. Rotation cadence on the team platform. Rotation is now easy enough to schedule; easy and done are still different things

References

• https://github.com/bitnami-labs/sealed-secrets
• https://external-secrets.io/
• https://github.com/dani-garcia/vaultwarden
• https://bitwarden.com/help/cli/