The Problem

I run several Kubernetes clusters: bare-metal prod (talos-metal), legacy prod on VMs (talos-02), plus dev and edge clusters that come and go. Every cluster needs roughly the same infrastructure: Cilium, cert-manager, external-dns, storage, monitoring. On top of that, each one has its own set of apps. Managing that by hand, or even with per-cluster ArgoCD Application manifests, turns into copy-paste drift almost immediately.

What I want: adding an app to a cluster is mkdir + git push, and bootstrapping an entire new cluster is one kubectl apply plus one label.

The Directory Convention

Everything hangs off one convention in the homelab repo:

apps/<name>/envs/<environment>/     # end-user applications
infra/<name>/envs/<environment>/    # operators and platform components
appsets/                            # the ApplicationSet definitions
root-argocd-app.yml                 # the single entry point

Each envs/<environment> directory is a kustomization that composes the app’s base/ with environment-specific patches. If apps/immich/envs/metal/ exists, immich runs on the metal cluster. If it doesn’t, it doesn’t. The filesystem is the deployment matrix.

The Root App

One Application to rule them all. It points at the appsets/ directory and nothing else:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: all-apps
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://gitlab.com/sirmysterion/homelab.git
    targetRevision: HEAD
    path: appsets
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

Bootstrap of a brand-new cluster is: install ArgoCD, apply this one file, label the cluster. Everything else follows from git.

Matrix Generators

Each environment gets a pair of ApplicationSets (apps + infra). The generator is a matrix of a git directory generator and a cluster selector:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: metal-appset
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://gitlab.com/sirmysterion/homelab.git
              revision: HEAD
              directories:
                - path: apps/*/envs/metal
          - clusters:
              selector:
                matchLabels:
                  argocd.argoproj.io/secret-type: cluster
                  environment: metal
  template:
    metadata:
      name: '{{index .path.segments 1}}'
    spec:
      project: default
      source:
        repoURL: https://gitlab.com/sirmysterion/homelab.git
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{index .path.segments 1}}'

The two generators multiply: the git generator produces one entry per matching directory, and the cluster selector gates the whole set on a cluster with the right environment label existing. {{index .path.segments 1}} extracts the app name from apps/<name>/envs/metal. That name becomes both the Application name and the target namespace.

Registering a cluster into an environment is a label:

argocd cluster set in-cluster --label environment=metal

Sync Policy: Where I Deviated from the Defaults

Every appset in this repo started life with the same sync policy, copied straight out of the ArgoCD docs:

syncPolicy:
  automated:
    prune: true
    selfHeal: true   # Specifies if partial app sync should be executed when resources are
                     # changed only in target Kubernetes cluster and no git change detected

That comment is a tell. Ten of my twelve appsets still carry it word for word, which means nobody has revisited their sync policy since the day the file was created. They work because nothing on those clusters has pushed back yet.

Metal is where something pushed back. It is the newest cluster and the one carrying the workloads I actually care about, so it is the first place the defaults met a real operator.

The Revert War

Several apps have operators or controllers that legitimately mutate their own resources at runtime. With selfHeal: true, ArgoCD sees the drift, reverts it to match git, the operator immediately re-applies its change, and the two settle into a delightful little loop. The Application sits at “Progressing” forever and the workload restarts every time the argument goes another round.

One commit fixed it on both metal appsets:

syncPolicy:
  syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    - ApplyOutOfSyncOnly=true
    - SkipDryRunOnMissingResource=true   # infra appset only
  automated:
    prune: true
    selfHeal: false   # operators reconcile their own fields; don't fight them

Turning selfHeal off means ArgoCD stops enforcing cluster state between git changes. That sounds like giving up a guarantee, and it is, but the guarantee was never real for resources with a controller behind them. Git remains the source of truth for what gets applied. The operator stays the source of truth for the fields it owns.

Naming the Fields Instead

selfHeal: false stops the fighting wholesale. The narrower fix is to name the specific fields that are allowed to drift, and the metal infra appset has accumulated a decent list of those:

ignoreDifferences:
  # kubectl rollout restart stamps this on every manual restart
  - group: apps
    kind: Deployment
    jsonPointers:
      - /spec/template/metadata/annotations/kubectl.kubernetes.io~1restartedAt
  - group: apps
    kind: StatefulSet
    jsonPointers:
      - /spec/template/metadata/annotations/kubectl.kubernetes.io~1restartedAt

  # operators rewrite their own CRD metadata after apply
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    jsonPointers:
      - /metadata/annotations/controller-gen.kubebuilder.io~1version
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    name: cdis.cdi.kubevirt.io
    jsonPointers:
      - /spec/versions

  # Prometheus operator relabels everything it scrapes
  - group: monitoring.coreos.com
    kind: PodMonitor
    jsonPointers:
      - /metadata/labels
      - /metadata/annotations
  - group: monitoring.coreos.com
    kind: ServiceMonitor
    jsonPointers:
      - /metadata/labels
      - /metadata/annotations

Every entry in that list is a bug I chased once. The KubeVirt CDI one is the clearest example: the operator rewrites /spec/versions on its own CRD after install, ArgoCD calls the Application OutOfSync forever, and nothing is actually wrong.

ServerSideApply=true earns its keep for a related reason. Some CRDs, and the Prometheus operator’s are the usual offenders, are too large to fit in the client-side apply annotation and fail with metadata.annotations: Too long. SkipDryRunOnMissingResource=true covers the bootstrap case on the infra appset, where a manifest references a CRD that the same sync is about to install.

The Honest Part

None of this is backported. The other six environments are still running selfHeal: true with the stock comment attached, not because I decided they should, but because they have not hurt me yet. Dev and edge clusters get rebuilt often enough that a revert war would be noise rather than an outage. That is a reason, but it is a retroactive one.

What This Buys Me

Gotchas

Outstanding Work

1. Backport the metal sync policy. Dev, prod, talos-03, talos-edge and staging still run the stock selfHeal: true with the docs comment attached; they should inherit the metal settings before they earn them the hard way
2. Consolidate the per-env appset pairs. A single appset with envs/* and the env name extracted as a template variable could replace most of the files in appsets/
3. Multi-cluster destinations. Everything currently targets kubernetes.default.svc from a per-cluster ArgoCD; a single management ArgoCD pushing to remote clusters is the other valid shape, and I keep going back and forth on it
4. Notifications. A failed sync currently announces itself by an app being broken

References

• https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/
• https://github.com/kostis-codefresh/many-appsets-demo