The Problem

I’ve been building a platform that runs the same application for multiple clients, with multiple users per client. Every instance needs its own config, its own storage, and its own hostname. Client A must never be able to reach client B, and that one has to be a guarantee rather than a convention. Users inside a client are a softer boundary, which is a distinction worth making early because the design treats them very differently. The naive approach is copy-pasting a pile of YAML per tenant, which works right up until you need to change one field across twenty instances.

Helm is the usual answer, but I wanted to stay in plain kustomize + ArgoCD like the rest of my infrastructure. It turns out kustomize’s namePrefix plus ApplicationSet git generators get you surprisingly far.

The Directory Layout

The whole design is a filesystem convention:

clients/
  _shared/
    agent/                      # kustomize base: the template
      deployment.yaml
      service.yaml
      pvc.yaml
      cilium-networkpolicy.yaml
  acme/                         # a client
    agent/
      envs/
        staging/
          namespace.yaml         # NS: acme-agent
          kustomization.yaml     # composes all users + routes
          httproute-alice.yaml
          alice/                 # a user instance
            kustomization.yaml   # namePrefix + instance label, resources from _shared
            configmap.yaml       # alice's app config
            secret.yaml          # gitignored, applied imperatively
  bigcorp/
    agent/
      envs/
        staging/
          ...

_shared/agent/ is the single source of truth for what an instance looks like. A user overlay is tiny:

# clients/acme/agent/envs/staging/alice/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: alice-

resources:
  - ../../../../../_shared/agent/deployment.yaml
  - ../../../../../_shared/agent/service.yaml
  - ../../../../../_shared/agent/pvc.yaml
  - ../configmap-base.yaml
  - configmap.yaml

labels:
  - pairs:
      instance: alice
    includeSelectors: true

namePrefix turns the shared agent Deployment/Service/PVC into alice-agent, alice-agent-data, and so on. Twenty users is twenty small overlay directories, all pointing at one template. Change the template, every tenant picks it up on the next sync.

The labels block is the part that took me longest to get right, and it is the one thing you cannot leave out. namePrefix renames objects but does not touch pod labels or selectors. Without includeSelectors: true, every user’s Service still selects app: agent, which means alice’s hostname happily load-balances across bob’s pods. Adding instance: alice to both the pod template and the selector is what actually keeps one user’s traffic on one user’s pods.

Note that the overlay lists individual files from _shared rather than the directory. That is deliberate: _shared/agent/kustomization.yaml is a kustomize Component, and pointing resources at the directory pulls it in the wrong way.

One ApplicationSet for All Tenants

The ArgoCD side is a single ApplicationSet with a two-level wildcard:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: staging-clients-appset
  namespace: argocd
spec:
  ignoreApplicationDifferences:
    - jsonPointers:
        - /spec/syncPolicy
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: <repo>
              revision: HEAD
              directories:
                - path: clients/*/*/envs/staging
          - clusters:
              selector:
                matchLabels:
                  argocd.argoproj.io/secret-type: cluster
                  environment: staging
  template:
    metadata:
      name: '{{index .path.segments 1}}-{{index .path.segments 2}}'
    spec:
      project: default
      source:
        repoURL: <repo>
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{index .path.segments 1}}-{{index .path.segments 2}}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        automated:
          prune: true
          selfHeal: true

goTemplate: true is not optional here. Without it the {{index .path.segments 1}} syntax silently does nothing useful, because ArgoCD falls back to its older fasttemplate engine. The generator and cluster-selector mechanics are the same ones I wrote about last week.

clients/acme/agent/envs/staging becomes Application acme-agent in namespace acme-agent. Onboarding a client is mkdir + push; offboarding is git rm and prune cleans up the namespace.

The namespace is the tenant boundary. One namespace per client-app pair keeps RBAC, quotas, and network policy all aligned to the same edge.

Network Isolation with Cilium

Namespaces alone don’t isolate anything. By default every pod can reach every other pod in the cluster. Each tenant namespace gets a CiliumNetworkPolicy that allows exactly three things: ingress from the Gateway, DNS, and the specific shared services the app needs:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: agent-network-policy
spec:
  endpointSelector:
    matchLabels:
      app: agent
  ingress:
    # Gateway traffic arrives with Cilium's reserved "ingress" identity,
    # not as a pod in whatever namespace the Gateway resource lives in.
    - fromEntities:
        - ingress
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: shared-llm-proxy
      toPorts:
        - ports:
            - port: "4000"
              protocol: TCP
    # Public internet yes, anything private no.
    - toCIDRSet:
        - cidr: 0.0.0.0/0
          except:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 100.64.0.0/10
            - 127.0.0.0/8
            - 169.254.0.0/16

That fromEntities: ingress line cost me an afternoon. The Gateway resource lives in its own namespace, so the obvious rule is to allow that namespace, and the obvious rule does not work. Cilium’s Gateway implementation proxies through Envoy, and the traffic shows up carrying the reserved ingress identity instead of a pod identity.

The toCIDRSet block is doing the real security work. Allowing all of 0.0.0.0/0 while excluding RFC1918, CGNAT, loopback and link-local means a tenant pod can reach the public internet but cannot reach another namespace, another node, or the cloud metadata endpoint. Client-to-client traffic isn’t blocked by a specific rule. It is simply never allowed, because a Cilium endpoint becomes default-deny the moment any policy selects it, and nothing in this policy ever names another tenant.

One honest caveat: because the policy selects app: agent and every instance in the namespace carries that label, this isolates clients from each other, not users within a client. Alice and Bob share a namespace and a network policy. The instance label keeps their traffic routed correctly, but it is not a security boundary, and if a client ever needs their users mutually isolated the namespace has to move down a level.

Where the Pattern Breaks Down

Everything above describes _shared as the single source of truth, and for the Deployment, the Service and the PVC that is exactly what it is. The network policy is the one resource where it isn’t.

There is a policy in _shared, and no client actually references it. Each client env has its own copy instead, six of them now, differing from the shared version by twenty to thirty lines apiece. The divergence is real rather than accidental: one client fronts the app with a Teams router, another adds a web auth router, and each of those needs its own ingress rule. Copying the file and editing it took a minute; expressing the same thing as a base plus per-client patches would have taken an afternoon, and I picked the minute.

The bill arrived later. The fromEntities: ingress fix from the previous section went into the client copies and never went into _shared, so the template is now the stale one. Anyone starting a new client from it inherits a policy that doesn’t work, and a fix applied to the shared file reaches nobody.

This is the honest limit of the convention. namePrefix and a shared base handle resources that are genuinely identical across tenants. The moment a resource needs per-tenant structure rather than per-tenant values, kustomize wants patches, and if you skip that step the shared base quietly stops being shared.

Per-User Routing

Each user gets an HTTPRoute at the client env level, mapping a hostname to their prefixed service:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alice-agent
spec:
  parentRefs:
    - kind: Gateway
      name: staging-gateway
      namespace: gateway-system
      sectionName: https
  hostnames:
    - alice.example-client-domain.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - kind: Service
          name: alice-agent
          port: 8080

The HTTPRoute sits at the client env level rather than inside the user overlay, which means namePrefix never touches it and alice-agent in that backendRefs is hand-written. Kustomize’s name reference transformer does not know about Gateway API CRDs, so if you change a prefix, the route quietly points at a Service that no longer exists.

external-dns watches the HTTPRoutes and publishes the records; cert-manager holds a wildcard cert on the Gateway. Adding a user never touches DNS or TLS by hand.

The Ugly Part: Secrets

Each user has a secret.yaml that is gitignored and applied imperatively at onboarding. This is the one part of the design that isn’t GitOps, and it’s a deliberate compromise: per-tenant credentials in git, even sealed, mean re-encrypting on every key rotation, and the blast radius of a mistake is a client’s credentials.

The longer-term answer is External Secrets with per-tenant paths in a secrets manager (next week’s post is a step in that direction).

Pitfalls

Outstanding Work

1. Secrets into a manager. Replace the imperative secret.yaml with External Secrets per tenant
2. Per-tenant resource quotas. Nothing currently stops one tenant from eating the node
3. Tenant usage metering. Namespace-level metrics exist in Prometheus, but need aggregating per client
4. User-level network isolation. Today the namespace is the client boundary and users inside it share a policy; if a client ever needs their users separated, that boundary has to move
5. Make the shared network policy an actual base. Port the ingress fix back into _shared and convert the six client copies into patches, before someone starts a new client from the stale template

References

• https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/nameprefix/
• https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/
• https://docs.cilium.io/en/stable/security/policy/