The Problem

I run Hermes - a persistent, tool-using AI agent - inside my Kubernetes cluster. It helps me operate the homelab: check on failing pods, read ArgoCD app status, dig through CloudNativePG backups, tell me why a node is unhappy. To do any of that it needs kubectl, cluster credentials, and network reach.

Which means I deliberately deployed a curious, tool-wielding, LLM-driven process inside my cluster, gave it a service account, and then - because apparently I like living dangerously - gave my friends their own instances.

An agent in a pod is a new kind of workload: it’s not malicious, but it’s creative, it follows instructions from whoever is chatting with it, and it will happily try things no static application ever would. Treat it like a permanent, well-meaning intruder and the security design falls out naturally. This post is that design - RBAC first, then network - and the friendly-fire incident that stress-tested it.

The Deployment

Nothing exotic: a Deployment per user in the hermes namespace, config from a ConfigMap, persistence on a PVC, HTTPRoute on the private gateway. I used to maintain a custom :slim build - unused features stripped out, but with the DevOps toolchain (kubectl, helm, talosctl, argocd, kubeseal) baked in so the agent could actually do anything. Once I got those tools installing themselves onto the persistent volume at ~/.local/bin instead (already on PATH), the custom build wasn’t worth maintaining anymore - there’s one image now, and tool versions survive image updates on their own.

The interesting part is what the pod is allowed to do.

The ServiceAccount: Look, Don’t Touch

The agent runs as a dedicated ServiceAccount, bound to the built-in view ClusterRole plus a supplementary role for everything view deliberately excludes:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hermes-ops-extra
rules:
  # view deliberately omits nodes and PVs
  - apiGroups: [""]
    resources: ["nodes", "persistentvolumes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["nodes", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["gateway.networking.k8s.io"]
    resources: ["gateways", "httproutes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["argoproj.io"]
    resources: ["applications", "applicationsets", "appprojects"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["cilium.io"]
    resources: ["ciliumclusterwidepolicies", "ciliumnetworkpolicies", "ciliumidentities"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["velero.io", "postgresql.cnpg.io", "kubevirt.io"]
    # ... backups, clusters, VMs - all get/list/watch

Every rule in that ClusterRole is read-only - mostly get/list/watch, plus a couple of get-only exceptions for health-check endpoints. The pattern:

The agent diagnoses; git remediates. Hermes can see everything - nodes, metrics, ArgoCD sync states, network policies, backup schedules - so it can answer “why is Immich down?” with actual evidence. But the fix it proposes comes back to me as a suggestion, and the change goes through the GitOps repo like any other change. There is no verb in that ClusterRole that mutates anything, so a prompt-injected or simply over-enthusiastic agent can’t delete a deployment, exec into a pod, or read a Secret.

That last one matters more than it looks: view excludes secrets, and so does my supplementary role. An agent that can read Secrets is an agent that can become anything else in the namespace.

Reaching Other Clusters, Narrowly

Hermes also monitors my legacy cluster (talos-02) via its API. That path goes through a Tailscale operator egress proxy, and the proxy pod is itself locked down so only Hermes pods can use it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: talos-02-api-restrict
  namespace: hermes
spec:
  podSelector:
    matchLabels:
      tailscale.com/proxy: "true"
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: hermes
      ports:
        - port: 443
          protocol: TCP

Without this, anything in the namespace could ride the tailnet to another cluster’s API server. With it, the proxy is a private door with a short guest list.

Then My Friends Started Red-Teaming Me

Here’s where theory met practice. Once friends had their own Hermes instances, the obvious game began: use the agent to attack the cluster from the inside. Ask it to scan the pod network. Ask it to curl the neighbors. Ask it what it can reach.

This instance didn’t have API server access to begin with, so RBAC wasn’t what got tested that day. The interesting failures were in the network. A default Kubernetes cluster is a flat network: any pod can open a connection to any other pod, in any namespace. My NetworkPolicies existed where I’d thought about them (the Tailscale proxy above, tenant namespaces) - but “where I’d thought about it” is precisely the wrong coverage model against an attacker who can enumerate everything and gets bored never.

An in-cluster agent can find every unauthenticated dashboard, every database that trusts its network position, every metrics endpoint, every operator webhook. Mine did, cheerfully, at a friend’s request.

The Fix: Default-Deny, For Real

The answer was flipping the model: instead of policies where I remembered to add them, Cilium’s policy-enforcement-mode: always - every pod is default-denied unless a CiliumNetworkPolicy explicitly allows its traffic. Access is scoped per instance, not blanket per namespace: every Hermes pod gets DNS and its LLM backend; my own instance also gets the gateway and the Tailscale proxy to talos-02. Other instances don’t have that path at all. Every other namespace states its dependencies or loses them.

Rolling that out was its own adventure, and the lessons apply to any cluster going default-deny:

The GitOps chicken-and-egg. Enforcement mode always on a fresh cluster default-denies ArgoCD itself - which can’t pull the network policies that would un-break it, because DNS is denied. The baseline CNPs (kube-system/CoreDNS, ArgoCD) have to be applied by the bootstrap tooling, outside GitOps, and kept byte-identical to the copies ArgoCD later manages so the two converge instead of fighting.

DNS is always the first casualty. Two separate lessons:

• Cilium’s DNS proxy intercepts port-53 egress rules; under some configurations that interception caused UDP timeouts. Broad toEntities: cluster egress for DNS (no port restriction) sidestepped the proxy where it misbehaved.
• fromEndpoints: {} does not mean “everything in the cluster” the way you’d hope - an empty selector on a namespaced CiliumNetworkPolicy only matches endpoints in that same namespace, so CoreDNS’s policy in kube-system was silently blocking every other namespace’s pods, not just distant nodes. fromEntities: cluster is the correct spelling for “the whole cluster, any namespace,” plus a return-path egress rule for the replies.

Every app becomes an inventory exercise. Default-deny forces you to actually know each workload’s dependencies: the LLM proxy needs plain internet egress, the Postgres operator needs its backup plugin, the backup jobs need S3/object storage egress. Tedious for a week; documentation-grade knowledge of your own traffic forever after.

What the Agent Threat Model Taught Me

The uncomfortable general truth: an AI agent is the best network policy auditor I’ve ever had, because it attacks like an attacker but files a report like a colleague. If you run one, assume everyone who can talk to it can try everything it can try.

What Bit Me

Outstanding Work

1. The container still runs as root - runAsUser: 0 is a scar from filesystem permissions on the PVC; fixing the UID story is overdue
2. DNS-aware egress policies - Cilium toFQDNs allowlists for the LLM APIs instead of broad world egress
3. Hubble alerting - denied-flow spikes from the hermes namespace should page me; right now the red team gets free retries
4. Scheduled red-team sessions - the friends found real gaps; that should be a recurring event, not an accident
5. An agent per teammate - the bigger goal. A scoped agent like this one is a safe on-ramp to Kubernetes for people who have never touched kubectl: ask the cluster questions, learn how it fits together, ship changes as merge requests. Rolling that out to a small ops team deserves a post of its own once there are real stories to tell

References

• https://github.com/SirMysterion/hermes-agent
• https://docs.cilium.io/en/stable/security/policy/intro/
• https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles
• https://tailscale.com/kb/1236/kubernetes-operator