The Problem

A platform I built runs from one GitOps repo: directory conventions, ApplicationSets, kustomize overlays per environment. It started on-prem: a Talos cluster on Proxmox VMs, the staging cluster that (as staging clusters do) quietly ended up hosting production workloads. Then the platform needed real cloud capacity, which meant an EKS cluster, and the question became: does the cloud cluster get its own tooling, or does it become just another envs/ directory?

Spoiler: it’s just another directory. What started as a dev/temporary cloud cluster has since graduated to running production workloads, and the promotion required no re-architecture, because it was never a special snowflake to begin with. The interesting part is what has to swap out underneath, and how much doesn’t.

Why EKS, and Not Talos on EC2?

Given everything on-prem runs Talos, the consistent move would have been Talos AMIs on EC2 and one OS everywhere. I went the other way, deliberately.

Self-managing Talos on EC2 means owning an image pipeline, control-plane upgrades, etcd health, and hand-rolling the integrations AWS gives EKS for free: IAM auth, load balancer wiring, EBS CSI. That’s real ongoing work, and none of it is the product. EKS makes the control plane someone else’s pager for ~$75/month, and the managed node groups turn capacity into a Terraform variable.

The reason this doesn’t hurt portability: the layers I actually care about keeping identical, meaning GitOps, network policy, secrets, DNS, and the apps, live above the substrate, and they’re already proven portable by the swap table below. Talos is my answer when I own the metal. When someone else owns the metal anyway, paying them to also own the Kubernetes control plane is the same logic applied one layer up.

The On-Prem Side: Talos on Proxmox

The incumbent is a small Talos cluster on Proxmox VMs, running the same talhelper workflow as my homelab metal cluster, minus the bare-metal firmware adventures: create VMs from the Talos ISO with the QEMU guest agent, talhelper genconfig, apply, bootstrap.

The stack on top:

Worth pausing on storage: the VMs already sit on a Proxmox host with a Ceph cluster, so instead of running storage inside Kubernetes, Ceph CSI mounts RBD volumes from the hypervisor’s own storage layer. The cluster stays stateless-ish; the data lives where the disks actually are.

The Cloud Side: Two-Stage Terraform Bootstrap

The EKS bootstrap is deliberately split into two Terraform roots:

bootstrap/eks/
  01-infra/     # VPC + EKS cluster + node groups
  02-argocd/    # ArgoCD Helm release + root app + repo credentials

Why not one? Because the kubernetes/helm providers in stage 2 need a cluster that exists to plan against. Keeping them separate avoids the classic “provider config depends on a resource in the same plan” knot, and means terraform destroy on the cluster doesn’t have to unwind Helm releases first.

Stage 1 highlights:

Stage 2 installs ArgoCD via the Helm provider, applies the root Application pointing at appsets/, and registers the cluster with its environment label:

cd bootstrap/eks/01-infra && terraform init && terraform apply
aws eks update-kubeconfig --region <region> --name <cluster>
cd ../02-argocd && terraform init && terraform apply

From terraform apply to fully populated cluster is one coffee: ArgoCD comes up, reads the appsets, sees a cluster labeled environment: eks, and deploys everything matching infra/*/envs/eks and apps/*/envs/eks.

Yes, IPv4. From Me.

Readers of the homelab posts will notice something: my homelab runs IPv6-only pods with BGP to the spine switches, and both of these work clusters run plain single-stack IPv4 pods. That’s not an accident, it’s a priority call. At home, the network is the project; at work, the network is plumbing under the actual product, and boring plumbing is a feature.

IPv6 still exists where it earns its keep, at the edges. Cloudflare fronts the on-prem services for v6 clients, and the EKS ingress is a dual-stack NLB. The dual-stack choice also lives in the VPC itself, because that’s the layer you cannot retrofit onto subnets full of workloads without pain. Pod-level IPv6 can wait until something needs it; until then, every AWS integration, every debugging session, and every teammate’s mental model stays simple.

Same Repo, Swapped Foundations

The applications don’t care where they run. The infrastructure layer under them is where the two environments diverge, and the envs/ convention absorbs all of it:

(The ingress row deserves its own post, and will get one: Gateway API looks identical to the apps across all three of my environments while the underlying load balancing is BGP at home, L2 announcements on-prem at work, and an NLB in the cloud.)

Each row is just a different directory existing or not: infra/aws-lbc/envs/eks/ exists, infra/aws-lbc/envs/onprem/ doesn’t. The ApplicationSet generator does the rest. No if cloud then logic anywhere; the filesystem is the feature flag.

Keeping Cilium on both sides (native on-prem, chaining mode on EKS) was deliberate: the CiliumNetworkPolicies that enforce tenant isolation and default-deny (the Hermes lesson) are the same manifests in both environments. Policy is the layer I least want forked per cloud.

What Stays Identical

This is the payoff list: everything that needed zero changes between on-prem and EKS.

• ArgoCD itself and the root-app bootstrap
• cert-manager with DNS-01 issuers
• external-dns (same chart, same HTTPRoute sources)
• External Secrets + Vaultwarden; the same ExternalSecret manifests resolve on both clusters
• K8up backups to S3-compatible storage
• CloudNativePG and the databases on top of it
• CiliumNetworkPolicies; same tenant isolation, both environments
• The applications themselves, including their kustomize bases
• Renovate keeping every image and chart current across both

The mental model that emerged: treat every substrate as a hardware vendor. Proxmox sells me VMs with Ceph disks; AWS sells me VMs with EBS disks and a fancier API. Everything above that line is portable, and the GitOps layer genuinely cannot tell the difference.

The Fine Print

Outstanding Work

1. Multi-AZ. The EKS cluster is single-AZ today, a deliberate cost/complexity trade that needs revisiting as workloads grow
2. IRSA everywhere. Pod-level IAM for external-dns and the LB controller instead of node roles
3. Cost visibility. OpenCost or Kubecost in the envs/eks tree

References

• https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest
• https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/
• https://kubernetes-sigs.github.io/aws-load-balancer-controller/
• https://github.com/ceph/ceph-csi