The Problem

I run three Kubernetes environments: a bare-metal IPv6 homelab, an on-prem Talos cluster on Proxmox at work, and a production EKS cluster (one repo drives the work pair). Every app in all three exposes itself the same way: an HTTPRoute attached to a Gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
    - name: the-gateway
      namespace: gateway
  hostnames:
    - "myapp.example.com"
  rules:
    - backendRefs:
        - name: myapp
          port: 8080

That manifest is genuinely portable. What is not portable is everything below it: how a packet from the internet actually arrives at the gateway. That’s what this post is about. Three environments, three completely different answers, zero changes to the apps.

The Common Layer

All three environments share:

• Gateway API (not Ingress). One Gateway per environment terminates TLS on 443, allowedRoutes: from: All
• cert-manager with a DNS-01 ClusterIssuer holding a wildcard cert per environment. DNS-01 because half these hostnames don’t resolve publicly, so HTTP-01 was never an option
• external-dns watching gateway-httproute sources. A new HTTPRoute means DNS records appear on their own

The app teams’ (and my own) contract is: ship an HTTPRoute, get a working HTTPS hostname. Everything else is the environment’s problem.

Environment 1: Homelab, Cilium Gateway + BGP

The homelab is the maximalist answer. Cilium implements Gateway API natively, the Gateway’s LoadBalancer IP comes from a CiliumLoadBalancerIPPool, and that IPv6 address is advertised via BGP to the Mikrotik spines (the multihomed saga). Any node can attract the traffic; the fabric ECMPs it; a node failure converges in about a second courtesy of BFD.

There are two Gateways, private and public, which is really a DNS story (covered next week in the split-horizon post): same wildcard, different external-dns instances, and “make this app public” is a one-line parentRefs change.

Character: the network is the project. BGP-advertised service IPs are the whole point.

Environment 2: Work On-Prem, Cilium Gateway + L2 Announcements

Same Cilium GatewayClass, same Gateway shape, but the LoadBalancer IP is announced with Cilium L2 announcements instead of BGP:

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-pool
spec:
  blocks:
    - cidr: 192.0.2.64/28   # a small block from the node VLAN

L2 announcements are the pragmatic on-prem answer when you don’t own the routers: Cilium replies to ARP for the pool IPs on the node VLAN, no router configuration, no BGP peering, works on any flat network. The trade-off is that failover rides on gratuitous ARP (slower and less deterministic than BFD-tracked routes) and one node carries each IP at a time, which is fine for this cluster’s size and a fraction of the moving parts.

One wrinkle worth stealing: the gateway’s pool IP is RFC1918, but the public DNS record must point at the site’s NAT’d public address. external-dns handles this with a target override on the Gateway:

metadata:
  annotations:
    external-dns.alpha.kubernetes.io/target: 203.0.113.10   # the site's public IP

Records for every HTTPRoute get created pointing at the public IP, the edge router port-forwards 443 to the pool IP, and nobody maintains DNS by hand.

Character: simplest thing that works on someone else’s flat network.

Environment 3: EKS, Traefik Gateway Behind One Shared NLB

The cloud answer looks the most different, and it took three tries to get there.

The requirements were more than “HTTPS in”:

Attempt 1: ALB Ingress + ACM. The AWS-native path. Worked, but every Ingress wants its own ALB, certs move into ACM (a special AWS thing instead of cert-manager like everywhere else), and Gateway API it is not.

Attempt 2: Cilium Gateway API. The consistency play, same GatewayClass as on-prem. It fell short of two hard requirements: with Cilium in chaining mode (AWS VPC CNI owns IPAM/routing, Cilium does policy), the Gateway dataplane couldn’t do a dual-stack LoadBalancer, and mixing protocols (TCP + UDP listeners) on one LB wasn’t happening either. Chaining mode buys policy portability; it does not buy the full Cilium dataplane.

Attempt 3: Traefik on the Gateway API experimental channel, one Service, one NLB:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: traefik
spec:
  controllerName: traefik.io/gateway-controller

The single-NLB trick is all in the AWS Load Balancer Controller annotations on Traefik’s Service:

apiVersion: v1
kind: Service
metadata:
  name: traefik
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
    service.beta.kubernetes.io/aws-load-balancer-name: <one-nlb-to-rule-them-all>
    service.beta.kubernetes.io/aws-load-balancer-group-name: <cluster-group>
spec:
  type: LoadBalancer

aws-load-balancer-type: external hands the Service to the LBC instead of the legacy in-tree controller, nlb-target-type: ip sends traffic straight to pod IPs (no NodePort hop), and the group-name means any future Services share this NLB instead of spawning their own. One dual-stack NLB, flat cost, every listener behind it.

The layering:

client (v4 or v6)
    → NLB (dual-stack, L4 passthrough, the only LB)
        → Traefik pods (Gateway API: HTTPRoutes terminated, TLSRoutes passed through)
            → app pods (single-stack v4)

HTTPS terminates at Traefik with cert-manager wildcards rather than ACM, so certs work identically in all three environments. The mesh VPN endpoints ride the same NLB as TLS passthrough listeners.

One more trick worth stealing: the per-tenant TLS listeners aren’t defined in the central gateway manifest. Each tenant’s kustomization owns its own listener entries on the shared Gateway via Server-Side Apply field ownership. The infra app manages the base Gateway, tenant apps manage their listeners, and ArgoCD doesn’t fight over the merge. Onboarding a tenant touches zero infra files.

Character: let AWS do what AWS is good at (the edge), pay for exactly one of it, keep everything above it portable.

The Comparison

Three philosophies in one table: BGP where I own the network, ARP where I’m a guest on it, NLB where the network is an API. The bottom row is the one that matters.

Rough Edges

Outstanding Work

1. Rate limiting / WAF at the gateways. Traefik middlewares make this easy on EKS; the Cilium gateways have no equivalent yet
2. TLSRoute on the Cilium environments. EKS has passthrough listeners; the on-prem gateways still assume HTTPS-only
3. Gateway API conformance drift. Two providers, three release cadences; Renovate updates them but conformance nuances occasionally differ

References

• https://gateway-api.sigs.k8s.io/
• https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/
• https://doc.traefik.io/traefik/providers/kubernetes-gateway/
• https://kubernetes-sigs.github.io/aws-load-balancer-controller/