Overview

In my pursuit to learn more and migrate my existing Proxmox-based infrastructure to Kubernetes, I faced the issue of physical redundancy. While having three nodes does provide failover there are still other single points of failure, primarily my physical network topology. I have previously been able to work with L2 RSTP in Proxmox using Open vSwitch to add redundancy as well as mixing and prioritizing network port speeds. Talos Linux and Cilium did not seem willing to participate in RSTP, so I needed to resort to other methods, Link Aggregation was my next approach. But I was quickly dissuaded by LACP’s inability to bond ports of unlike speed and Cilium’s eBPF binding to the raw interface instead of the virtual Bond. The next idea on the list that I could think of was a full Layer3 Routed Spine-Leaf topology setup.

Planning

Recently acquiring another 10GB capable Mikrotik switch, I finally had a proper dual uplink from each of my physical servers to the core network and ability to maintain cluster connectivity during switch maintenance.

The following is an overview of the physical network topology utilizing Mikrotik Network and Talos + Cilium Nodes.

graph TB
    subgraph "Edge / WAN"
        R1["R1 Edge Router
AS 65534
2001:db8:abcd:e00a::5009
2001:db8:abcd:e00b::5009"]
    end

    subgraph "Spine Layer"
        TOR1["TOR1 Spine RR
AS 65534
Router-ID: 0.3.0.9
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00a::1/64"]
        TOR2["TOR2 Spine RR
AS 65534
Router-ID: 0.3.5.4
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00b::1/64"]
    end

    subgraph "Kubernetes Nodes"
        Node1["Node 1
e0b0::/64 Pod Subnet"]
        Node2["Node 2
e0b1::/64 Pod Subnet"]
        NodeX["..."]
        NodeN["Node N
e0bf::/64 Pod Subnet"]
    end

    R1 -. iBGP + Default Originate .-> TOR1
    R1 -. iBGP + Default Originate .-> TOR2

    TOR1 -- iBGP RR --> Node1
    TOR1 -- iBGP RR --> Node2
    TOR1 -- iBGP RR --> NodeN

    TOR2 -- iBGP RR --> Node1
    TOR2 -- iBGP RR --> Node2
    TOR2 -- iBGP RR --> NodeN

    style R1 fill:#1e3a5f,stroke:#4fc3f7,stroke-width:2px,color:#fff
    style TOR1 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style TOR2 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style Node1 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style Node2 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style NodeN fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff

Mikrotik

One of the first hurdles was understanding Mikrotik’s L3 Hardware Offloading capabilities and how to properly address and logically configure each of the switches. Until this point, I had avoided using switches to route due to their poor IPv6 compatibility or due to the throughput limitations of the hardware. Finding proper offloading capability that also supports IPv6 and not just IPv4 in recent RouterOS firmware has been exciting. In the end both Mikrotik’s hardware offloading may not see a lot of use as they are acting as BGP route reflectors.

Cilium

Previously I had already succeeded in using Cilium’s Native Routing and a BGP connection to my WAN Router for full routing. Adding the Multi-Homed aspect was to fix the issue with inter-node communications during WAN router interruptions, whether from the router itself rebooting or intermediate switch failures. While working through Cilium’s support for link-local BGP connections, I had to wrap my head around managing host node connectivity as well as Cilium containers using the clusterPodNets subnets to communicate. The first issue was the bootstrap paradox: adding a loopback interface in Talos that could still be reachable before Cilium and BGP were able to peer and advertise the loopback IP.

Summary

There are still improvements to be made and testing to be done on how tolerant this setup is to losing its physical uplinks. I have already found that the BGPPeerConfig timers need to be tweaked and research into possibly implementing BGP BFD for faster link failure detection.

Notes

Addressing Plan:

20├│││├││└0───1222:0├├└0├└0├├├└d0───0─20────b1222120122.28:000:00:00.0:d000d01d00.0ab111b1:b111b8:::8:d8:::c:ddd:db:ddddabbbab8abbb:b888b8:b888ec:::c:ac:::0daaadabdaaa0:bbb:bc:bbb0ecccecdeccc:0ddd0d:0ddd:0:::a:eb:::/0eee0e00eee5:000:0a:0006:000:a1:bbb/0ab/0:/01f6:::6::6:::0:::0:0:::///6///-666-14-6664440444MC8-cG---Ll---MU-CuTNTTSisNNNOOOTcltoooDRREliedddE12RuureeesmPlto12noeLdoroNPPPp_aeooobsdtdddaeBscraSSSkvluuuiabbbcnnnneceeeSettturbnIePtPsool

Configuring TOR1 Spine Switch

# Enable L3 Hardware offloading if available
/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

# Add Routed Interface and static address on TOR1
/interface/vlan add interface=bridge vlan-id=901 name=901-Spine
/ipv6/address add address=2001:db8:abcd:e00a::1 advertise=yes interface=901-Spine
/ipv6/nd set [ find default=yes ] advertise-dns=yes dns=2606:4700:4700::64,2606:4700:4700::6400 hop-limit=64 other-configuration=yes ra-delay=0s
/ipv6/nd add advertise-dns=yes interface=901-Spine other-configuration=yes

# Configure iBGP Route Reflector
/routing/bgp/instance add as=65534 name=spine-rr router-id=0.3.0.9 cluster-id=5.0.0.9

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-template

# Add iBGP Connection for dynamic listener for K8s nodes on VLAN901-Spine
/routing/bgp/connection add instance=spine-rr local.role=ibgp-rr listen=yes local.address=2001:db8:abcd:e00a::1 name=nodes-dynamic remote.address=2001:db8:abcd:e00a::/64 template=spine-template

Configuring TOR2 Spine Switch

# Enable L3 Hardware offloading if available
/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

# Add Routed Interface and static address on TOR2
/interface/vlan add interface=bridge vlan-id=902 name=902-Spine
/ipv6/address add address=2001:db8:abcd:e00b::1 advertise=yes interface=902-Spine
/ipv6/nd set [ find default=yes ] advertise-dns=yes dns=2606:4700:4700::64,2606:4700:4700::6400 hop-limit=64 other-configuration=yes ra-delay=0s
/ipv6/nd add advertise-dns=yes interface=902-Spine other-configuration=yes

# Configure iBGP Route Reflector
/routing/bgp/instance add as=65534 name=spine-rr router-id=0.3.5.4 cluster-id=5.0.0.9

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-template

# Add iBGP Connection for dynamic listener for K8s nodes on VLAN902-Spine
/routing/bgp/connection add instance=spine-rr local.role=ibgp-rr listen=yes local.address=2001:db8:abcd:e00b::1 name=nodes-dynamic remote.address=2001:db8:abcd:e00b::/64 template=spine-template

Configure R1 Edge Leaf WAN Router

# Add routed interface and static address to both Spine Switches
/interface/vlan add interface=bridge vlan-id=901 name=901-Spine
/interface/vlan add interface=bridge vlan-id=902 name=902-Spine
/ipv6 address add address=2001:db8:abcd:e00a::5009 advertise=no interface=901-Spine
/ipv6 address add address=2001:db8:abcd:e00b::5009 advertise=no interface=902-Spine

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-connection-template
/routing/bgp/instance add as=65534 name=edge router-id=5.0.0.9

# Add iBGP Connection to both Spine Switches
/routing/bgp/connection add instance=edge local.address=2001:db8:abcd:e00a::5009 .role=ibgp name=to-TOR1 output.default-originate=if-installed remote.address=2001:db8:abcd:e00a::1 template=spine-template
/routing/bgp/connection add instance=edge local.address=2001:db8:abcd:e00b::5009 .role=ibgp name=to-TOR2 output.default-originate=if-installed remote.address=2001:db8:abcd:e00b::1 template=spine-template

Cilium BGP Configuration

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: cluster-v6-pool
spec:
  blocks:
    - cidr: 2001:db8:abcd:e0a1::/64
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  bgpInstances:
    - localASN: 65534
      name: instance-65534
      peers:
        - name: peer-tor1
          peerASN: 65534
          peerAddress: 2001:db8:abcd:e00a::1 # TOR1 Spine Router VLAN901
          peerConfigRef:
            name: cilium-peer
        - name: peer-tor2
          peerASN: 65534
          peerAddress: 2001:db8:abcd:e00b::1 # TOR2 Spine Router VLAN902
          peerConfigRef:
            name: cilium-peer
  nodeSelector:
    matchLabels:
      kubernetes.io/arch: amd64
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  families:
    - advertisements:
        matchLabels:
          advertise: bgp
      afi: ipv6
      safi: unicast
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  timers:
    holdTimeSeconds: 9
    keepAliveTimeSeconds: 3
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  labels:
    advertise: bgp
  name: bgp-advertisements
spec:
  advertisements:
    - advertisementType: Service
      selector:
        matchExpressions: []
      service:
        addresses:
          - ClusterIP
          - LoadBalancerIP
    - advertisementType: CiliumPodIPPool
    - advertisementType: PodCIDR
    - advertisementType: Interface
      interface:
        name: dummy0