In my pursuit to learn more and migrate my existing Proxmox-based infrastructure to Kubernetes, I faced the issue of physical redundancy. While having three nodes does provide failover there are still other single points of failure, primarily my physical network topology. I have previously been able to work with L2 RSTP in Proxmox using Open vSwitch to add redundancy as well as mixing and prioritizing network port speeds. Talos Linux and Cilium did not seem willing to participate in RSTP, so I needed to resort to other methods, Link Aggregation was my next approach. But I was quickly dissuaded by LACP’s inability to bond ports of unlike speed and Cilium’s eBPF binding to the raw interface instead of the virtual Bond. The next idea on the list that I could think of was a full Layer3 Routed Spine-Leaf topology setup.

Planning

Recently acquiring another 10GB capable Mikrotik switch, I finally had a proper dual uplink from each of my physical servers to the core network and ability to maintain cluster connectivity during switch maintenance.

The following is an overview of the physical network topology utilizing Mikrotik Network and Talos + Cilium Nodes.

graph TB
    subgraph "Edge / WAN"
        R1["R1 Edge Router
AS 65534
2001:db8:abcd:e00a::5009
2001:db8:abcd:e00b::5009"]
    end

    subgraph "Spine Layer"
        TOR1["TOR1 Spine RR
AS 65534
Router-ID: 0.3.0.9
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00a::1/64"]
        TOR2["TOR2 Spine RR
AS 65534
Router-ID: 0.3.5.4
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00b::1/64"]
    end

    subgraph "Kubernetes Nodes"
        Node1["Node 1
e0b0::/64 Pod Subnet"]
        Node2["Node 2
e0b1::/64 Pod Subnet"]
        NodeX["..."]
        NodeN["Node N
e0bf::/64 Pod Subnet"]
    end

    R1 -. iBGP + Default Originate .-> TOR1
    R1 -. iBGP + Default Originate .-> TOR2

    TOR1 -- iBGP RR --> Node1
    TOR1 -- iBGP RR --> Node2
    TOR1 -- iBGP RR --> NodeN

    TOR2 -- iBGP RR --> Node1
    TOR2 -- iBGP RR --> Node2
    TOR2 -- iBGP RR --> NodeN

    style R1 fill:#1e3a5f,stroke:#4fc3f7,stroke-width:2px,color:#fff
    style TOR1 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style TOR2 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style Node1 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style Node2 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style NodeN fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff

Mikrotik

One of the first hurdles was understanding Mikrotik’s L3 Hardware Offloading capabilities and how to properly address and logically configure each of the switches. Until this point, I had avoided using switches to route due to their poor IPv6 compatibility or due to the throughput limitations of the hardware. Finding proper offloading capability that also supports IPv6 and not just IPv4 in recent RouterOS firmware has been exciting. In the end both Mikrotik’s hardware offloading may not see a lot of use as they are acting as BGP route reflectors.

Cilium

Previously I had already succeeded in using Cilium’s Native Routing and a BGP connection to my WAN Router for full routing. Adding the Multi-Homed aspect was to fix the issue with inter-node communications during WAN router interruptions, whether from the router itself rebooting or intermediate switch failures. While working through Cilium’s support for link-local BGP connections, I had to wrap my head around managing host node connectivity as well as Cilium containers using the clusterPodNets subnets to communicate. The first issue was the bootstrap paradox: adding a loopback interface in Talos that could still be reachable before Cilium and BGP were able to peer and advertise the loopback IP.

Summary

There are still improvements to be made and testing to be done on how tolerant this setup is to losing its physical uplinks. I have already found that the BGPPeerConfig timers need to be tweaked and research into possibly implementing BGP BFD for faster link failure detection.

Notes

Addressing Plan:

Configuring TOR1 Spine Switch

# Enable L3 Hardware offloading if available
/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

# Add Routed Interface and static address on TOR1
/interface/vlan add interface=bridge vlan-id=901 name=901-Spine
/ipv6/address add address=2001:db8:abcd:e00a::1 advertise=yes interface=901-Spine
/ipv6/nd set [ find default=yes ] advertise-dns=yes dns=2606:4700:4700::64,2606:4700:4700::6400 hop-limit=64 other-configuration=yes ra-delay=0s
/ipv6/nd add advertise-dns=yes interface=901-Spine other-configuration=yes

# Configure iBGP Route Reflector
/routing/bgp/instance add as=65534 name=spine-rr router-id=0.3.0.9 cluster-id=5.0.0.9

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-template

# Add iBGP Connection for dynamic listener for K8s nodes on VLAN901-Spine
/routing/bgp/connection add instance=spine-rr local.role=ibgp-rr listen=yes local.address=2001:db8:abcd:e00a::1 name=nodes-dynamic remote.address=2001:db8:abcd:e00a::/64 template=spine-template

Configuring TOR2 Spine Switch

# Enable L3 Hardware offloading if available
/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch/port set [find] l3-hw-offloading=yes

# Add Routed Interface and static address on TOR2
/interface/vlan add interface=bridge vlan-id=902 name=902-Spine
/ipv6/address add address=2001:db8:abcd:e00b::1 advertise=yes interface=902-Spine
/ipv6/nd set [ find default=yes ] advertise-dns=yes dns=2606:4700:4700::64,2606:4700:4700::6400 hop-limit=64 other-configuration=yes ra-delay=0s
/ipv6/nd add advertise-dns=yes interface=902-Spine other-configuration=yes

# Configure iBGP Route Reflector
/routing/bgp/instance add as=65534 name=spine-rr router-id=0.3.5.4 cluster-id=5.0.0.9

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-template

# Add iBGP Connection for dynamic listener for K8s nodes on VLAN902-Spine
/routing/bgp/connection add instance=spine-rr local.role=ibgp-rr listen=yes local.address=2001:db8:abcd:e00b::1 name=nodes-dynamic remote.address=2001:db8:abcd:e00b::/64 template=spine-template

Configure R1 Edge Leaf WAN Router

# Add routed interface and static address to both Spine Switches
/interface/vlan add interface=bridge vlan-id=901 name=901-Spine
/interface/vlan add interface=bridge vlan-id=902 name=902-Spine
/ipv6 address add address=2001:db8:abcd:e00a::5009 advertise=no interface=901-Spine
/ipv6 address add address=2001:db8:abcd:e00b::5009 advertise=no interface=902-Spine

# Add BGP Connection Templates
/routing/bgp/template add as=65534 name=spine-connection-template
/routing/bgp/instance add as=65534 name=edge router-id=5.0.0.9

# Add iBGP Connection to both Spine Switches
/routing/bgp/connection add instance=edge local.address=2001:db8:abcd:e00a::5009 .role=ibgp name=to-TOR1 output.default-originate=if-installed remote.address=2001:db8:abcd:e00a::1 template=spine-template
/routing/bgp/connection add instance=edge local.address=2001:db8:abcd:e00b::5009 .role=ibgp name=to-TOR2 output.default-originate=if-installed remote.address=2001:db8:abcd:e00b::1 template=spine-template

Cilium BGP Configuration

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: cluster-v6-pool
spec:
  blocks:
    - cidr: 2001:db8:abcd:e0a1::/64
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  bgpInstances:
    - localASN: 65534
      name: instance-65534
      peers:
        - name: peer-tor1
          peerASN: 65534
          peerAddress: 2001:db8:abcd:e00a::1 # TOR1 Spine Router VLAN901
          peerConfigRef:
            name: cilium-peer
        - name: peer-tor2
          peerASN: 65534
          peerAddress: 2001:db8:abcd:e00b::1 # TOR2 Spine Router VLAN902
          peerConfigRef:
            name: cilium-peer
  nodeSelector:
    matchLabels:
      kubernetes.io/arch: amd64
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  families:
    - advertisements:
        matchLabels:
          advertise: bgp
      afi: ipv6
      safi: unicast
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  timers:
    holdTimeSeconds: 9
    keepAliveTimeSeconds: 3
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  labels:
    advertise: bgp
  name: bgp-advertisements
spec:
  advertisements:
    - advertisementType: Service
      selector:
        matchExpressions: []
      service:
        addresses:
          - ClusterIP
          - LoadBalancerIP
    - advertisementType: CiliumPodIPPool
    - advertisementType: PodCIDR
    - advertisementType: Interface
      interface:
        name: dummy0

The Problem

The Network monitoring solutions that I have seen seem to be stuck in the past with IPv4 only networks, seemingly relying on ARP and Nmap scans of network blocks to find devices and services on the network. From what I have heard, a single IPv6 Subnet with a /64 prefix has some 18 quintillion Addresses. That does not include the extra 64 thousand other subnets in the /48 that likely is assigned to location

I have not done the math myself but scanning one address after another does not seem to be a good use of time. But good news! IPv6 neighbor discovery or ND tables do exist. I have just not found anyone else but routers/switches using them. So time to build my own with my good buddy Claude.

The Wish list:

• I want to know how many host clients are on my network.
• I want to be able to find IPv6 Addresses based on MAC addresses and MAC address based on IPv6
• I also would to know which port on my switches the device can be found.
  • This wont’ tell me down to the VM but it will tell me which VM Host to look at and which MAC Address to search for in the NIC config

The Partial Solution

LibreNMS has some nice out of the box features, adds SNMP Monitoring and a basic Network hardware neighbor map Oh and there some nice uptime and health metrics in there. MikroTik’s RouterOS does not expose IPv6 neighbor tables via SNMP, so some more work was required.

The Solution:

Built a custom stack combining:

• LibreNMS for base device/port discovery via SNMP
• Custom sync service polling MikroTik REST APIs every 5 minutes
  • /rest/ipv6/neighbor for IPv6↔MAC mappings from routers
  • /rest/interface/bridge/host for MAC↔Port mappings from switches
• Python web dashboard with real-time filtering and multiple view modes (IPv6, MAC-grouped)

Bonus Discovery:

During testing, found one switch had 28,000+ bridge table entries due to misconfigured VLAN trunking That can’t be right, am I getting mac flooding or a switching loop somewhere? Further investigation showed each of my 8 SFP+ ports on my CRS309 had ~4000 MAC addresses in the table. Turns out that for every configured VLAN on a port creates a mac address entry for the ports own mac address. 4000 Vlans, by 8 ports. That’s a lot of MAC Addresses.

/interface bridge vlan
add bridge=bridge tagged=all vlan-ids=2-3999

While Allowing VLANs 1-3999 to be tagged on the port makes it easy to trunk any VLAN I throw at it, I realistically am only using about 15 Once I sorted out the tagged VLANs to those in use, the host table went from 28,000 hosts to about 215. And my REST API calls having to generate that many hosts went from 40+ seconds to sub second queries.

Sometimes the best monitoring tools reveal configuration issues you didn’t know you had!

Tech Stack: Docker, Python, LibreNMS, MariaDB, MikroTik RouterOS v7+ REST API8

Outstanding Issues

There are still a couple of issues to resolve:

1. DNS64/NAT64

   • CoreDNS sometimes still gives out A records instead of DNS64 synthesized DNS records. As a result, some images fail to pull if they come from repositories with IPv4-only records.

2. IaC Deployment - Mostly Complete

   • There is a mix of Terraform, Helm, Kustomize, and ArgoCD. The homelab repo needs some cleanup and consolidation to streamline the deployment process.

3. Virtualized Compute

   • Currently, Talos Linux is running in VMs inside of Proxmox. Next steps include replacing Proxmox with bare-metal Talos Linux and looking into KubeVirt for any remaining virtualized workloads.

4. IPv4 Accessibility

   • Using Cloudflare’s proxy to serve IPv4-only clients from the IPv6-only ingress. This works but adds complexity and an external dependency when using anything but standard HTTP/HTTPS connections.

The Good News

But there is good news! We do have a full-stack Kubernetes cluster running. Backups have been tested, and my Immich app has been successfully restored (full documentation still needed).

LetsEncrypt Certificates and Gateway API have been working well! This website is a testament to that. (Disregard the Cloudflare SSL cert; I’m using their proxy to simplify serving IPv4-only clients from my IPv6-only ingress.)

What’s Next

Upcoming posts will dive deeper into:

• The Talos Linux experience and migration from Proxmox
• Gateway API configuration and best practices
• KubeVirt - For added virtual machine management
• Renovate - For Automated dependency updates
• Disaster recovery testing with Velero The infrastructure might not be perfect yet, but it’s running production workloads and improving. Stay tuned for more technical deep dives as I work through each challenge.

After some time away, I’ve decided to reboot the blog with a new infrastructure approach. Rather than relying on traditional hosting or managed services, I’ve moved everything to a self-hosted Kubernetes cluster. This gives me complete control over the deployment pipeline, scaling, and infrastructure management.

What’s New

• Self-Hosted Kubernetes: Running on my own hardware, giving me full control over the entire stack
• Modern CI/CD: Automated deployments with proper containerization
• Infrastructure as Code: Everything version controlled and reproducible

Stay tuned for more posts about the technical details of the setup, lessons learned, and future projects. The blog is back and ready to go!

Quite some time ago I was able to utilize this address space with My Ubiquity Router following [this guide.] (https://heald.ca/configuring-telus-optik-ipv6-ubiquiti-edgerouter/) Only recently I looked into My EdgeRouter’s v6 firewall rules. While lacking NETv6_interface address groups, the following rule allows for a Dynamic Prefix Destination rule for a EUI-64 address. The mask is modified limit it to the 5th Prefix or hex

set firewall ipv6-name IPv6-In rule 4 destination address ::5:224:2FFF:FE85:6E10/::00ff:ffff:ffff:ffff:ffff

Once I had figured out that the rule would work with a changing address the next problem was updating DNS. Using Cloudflare, it was easy enough to find scripts to update the entries using their API. Updating my own on premise BIND DNS took a little longer. The following scripts were added to my crontab and and be used as needed.

One of the main benefits of moving from my Static Hurricane Electric /48 was that I was now able to use Cloudflare’s Proxy service and take advantage of their NAT64 Service. While I had implemented my own NAT64/SIIT-DC Cloudflare was easier. I had come across an issue that was little known; Cloudflare will provide DNS to free Hurricane Electric addresses but when you try and proxy them or use the NAT64 it will result in a generic error.

crontab:

*/5 * * * * /root/Cloudflare.sh HOSTNAME.FQDN
*/5 * * * * /root/Bind.sh HOSTNAME.FQDN

Cloudflare.sh
Bind.sh

Sources:

[Inbound Firewall rules with DHCPv6-PD] (https://community.ui.com/questions/Inbound-Firewall-rules-with-DHCPv6-PD/31aac051-4565-451c-89b2-8146a36724e8#comment/36880eb8-b3ef-4431-9e87-6049d2e720bb).

Replacing variables where necessary they should be easily adapted.

/etc/wireguard/updateLocal.sh

#!/bin/bash
sed -i -r -e "s/(define $1\s*=\s*).*/\1$2;/" /etc/bird/bird.conf
birdc configure

/etc/wireguard/updateUS_SFO1.sh

#!/bin/bash
localNode="CA_YYC1"
LinkLocal="$(ip addr show dn42-us-sfo1 | grep "scope link" |  cut -d " " -f6 | cut -d "/" -f1);"
ssh root@dn42-us-sfo1 "/etc/wireguard/updateLocal.sh $localNode $LinkLocal"

The following excerpt was added in BIRD routing Damon as the OSPFv3 configurations on the remote node:

interface "dn42-us-sfo1" {
  hello 5;
  cost 100;
  type nbma;  #nonbroadcast Network
  neighbors {
    CA_YYC1 eligible; # Defined Varable that is edited by above script
  };
};

postup = ip -4 address del ${YourPeerIP} dev ${devName}
postup = ip -4 address add ${YourPeerIP} dev ${devName} peer ${peerIP}

Sources:

WireGuard

PS C:\>Get-NetConnectionProfile
PS C:\>$Profile = Get-NetConnectionProfile -InterfaceAlias Ethernet1
PS C:\>$Profile.NetworkCategory = "Private"
PS C:\>Set-NetConnectionProfile -InputObject $Profile

Sources:

Server Fault

What you see now is running on Hugo’s framework. I also managed to throw in some version control and automatic building and deployment. It took a little bit of work but following the links below I was able to coax my GitLab into playing nice with it’s CI/CD work flow.

Sources:

Hugo

Deploying a Hugo Static Site Using GitLab, CI/CD, & SSH

Docker-in-Docker

interface eth0 {
    AdvSendAdvert on;
    AdvDefaultLifetime 0;
    prefix fd94:c9b4:7bda:1::/64 {
        AdvOnLink on;
        AdvAutonomous on;
    };
    route fd00::/8 {};
};

Sources:

Subnet with Two IPv6 Routers.

Photo by Echo Grid on Unsplash

nano /etc/systemd/system/rc-local.service

    [Unit]
    Description=/etc/rc.local
    ConditionPathExists=/etc/rc.local

    [Service]
    Type=forking
    ExecStart=/etc/rc.local start
    TimeoutSec=0
    StandardOutput=tty
    RemainAfterExit=yes
    SysVStartPriority=99

    [Install]
    WantedBy=multi-user.target

Modify /etc/rc.local to contain the following

#!/bin/sh -e
if ! [ -c /dev/net/tun ]; then
 mkdir -p /dev/net
 mknod -m 666 /dev/net/tun c 10 200
fi
# Shouldn\'t need this but this makes it work anyway
echo 1 > /proc/sys/net/ipv4/ip_forward
# VM Hack: Restart openvpn server so that it works
# (tun interface didn\'t exist at usual systemctl startup but it does now!)
systemctl restart openvpn@server.service
exit 0

Sources:

Re-enable rc.local Soruce

/dev/tun passthru Soruce

Systemd Service

/etc/systemd/system/ryzen-overclock.service

[Unit]
Description=Ryzen Overclock
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=basic.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/enable-pstates

[Install]
WantedBy=basic.target

Script

/usr/local/bin/enable-pstates

## P0 = 3.800GHz, 1.3750v
zenstates.py -p 0 -f 98 -d 8 -v 1c
## List the result.
zenstates.py -l

Zenstates.

Being that this one of the older servers I manage, All the server remote management had to offer was iDrac 5. I was lucky enough to not need a java install as often is the case, yet Google Chrome would play nice with the Active-X Console. Well off to the trusty legacy browser IE it is. Once the Active-X plugin got itself installed I actually got a view of what I was dealing with only to be followed by a sinking feeling.

Foreign configuration(s) found on adapter. Press any key to continue

I go to press a key, Press the key again, Press and hold all the keys….. Nothing. I had a windows into the issue but lacked any control to do anything about it. After search around I finaly found it. The simple fix. Add the url to the Trusted Site’s security list within IE. Refresh the Page, DONE!