The Problem

In Part 1, I set up a multi-homed spine-leaf topology with Cilium speaking iBGP directly to two Mikrotik route reflectors. The topology was sound (two uplinks, two spine switches, ECMP in theory), but in practice it fell apart the moment a cable came unplugged.

A convergence test on node-d told the whole story:

Last ping:    1776725680.888   (T+0)
Link down:                      (~T+0)
Ping restored: 1776725878.146  (T+197s, only after physical replug)
BGP restored:  1776725941.656  (T+260s)
Ping to BGP restore gap: 63s

197 seconds with no connectivity, and it only recovered because I physically replugged the cable. The CRS354 session stayed up the entire time; the traffic just never failed over to it. Meanwhile the CRS309 session dropped and re-established on its own after replug, but took another 63 seconds to converge.

The root cause: only one IPv6 default route existed at a time, installed via SLAAC router advertisements from whichever interface’s RA the node had accepted. There was no second default route to fall back on. When that interface went down, the default route disappeared with it, and the node had nothing left to route out on. Existing TCP connections pinned to the dead gateway broke immediately, and no new connections could be established either, since there was no surviving default route until the interface came back. Cilium’s BGP implementation had no BFD support, no default route injection, and no ECMP source routing.

The conclusion was clear: I needed a proper routing daemon between Cilium and the spines.

Why BIRD2

BIRD2 is a production-grade routing daemon that supports BFD, ECMP, add-path, and flexible route filtering, none of which Cilium’s BGP implementation provides.

The key insight was that Cilium and BIRD2 needed to be in different autonomous systems. With both in AS 64529 (iBGP), the split horizon rule prevents BIRD2 from readvertising Cilium’s routes to the spines. With Cilium in AS 64530 and BIRD2 in AS 64529 (eBGP on localhost), BIRD2 can freely redistribute Cilium routes to the iBGP fabric.

Cilium (AS 64530) → BIRD2 (AS 64529) → Spines (AS 64529, iBGP RR)

This works because iBGP split-horizon only blocks re-advertising iBGP-learned routes; eBGP-learned routes aren’t restricted. Running BIRD2 and Cilium as separate ASes over localhost turns Cilium’s routes into eBGP-learned routes from BIRD2’s perspective, so they’re free to redistribute to the iBGP fabric. I don’t know of a standard name for this, but it’s effectively a “BGP sidecar”.

Architecture

graph TB
    subgraph "Edge / WAN"
        R1["RB5009 Edge Router
AS 64529
2001:db8:abcd:500::1"]
    end

    subgraph "Spine Layer"
        TOR1["CRS309 Spine RR
AS 64529
Router-ID: 0.3.0.9
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00a::1/64"]
        TOR2["CRS354 Spine RR
AS 64529
Router-ID: 0.3.5.4
Cluster-ID: 5.0.0.9
2001:db8:abcd:e00b::1/64"]
    end

    subgraph "Kubernetes Nodes"
        Node1["Node-b
┌─────────────┐
│ BIRD2 AS64529│
│ Cilium AS64530│
└─────────────┘"]
        Node2["Node-c
┌─────────────┐
│ BIRD2 AS64529│
│ Cilium AS64530│
└─────────────┘"]
        NodeX["Node-d
┌─────────────┐
│ BIRD2 AS64529│
│ Cilium AS64530│
└─────────────┘"]
    end

    R1 -- iBGP + Default Originate + BFD --> TOR1
    R1 -- iBGP + Default Originate + BFD --> TOR2

    TOR1 -- iBGP RR + BFD + add-path --> Node1
    TOR1 -- iBGP RR + BFD + add-path --> Node2
    TOR1 -- iBGP RR + BFD + add-path --> NodeX

    TOR2 -- iBGP RR + BFD + add-path --> Node1
    TOR2 -- iBGP RR + BFD + add-path --> Node2
    TOR2 -- iBGP RR + BFD + add-path --> NodeX

    style R1 fill:#1e3a5f,stroke:#4fc3f7,stroke-width:2px,color:#fff
    style TOR1 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style TOR2 fill:#3e2723,stroke:#ffab91,stroke-width:2px,color:#fff
    style Node1 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style Node2 fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff
    style NodeX fill:#1b5e20,stroke:#81c784,stroke-width:2px,color:#fff

Each node runs BIRD2 as a Talos system extension with per-node configuration. BIRD2 has three peerings per node:

BIRD2 on Talos

BIRD2 is deployed as a Talos system extension (siderolabs/bird2) in the node schematic. The configuration is embedded directly in talconfig.yaml using per-node extensionServices blocks, with each node getting a unique router id (required for BGP, and must be IPv4, since IPv6-only interfaces like dummy0 can’t provide one).

router id 192.168.56.2;   # node-b
router id 192.168.56.3;   # node-c
router id 192.168.56.4;   # node-d

The dummy0 interface on each node carries a /128 IPv6 address that serves as the node’s stable identity. BIRD2 imports it via a direct protocol and advertises it to the spines:

protocol direct loopback {
    interface "dummy0";
    ipv6 { import all; export none; };
}

This replaces Cilium’s Interface advertisement for dummy0, which was removed from the Cilium BGP config.

BFD

BFD (Bidirectional Forwarding Detection) is the key to sub-second failover. Without it, BGP relies on hold timers (typically 30+ seconds) to detect link failure.

Single-Hop vs Multihop

This was one of the harder lessons. BFD comes in two flavors:

• Single-hop BFD: Tied to a specific interface, detects direct link failures
• Multihop BFD: Works across multiple hops, but Mikrotik RouterOS doesn’t support it well

Initially I tried multihop BFD sessions because the iBGP peers were using GUA addresses. This caused Mikrotik to log “BFD forbidden for interface”; it turns out RouterOS determines single-hop vs multihop BFD based on whether the BGP connection has an interface binding (%interface suffix on local.address).

The fix: use GUA addresses with direct + source address in BIRD2, and add %interface to the Mikrotik local.address on each spine connection:

/routing bgp connection add name=to-node-b remote.address=2001:db8:abcd:e00a:ec4:7aff:feba:bf86 local.address=2001:db8:abcd:e00a::1%901-Spine .role=ibgp-rr use-bfd=yes

The %901-Spine suffix tells RouterOS this is a directly-connected session, enabling single-hop BFD.

BIRD2 BFD Configuration

BIRD2 requires a separate protocol bfd block, since inline BFD parameters in BGP protocols are invalid syntax:

protocol bfd {
    interface "ens1f0" { };
    interface "ens1f1" { };
}

Then each BGP peer gets bfd on; and direct;:

protocol bgp spine_crs309 {
    local as 64529;
    source address <node-ens1f0-slaac>;
    neighbor 2001:db8:abcd:e00a::1 as 64529;
    direct;
    bfd on;
    ...
}

The direct keyword overrides iBGP’s implicit multihop behavior, making the session interface-bound and compatible with single-hop BFD.

The Default Route Problem

Even with BFD and BGP working perfectly, nodes had no default route from BGP. They relied entirely on SLAAC router advertisements from the switches.

This caused two problems:

1. SLAAC defaults don’t fail over: The default route came from whichever interface’s RA the node had accepted, with no second default route in reserve. When that interface went down, the default route vanished with it, and the node had nothing left to route out on until the interface came back.

2. No path awareness: SLAAC just says “default via link-local on this interface”. It has no concept of whether the upstream path is actually reachable.

The solution: BIRD2 installs BFD-tracked ECMP default routes into the kernel routing table:

protocol static default_route {
    ipv6;
    route ::/0 via 2001:db8:abcd:e00a::1%ens1f0 bfd;
    route ::/0 via 2001:db8:abcd:e00b::1%ens1f1 bfd;
}

These routes have a lower metric (32) than SLAAC defaults (1024), so they’re preferred. When BFD detects a link failure (~1 second), BIRD2 withdraws that nexthop, and traffic shifts to the surviving path.

The kernel export filter ensures only useful routes get installed:

protocol kernel k6 {
    merge paths on;
    ipv6 {
        export filter {
            if proto = "cilium_local" then reject;   # Cilium handles its own routes
            if proto = "loopback" then reject;        # Don't install dummy0 /128 as kernel route
            accept;
        };
        import none;
    };
}

BGP Timer Tuning

BFD handles failure detection (~1 second), but BGP session recovery after a link comes back depends on BGP timers. The defaults are painfully slow:

After tuning:

# Spine peers - 15s hold time matches Mikrotik side
hold time 15;
keepalive time 5;
connect retry time 5;
connect delay time 2;
error wait time 5, 30;      # 5s initial, 30s max
error forget time 60;        # Reset backoff after 60s stable
graceful restart on;         # Preserve forwarding during session resets

# Cilium localhost peer - aggressive timers (no packet loss possible)
hold time 3;
keepalive time 1;
connect retry time 2;
connect delay time 1;
error wait time 2, 10;
error forget time 30;
graceful restart on;

On the Mikrotik spines, matching timers:

/routing/bgp/connection/set to-node-b hold-time=15s keepalive-time=5s
/routing/bgp/connection/set to-node-c hold-time=15s keepalive-time=5s
/routing/bgp/connection/set to-node-d hold-time=15s keepalive-time=5s

Note: RouterOS uses keepalive-time, not keepalive-interval.

Cilium eBGP Configuration

The Cilium BGP config changed significantly from Part 1. Instead of peering directly with the spines, Cilium now peers with BIRD2 on localhost:

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  bgpInstances:
    - localASN: 64530                    # Was 64529 (iBGP), now eBGP
      name: instance-64530
      peers:
        - name: peer-bird2-localhost      # Single peer, not two
          peerASN: 64529
          peerAddress: "::1"              # Localhost
          peerConfigRef:
            name: cilium-peer
  nodeSelector:
    matchLabels:
      kubernetes.io/arch: amd64
---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  ebgpMultihop: 4                          # Required for localhost peering
  families:
    - advertisements:
        matchLabels:
          advertise: bgp
      afi: ipv6
      safi: unicast
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  timers:
    holdTimeSeconds: 3
    keepAliveTimeSeconds: 1
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  labels:
    advertise: bgp
  name: bgp-advertisements
spec:
  advertisements:
    - advertisementType: Service
      selector:
        matchExpressions: []
      service:
        addresses:
          - ClusterIP
          - LoadBalancerIP
    - advertisementType: CiliumPodIPPool
    - advertisementType: PodCIDR
    # REMOVED: Interface advertisement for dummy0
    # BIRD2 now advertises dummy0 via its direct protocol

Key changes:

• localASN changed from 64529 to 64530 (eBGP to bypass split horizon)
• Single peer at ::1 instead of two spine peers
• ebgpMultihop: 4 added (required for localhost peering)
• dummy0 Interface advertisement removed (BIRD2 handles it)
• Timers made more aggressive (3s hold, 1s keepalive)

Routing Flow

With BIRD2 in the path, the route propagation looks like:

Pod/Service IP (Cilium)
    ↓ eBGP (::1)
BIRD2 on node
    ↓ iBGP + BFD + add-path
Spine switches (CRS309, CRS354)
    ↓ iBGP + add-path
RB5009 (edge router, ECMP)
    ↓ default route
Internet

Northbound (pod → internet):

1. Pod traffic hits node’s default route (BIRD2 static ECMP, metric 32)
2. Traffic exits via ens1f0 or ens1f1 (ECMP load-balanced)
3. Spine forwards to RB5009
4. RB5009 has default route to ISP

Southbound (internet → pod):

1. Traffic arrives at RB5009
2. RB5009 has ECMP routes for pod/service CIDRs via both spines
3. Spines reflect to BIRD2 via iBGP (add-path enabled for ECMP)
4. BIRD2 passes to Cilium via eBGP
5. Cilium routes to the correct pod

Mikrotik Configuration

The biggest change on the Mikrotik side was replacing dynamic BGP listeners with explicit per-node connections. Dynamic listeners with use-bfd=yes caused “BFD forbidden for interface” errors with GUA addresses. The fix: explicit connections with %interface suffix on local.address.

CRS309 (VLAN901)

/routing bgp instance add as=64529 cluster-id=5.0.0.9 name=spine-rr router-id=0.3.0.9
/routing bgp template add add-path-out=all as=64529 name=spine-template

# Per-node explicit connections with %interface suffix for BFD
/routing bgp connection add instance=spine-rr name=to-node-b \
    remote.address=2001:db8:abcd:e00a:ec4:7aff:feba:bf86 \
    local.address=2001:db8:abcd:e00a::1%901-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

/routing bgp connection add instance=spine-rr name=to-node-c \
    remote.address=2001:db8:abcd:e00a:ec4:7aff:feba:c020 \
    local.address=2001:db8:abcd:e00a::1%901-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

/routing bgp connection add instance=spine-rr name=to-node-d \
    remote.address=2001:db8:abcd:e00a:ec4:7aff:feba:c284 \
    local.address=2001:db8:abcd:e00a::1%901-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

# Gateway connection
/routing bgp connection add instance=spine-rr name=to-rb5009 \
    remote.address=2001:db8:abcd:500::1 \
    local.address=2001:db8:abcd:500::4%CORE \
    .role=ibgp-rr routing-table=main as=64529 use-bfd=yes add-path-out=all \
    output.redistribute=connected

# BFD
/routing/bfd/configuration/add addresses=2001:db8:abcd:e00a:ec4:7aff:feba:bf86,2001:db8:abcd:e00a:ec4:7aff:feba:c020,2001:db8:abcd:e00a:ec4:7aff:feba:c284 min-rx=200ms min-tx=200ms multiplier=5

CRS354 (VLAN902)

Same pattern, different VLAN and addresses:

/routing bgp instance add as=64529 cluster-id=5.0.0.9 name=spine-rr router-id=0.3.5.4
/routing bgp template add add-path-out=all as=64529 name=spine-template

/routing bgp connection add instance=spine-rr name=to-node-b \
    remote.address=2001:db8:abcd:e00b:ec4:7aff:feba:bf87 \
    local.address=2001:db8:abcd:e00b::1%902-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

/routing bgp connection add instance=spine-rr name=to-node-c \
    remote.address=2001:db8:abcd:e00b:ec4:7aff:feba:c021 \
    local.address=2001:db8:abcd:e00b::1%902-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

/routing bgp connection add instance=spine-rr name=to-node-d \
    remote.address=2001:db8:abcd:e00b:ec4:7aff:feba:c285 \
    local.address=2001:db8:abcd:e00b::1%902-Spine \
    .role=ibgp-rr templates=spine-template use-bfd=yes add-path-out=all \
    hold-time=15s keepalive-time=5s

/routing bgp connection add instance=spine-rr name=to-rb5009 \
    remote.address=2001:db8:abcd:500::1 \
    local.address=2001:db8:abcd:500::5%CORE \
    .role=ibgp-rr routing-table=main as=64529 use-bfd=yes add-path-out=all \
    output.redistribute=connected

/routing/bfd/configuration/add addresses=2001:db8:abcd:e00b:ec4:7aff:feba:bf87,2001:db8:abcd:e00b:ec4:7aff:feba:c021,2001:db8:abcd:e00b:ec4:7aff:feba:c285 min-rx=200ms min-tx=200ms multiplier=5

Syntax Landmines

Getting BIRD2 and Mikrotik to cooperate on IPv6-only BFD iBGP was a journey. Here’s every syntax issue I ran into, in case it saves someone else the debugging:

BIRD2 Configuration (Complete)

The full BIRD2 config per node, embedded in talconfig.yaml:

log stderr all;
debug protocols off;
router id 192.168.56.X;   # .2 for node-b, .3 for node-c, .4 for node-d

protocol device { scan time 10; }

protocol bfd {
    interface "ens1f0" { };
    interface "ens1f1" { };
}

protocol static default_route {
    ipv6;
    route ::/0 via 2001:db8:abcd:e00a::1%ens1f0 bfd;
    route ::/0 via 2001:db8:abcd:e00b::1%ens1f1 bfd;
}

protocol direct loopback {
    interface "dummy0";
    ipv6 { import all; export none; };
}

protocol kernel k6 {
    merge paths on;
    learn off;
    ipv6 {
        export filter {
            if proto = "cilium_local" then reject;
            if proto = "loopback" then reject;
            accept;
        };
        import none;
    };
}

protocol bgp spine_crs309 {
    local as 64529;
    source address <node-ens1f0-slaac>;
    neighbor 2001:db8:abcd:e00a::1 as 64529;
    direct;
    bfd on;
    hold time 15;
    keepalive time 5;
    connect retry time 5;
    connect delay time 2;
    error wait time 5, 30;
    error forget time 60;
    graceful restart on;
    ipv6 {
        import all;
        export filter {
            if proto = "cilium_local" then accept;
            if proto = "loopback" then accept;
            reject;
        };
        next hop self;
        add paths on;
    };
}

protocol bgp spine_crs354 {
    local as 64529;
    source address <node-ens1f1-slaac>;
    neighbor 2001:db8:abcd:e00b::1 as 64529;
    direct;
    bfd on;
    hold time 15;
    keepalive time 5;
    connect retry time 5;
    connect delay time 2;
    error wait time 5, 30;
    error forget time 60;
    graceful restart on;
    ipv6 {
        import all;
        export filter {
            if proto = "cilium_local" then accept;
            if proto = "loopback" then accept;
            reject;
        };
        next hop self;
        add paths on;
    };
}

protocol bgp cilium_local {
    passive on;
    local as 64529;
    neighbor ::1 as 64530;
    multihop 2;
    hold time 3;
    keepalive time 1;
    connect retry time 2;
    connect delay time 1;
    error wait time 2, 10;
    error forget time 30;
    graceful restart on;
    ipv6 {
        import all;
        export all;
        next hop self;
        next hop address <node-dummy0-GUA>;
    };
}

Results

The BFD sessions come up within seconds and the BGP sessions establish immediately after:

bird> show protocols
Name       Proto      Table      State  Since         Info
device1    Device     ---        up     22:21:25.047  
bfd1       BFD        ---        up     22:21:25.047  
static1    Static     master6    up     22:21:25.047  
loopback   Direct     ---        up     22:21:25.047  
k6         Kernel     master6    up     22:21:25.047  
spine_crs309 BGP     ---        up     22:21:29.121  Established
spine_crs354 BGP     ---        up     22:21:29.866  Established
cilium_local BGP      ---        up     22:21:38.679  Established

bird> show bfd sessions
IP address                Interface  State      Since         Interval  Timeout
2001:db8:abcd:e00b::1     ens1f1     Up         22:21:30.860    0.200    1.000
2001:db8:abcd:e00a::1     ens1f0     Up         22:21:30.018    0.200    1.000

bird> show route for ::/0
::/0                 unicast [static1 22:21:30.019] * (200)
	via 2001:db8:abcd:e00a::1 on ens1f0
                     unicast [static1 22:21:30.860] (200)
	via 2001:db8:abcd:e00b::1 on ens1f1

Outstanding Work

1. Verify ECMP kernel routes: ip -6 route show default metric 32 should show two nexthops
2. Full convergence test: physically unplug ens1f0 and measure failover time
3. BGP default route from RB5009: Currently BIRD2 uses static defaults pointing at spine IPs. If both spines are up but the upstream link is down, traffic blackholes. Receiving ::/0 via BGP from RB5009 through the spines would provide true path awareness.
4. Clean up stale Mikrotik sessions: Remove .as=0 and .hold-time=infinity sessions on CRS309
5. Consider removing SLAAC defaults: BIRD2’s BFD-tracked static defaults provide better failover

Notes

Addressing Plan (unchanged from Part 1):

BIRD2 Router IDs:

Recovery Time Breakdown