05 — MetalLB: routing real service addresses with BGP

The app-of-apps from docs/04-app-of-apps.md installs MetalLB alongside the demo. This cluster uses a dedicated service network, 10.10.60.0/24, instead of borrowing addresses from the nodes' 10.10.10.0/24 LAN.

Why this is needed

A Kubernetes Service of type: LoadBalancer asks for an address reachable from outside the cluster. A cloud platform would allocate and route one, but bare-metal Kubernetes has no external load-balancer service. Without one, the Service remains at <pending>.

MetalLB fills that gap. It allocates an address from a controlled pool and advertises a route for it with BGP (Border Gateway Protocol). BGP is how the Kubernetes workers tell the MikroTik router, "this service address is reachable through me."

This cluster's routing design

MikroTik RB5009
  address: 10.10.10.1
  ASN:     64512
       │
       │ eBGP, two equal-cost sessions
       ├──────────────────────┐
       ▼                      ▼
worker01 / Gimli         worker02 / Titus
10.10.10.203             10.10.10.204
ASN 64520                ASN 64520
       └──────────┬───────────┘
                  │ advertise allocated /32 routes
                  ▼
       MetalLB pool: 10.10.60.10-10.10.60.250

An ASN (Autonomous System Number) identifies one side of a BGP relationship. These are private ASNs used only inside this network.

The router has multipath=2, enabling ECMP (Equal-Cost Multi-Path): when both workers advertise the same service route with equal attributes, the router can use both paths. If either physical Proxmox host fails, its route is withdrawn and the surviving worker remains.

What the Git configuration contains

apps/metallb/ is an umbrella Helm chart. It installs the official MetalLB chart and three cluster-specific resources:

ipaddresspool.yaml permits allocation from 10.10.60.10-10.10.60.250.
bgppeer.yaml creates sessions from only the two labelled worker nodes to 10.10.10.1.
bgpadvertisement.yaml advertises each allocated service address as an individual /32 route.

Advertising /32s matters: the router learns only addresses currently owned by Services, rather than sending the entire /24 toward the cluster.

The resources use Argo CD sync wave 1; MetalLB's chart and CRDs apply in the default wave first. A CRD (Custom Resource Definition) teaches Kubernetes resource types such as BGPPeer before those objects are created.

MikroTik configuration

The RB5009 already has two passive RouterOS 7.23.3 connections: .203 and .204 initiate sessions from ASN 64520 to router ASN 64512. Its input filter accepts only /32 routes inside the dedicated service prefix:

/routing/filter/rule
add chain=cilium-bgp-in rule="if (dst in 10.10.60.0/24 && dst-len==32) { accept }"
add chain=cilium-bgp-out rule="if (dst-len>=0) { reject }"

The existing names retain cilium- from the previous cluster, but names are only labels; they do not couple the peers to a CNI implementation. No BGP password is used. Never add one to these manifests because this repository is public.

Verify after the app-of-apps syncs

Check MetalLB's resources:

kubectl -n metallb-system get ipaddresspools,bgppeers,bgpadvertisements

On the MikroTik, both sessions should be established:

/routing/bgp/session/print detail without-paging

The demo requests 10.10.60.11 in apps/demo/service.yaml:

kubectl -n demo get svc demo

Its EXTERNAL-IP should become 10.10.60.11. The router should learn a host route for it, and http://10.10.60.11 should show the demo page.

Give Argo CD a routed address

Argo CD itself was installed imperatively rather than through this repository, so its Service is patched directly. The address is already reserved in docs/network.md:

kubectl -n argocd patch svc argocd-server --type merge -p \
  '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}'

Verify:

kubectl -n argocd get svc argocd-server

Once EXTERNAL-IP is 10.10.60.10, visit https://10.10.60.10. Argo CD's default certificate is self-signed, so a browser warning is expected.

Ask Claude: why does the demo's address live in Git while Argo CD's is applied by hand? The answer is the bootstrap boundary: Argo CD manages the demo, but its own initial installation remains outside its GitOps loop.

Checklist

MetalLB Application is Synced and Healthy
Both MikroTik BGP sessions are established
Router accepts only 10.10.60.0/24 /32 routes from the workers
Demo Service has EXTERNAL-IP 10.10.60.11
http://10.10.60.11 loads
Argo CD Service has EXTERNAL-IP 10.10.60.10
https://10.10.60.10 reaches Argo CD

Continue to docs/06-longhorn.md.

5.1 KiB Raw Blame History