k8s-starter-pack/docs/05-metallb.md

# 05 — MetalLB: routing real service addresses with BGP

The app-of-apps from [`docs/04-app-of-apps.md`](04-app-of-apps.md) installs
MetalLB alongside the demo. This cluster uses a dedicated service network,
`10.10.60.0/24`, instead of borrowing addresses from the nodes'
`10.10.10.0/24` LAN.

## Why this is needed

A Kubernetes Service of `type: LoadBalancer` asks for an address reachable
from outside the cluster. A cloud platform would allocate and route one, but
bare-metal Kubernetes has no external load-balancer service. Without one, the
Service remains at `<pending>`.

**MetalLB** fills that gap. It allocates an address from a controlled pool and
advertises a route for it with **BGP (Border Gateway Protocol)**. BGP is how
the Kubernetes workers tell the MikroTik router, "this service address is
reachable through me."

## This cluster's routing design

```text
MikroTik RB5009
  address: 10.10.10.1
  ASN:     64512
       │
       │ eBGP, two equal-cost sessions
       ├──────────────────────┐
       ▼                      ▼
worker01 / Gimli         worker02 / Titus
10.10.10.203             10.10.10.204
ASN 64520                ASN 64520
       └──────────┬───────────┘
                  │ advertise allocated /32 routes
                  ▼
       MetalLB pool: 10.10.60.10-10.10.60.250
```

An **ASN (Autonomous System Number)** identifies one side of a BGP
relationship. These are private ASNs used only inside this network.

The router has `multipath=2`, enabling **ECMP (Equal-Cost Multi-Path)**: when
both workers advertise the same service route with equal attributes, the
router can use both paths. If either physical Proxmox host fails, its route is
withdrawn and the surviving worker remains.

## What the Git configuration contains

[`apps/metallb/`](../apps/metallb/) is an umbrella Helm chart. It installs the
official MetalLB chart and three cluster-specific resources:

- [`ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) permits
  allocation from `10.10.60.10-10.10.60.250`.
- [`bgppeer.yaml`](../apps/metallb/templates/bgppeer.yaml) creates sessions
  from only the two labelled worker nodes to `10.10.10.1`.
- [`bgpadvertisement.yaml`](../apps/metallb/templates/bgpadvertisement.yaml)
  advertises each allocated service address as an individual `/32` route.

Advertising `/32`s matters: the router learns only addresses currently owned
by Services, rather than sending the entire `/24` toward the cluster.

The resources use Argo CD sync wave `1`; MetalLB's chart and CRDs apply in the
default wave first. A **CRD (Custom Resource Definition)** teaches Kubernetes
resource types such as `BGPPeer` before those objects are created.

## MikroTik configuration

The RB5009 already has two passive RouterOS 7.23.3 connections: `.203` and
`.204` initiate sessions from ASN `64520` to router ASN `64512`. Its input
filter accepts only `/32` routes inside the dedicated service prefix:

```routeros
/routing/filter/rule
add chain=cilium-bgp-in rule="if (dst in 10.10.60.0/24 && dst-len==32) { accept }"
add chain=cilium-bgp-out rule="if (dst-len>=0) { reject }"
```

The existing names retain `cilium-` from the previous cluster, but names are
only labels; they do not couple the peers to a CNI implementation. No BGP
password is used. Never add one to these manifests because this repository is
public.

## Verify after the app-of-apps syncs

Check MetalLB's resources:

```sh
kubectl -n metallb-system get ipaddresspools,bgppeers,bgpadvertisements
```

On the MikroTik, both sessions should be established:

```routeros
/routing/bgp/session/print detail without-paging
```

The demo requests `10.10.60.11` in
[`apps/demo/service.yaml`](../apps/demo/service.yaml):

```sh
kubectl -n demo get svc demo
```

Its `EXTERNAL-IP` should become `10.10.60.11`. The router should learn a host
route for it, and `http://10.10.60.11` should show the demo page.

## Give Argo CD a routed address

Argo CD itself was installed imperatively rather than through this repository,
so its Service is patched directly. The address is already reserved in
[`docs/network.md`](network.md):

```sh
kubectl -n argocd patch svc argocd-server --type merge -p \
  '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}'
```

Verify:

```sh
kubectl -n argocd get svc argocd-server
```

Once `EXTERNAL-IP` is `10.10.60.10`, visit `https://10.10.60.10`. Argo CD's
default certificate is self-signed, so a browser warning is expected.

> **Ask Claude:** why does the demo's address live in Git while Argo CD's is
> applied by hand? The answer is the bootstrap boundary: Argo CD manages the
> demo, but its own initial installation remains outside its GitOps loop.

## Checklist

- [ ] MetalLB Application is `Synced` and `Healthy`
- [ ] Both MikroTik BGP sessions are established
- [ ] Router accepts only `10.10.60.0/24` `/32` routes from the workers
- [ ] Demo Service has `EXTERNAL-IP` `10.10.60.11`
- [ ] `http://10.10.60.11` loads
- [ ] Argo CD Service has `EXTERNAL-IP` `10.10.60.10`
- [ ] `https://10.10.60.10` reaches Argo CD

Continue to [`docs/06-longhorn.md`](06-longhorn.md).