ahmza.com/k8s-starter-pack
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Configure MetalLB BGP service network

Browse Source
This commit is contained in:
ahmza
2026-09-03 02:19:46 +01:00
parent 4cb6bc9ec4
commit 502de4fdf0
11 changed files with 186 additions and 173 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/05-metallb.md
+102 -116
View File
@@ -1,155 +1,141 @@
# 05 — MetalLB: getting real IP addresses on bare metal
# 05 — MetalLB: routing real service addresses with BGP
The app-of-apps you just applied ([`docs/04-app-of-apps.md`](04-app-of-apps.md))
already brought up `metallb` alongside `demo`. This doc explains what it
does, walks you through the one piece only you can decide (which IP
addresses to actually use), and finishes by giving Argo CD's own UI a real
address too.
The app-of-apps from [`docs/04-app-of-apps.md`](04-app-of-apps.md) installs
MetalLB alongside the demo. This cluster uses a dedicated service network,
`10.10.60.0/24`, instead of borrowing addresses from the nodes'
`10.10.10.0/24` LAN.
## Why this is needed at all
## Why this is needed
On a cloud provider, a Kubernetes Service of `type: LoadBalancer` triggers
the cloud to provision an actual load balancer and hand your Service a
real, routable IP automatically. Bare metal — your Talos node, sitting on
your own network — has no such thing behind it. Without help, a
`LoadBalancer` Service on bare metal just sits with its address stuck at
`<pending>` forever.
A Kubernetes Service of `type: LoadBalancer` asks for an address reachable
from outside the cluster. A cloud platform would allocate and route one, but
bare-metal Kubernetes has no external load-balancer service. Without one, the
Service remains at `<pending>`.
**MetalLB** is that help: software that runs inside your cluster,
watches for `LoadBalancer` Services, and hands them a real address from a
range *you* configure — then makes that address actually reachable on your
network. It's the standard, widely-used answer to this exact gap, which is
why it's installed as a pre-installed app in this repo rather than left as
a "next step."
**MetalLB** fills that gap. It allocates an address from a controlled pool and
advertises a route for it with **BGP (Border Gateway Protocol)**. BGP is how
the Kubernetes workers tell the MikroTik router, "this service address is
reachable through me."
`apps/demo/service.yaml` is already `type: LoadBalancer` — that's what you
saw sitting at `<pending>` if you looked closely in the last doc. This doc
is what turns that into a real, reachable address.
## This cluster's routing design
## What's already running
```text
MikroTik RB5009
address: 10.10.10.1
ASN: 64512
│
│ eBGP, two equal-cost sessions
├──────────────────────┐
▼ ▼
worker01 / Gimli worker02 / Titus
10.10.10.203 10.10.10.204
ASN 64520 ASN 64520
└──────────┬───────────┘
│ advertise allocated /32 routes
▼
MetalLB pool: 10.10.60.10-10.10.60.250
```
**`apps/metallb.yaml`** → one Application, pointing at `apps/metallb`, a
Helm chart that does two things at once (see
[`apps/metallb/Chart.yaml`](../apps/metallb/Chart.yaml)):
An **ASN (Autonomous System Number)** identifies one side of a BGP
relationship. These are private ASNs used only inside this network.
- Pulls in the official upstream MetalLB chart as a dependency — this is
also the real, working example of a Helm-based app mentioned in
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
- Carries two templates of its own
([`apps/metallb/templates/`](../apps/metallb/templates)): an
`IPAddressPool` and an `L2Advertisement` — MetalLB's own custom
resources, which tell it which addresses it's allowed to use and how to
advertise them. Right now these still contain **placeholder** IP
addresses that won't work on your network — that's what you're about to
fix.
The router has `multipath=2`, enabling **ECMP (Equal-Cost Multi-Path)**: when
both workers advertise the same service route with equal attributes, the
router can use both paths. If either physical Proxmox host fails, its route is
withdrawn and the surviving worker remains.
Both the upstream chart and our two extra resources sync together as one
Application. Look at the `sync-wave` annotation on
[`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) —
it's what keeps this safe despite being one sync: MetalLB's own resources
(including its CRDs) apply first, and Argo CD waits for them to be healthy
before applying our two resources, which depend on those CRDs existing.
## What the Git configuration contains
## Answer three questions before editing anything
[`apps/metallb/`](../apps/metallb/) is an umbrella Helm chart. It installs the
official MetalLB chart and three cluster-specific resources:
> **Ask Claude:** literally ask — "help me pick MetalLB's IP range and the
> addresses for Argo CD and the demo app." This is the one part of this
> whole repo that depends entirely on *your* network, which nothing in
> this repo can know in advance. Have your router's admin page or DHCP
> settings handy; Claude will ask what your LAN's subnet is and what range
> your DHCP server already hands out, so the pool it suggests doesn't
> collide with anything.
- [`ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) permits
allocation from `10.10.60.10-10.10.60.250`.
- [`bgppeer.yaml`](../apps/metallb/templates/bgppeer.yaml) creates sessions
from only the two labelled worker nodes to `10.10.10.1`.
- [`bgpadvertisement.yaml`](../apps/metallb/templates/bgpadvertisement.yaml)
advertises each allocated service address as an individual `/32` route.
You need three decisions:
Advertising `/32`s matters: the router learns only addresses currently owned
by Services, rather than sending the entire `/24` toward the cluster.
1. **The pool range** — a handful of addresses (5–10 is plenty for this
repo) on your LAN's subnet that your DHCP server will never hand to
anything else. Commonly a slice just above or below your DHCP range,
e.g. if your router hands out `.100`–`.199`, something like
`.240`–`.250` is a safe, unused slice.
2. **Argo CD's UI address** — one address from that pool.
3. **The demo app's address** — a different address from that pool.
The resources use Argo CD sync wave `1`; MetalLB's chart and CRDs apply in the
default wave first. A **CRD (Custom Resource Definition)** teaches Kubernetes
resource types such as `BGPPeer` before those objects are created.
## Edit the placeholders
## MikroTik configuration
1. [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml)
— replace the `addresses` range with your chosen pool.
2. [`apps/demo/service.yaml`](../apps/demo/service.yaml) — replace the
`metallb.universe.tf/loadBalancerIPs` annotation with the demo app's
chosen address (must be inside the pool from step 1).
3. [`docs/network.md`](network.md) — record the pool range and the demo
app's address in the tables there (Argo CD's address goes in once you've
picked it below).
4. Commit and push all three:
```sh
git add apps/metallb/templates/ipaddresspool.yaml apps/demo/service.yaml docs/network.md
git commit -m "Configure MetalLB's address pool for my network"
git push
```
5. Force a sync rather than waiting for the next poll:
```sh
argocd app sync metallb demo
```
The RB5009 already has two passive RouterOS 7.23.3 connections: `.203` and
`.204` initiate sessions from ASN `64520` to router ASN `64512`. Its input
filter accepts only `/32` routes inside the dedicated service prefix:
## Verify
```routeros
/routing/filter/rule
add chain=cilium-bgp-in rule="if (dst in 10.10.60.0/24 && dst-len==32) { accept }"
add chain=cilium-bgp-out rule="if (dst-len>=0) { reject }"
```
The existing names retain `cilium-` from the previous cluster, but names are
only labels; they do not couple the peers to a CNI implementation. No BGP
password is used. Never add one to these manifests because this repository is
public.
## Verify after the app-of-apps syncs
Check MetalLB's resources:
```sh
kubectl -n metallb-system get ipaddresspools,bgppeers,bgpadvertisements
```
On the MikroTik, both sessions should be established:
```routeros
/routing/bgp/session/print detail without-paging
```
The demo requests `10.10.60.11` in
[`apps/demo/service.yaml`](../apps/demo/service.yaml):
```sh
kubectl -n demo get svc demo
```
`EXTERNAL-IP` should now show the address you chose instead of `<pending>`.
Visit `http://<that-address>` in a browser — same "It works!" page as
before, now on its own real IP instead of a NodePort.
```sh
kubectl -n metallb-system get ipaddresspools,l2advertisements
```
Confirms your pool and advertisement were applied as expected.
Its `EXTERNAL-IP` should become `10.10.60.11`. The router should learn a host
route for it, and `http://10.10.60.11` should show the demo page.
## Give Argo CD's UI a real address too
## Give Argo CD a routed address
So far you've reached the Argo CD UI through `kubectl port-forward`
(see [`docs/03-installing-argocd.md`](03-installing-argocd.md)). Now that
MetalLB is running, you can give it a permanent address instead. Argo CD's
own install isn't managed through this repo's GitOps loop (it's the
imperative bootstrap step — see `docs/03`), so this one change is applied
directly rather than through Git:
Argo CD itself was installed imperatively rather than through this repository,
so its Service is patched directly. The address is already reserved in
[`docs/network.md`](network.md):
```sh
kubectl -n argocd patch svc argocd-server --type merge -p \
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.universe.tf/loadBalancerIPs":"<ARGOCD_IP>"}}}'
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}'
```
Replace `<ARGOCD_IP>` with the address you chose in step 2 above. Then:
Verify:
```sh
kubectl -n argocd get svc argocd-server
```
Once `EXTERNAL-IP` shows your chosen address, visit
`https://<that-address>` directly — no port-forward needed from now on.
(You'll still hit the same self-signed certificate warning as before; see
[`docs/03-installing-argocd.md`](03-installing-argocd.md) for why that's
expected here.)
Once `EXTERNAL-IP` is `10.10.60.10`, visit `https://10.10.60.10`. Argo CD's
default certificate is self-signed, so a browser warning is expected.
Update [`docs/network.md`](network.md) one more time with this address —
commit and push it. This one's a genuine exception to "everything's driven
by Git": the patch command above isn't a file in this repo, so this file is
the only place this address gets recorded at all.
> **Ask Claude:** "why did the demo app get its LoadBalancer IP through
> Git, but Argo CD's didn't?" — a good check that the boundary between
> "managed by this repo" and "the imperative bootstrap that starts it"
> has actually landed.
> **Ask Claude:** why does the demo's address live in Git while Argo CD's is
> applied by hand? The answer is the bootstrap boundary: Argo CD manages the
> demo, but its own initial installation remains outside its GitOps loop.
## Checklist
- [ ] Chosen a pool range, an Argo CD UI address, and a demo app address,
with Claude's help, that fit your actual network
- [ ] `apps/metallb/templates/ipaddresspool.yaml` edited to your real range
- [ ] `apps/demo/service.yaml` edited to your real demo address
- [ ] `docs/network.md` updated with the pool range and both addresses
- [ ] Committed, pushed, synced
- [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser
- [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding
- [ ] MetalLB Application is `Synced` and `Healthy`
- [ ] Both MikroTik BGP sessions are established
- [ ] Router accepts only `10.10.60.0/24` `/32` routes from the workers
- [ ] Demo Service has `EXTERNAL-IP` `10.10.60.11`
- [ ] `http://10.10.60.11` loads
- [ ] Argo CD Service has `EXTERNAL-IP` `10.10.60.10`
- [ ] `https://10.10.60.10` reaches Argo CD
Continue to [`docs/06-longhorn.md`](06-longhorn.md).