ahmza.com/k8s-starter-pack
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Configure MetalLB BGP service network

Browse Source
This commit is contained in:
ahmza
2026-09-03 02:19:46 +01:00
parent 4cb6bc9ec4
commit 502de4fdf0
11 changed files with 186 additions and 173 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/demo/service.yaml
+1 -1
View File
@@ -22,7 +22,7 @@ metadata:
# Must fall inside the range in
# apps/metallb/templates/ipaddresspool.yaml, or MetalLB will never
# satisfy the request.
metallb.universe.tf/loadBalancerIPs: "192.168.1.241" # EDIT ME
metallb.io/loadBalancerIPs: "10.10.60.11"
spec:
type: LoadBalancer
selector:
apps/metallb.yaml
+5 -8
View File
@@ -1,12 +1,9 @@
# MetalLB, installed like any other app: an Application pointing at
# apps/metallb, a Helm chart (see apps/metallb/Chart.yaml) that both pulls
# in the upstream MetalLB chart AND carries our own IPAddressPool and
# L2Advertisement as extra templates (apps/metallb/templates/). Once this
# syncs, the cluster can hand out real IP addresses to Services of type
# LoadBalancer — something bare-metal Kubernetes doesn't do on its own
# (that's normally a cloud provider's job). See docs/05-metallb.md for the
# full explanation and the setup questions you'll need to answer before
# this is actually useful.
# apps/metallb, a Helm chart (see apps/metallb/Chart.yaml) that pulls in the
# upstream chart and carries this cluster's IPAddressPool, BGPPeer, and
# BGPAdvertisement templates. Together they let the MikroTik route dedicated
# 10.10.60.x addresses to LoadBalancer Services—something bare-metal
# Kubernetes does not provide on its own. See docs/05-metallb.md.
#
# Everything — MetalLB's own resources (including its CRDs) and our two
# custom resources that depend on them — comes from this one Application,
apps/metallb/Chart.yaml
+3 -3
View File
@@ -1,7 +1,7 @@
# An "umbrella" chart: it declares a dependency on the upstream MetalLB
# chart below, AND carries two templates of its own
# (apps/metallb/templates/) — the IPAddressPool and L2Advertisement that
# configure it. Argo CD detects this as a Helm source because a Chart.yaml
# chart below, AND carries three templates of its own
# (apps/metallb/templates/) — the IPAddressPool, BGPPeer, and
# BGPAdvertisement that configure it. Argo CD detects this as a Helm source because a Chart.yaml
# exists at this path, resolves the dependency, and syncs the combined
# rendered output as one Application — exactly like any other app in
# apps/. This is the real, working example of the Helm-based app pattern
apps/metallb/templates/bgpadvertisement.yaml
+23
View File
@@ -0,0 +1,23 @@
# BGPAdvertisement tells MetalLB to announce allocated Service addresses as
# routes to the MikroTik peer defined in bgppeer.yaml. Unlike Layer 2 mode,
# the workers do not need an interface inside 10.10.60.0/24: the router learns
# that each allocated address is reachable through the Kubernetes nodes.
#
# `aggregationLength: 32` advertises only addresses that a Service actually
# owns, as individual IPv4 host routes. Advertising the whole /24 would make
# the router send every address in the pool toward the cluster, including
# unallocated ones.
#
# Sync wave 1 defers this custom resource until the MetalLB chart's CRDs and
# controllers from wave 0 are available.
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: default
namespace: metallb-system
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
ipAddressPools:
- default-pool
aggregationLength: 32
apps/metallb/templates/bgppeer.yaml
+27
View File
@@ -0,0 +1,27 @@
# Establishes one external BGP session from each labelled worker to the
# MikroTik RB5009. BGP (Border Gateway Protocol) lets the workers advertise
# service IP routes from the dedicated 10.10.60.0/24 range even though their
# own interfaces remain in 10.10.10.0/24.
#
# The node selector deliberately excludes the Raspberry Pi control planes.
# Both workers share private ASN 64520 and peer with the router's private ASN
# 64512. The router supports ECMP, so it can retain equal-cost routes through
# both workers when both advertise the same Service address.
#
# No BGP password is used on this trusted internal network. Never add one in
# plaintext here: this repository is public, so authentication would require
# a separately managed Kubernetes Secret.
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: mikrotik
namespace: metallb-system
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
myASN: 64520
peerASN: 64512
peerAddress: 10.10.10.1
nodeSelectors:
- matchLabels:
node-role.kubernetes.io/worker: worker
apps/metallb/templates/ipaddresspool.yaml
+7 -11
View File
@@ -1,14 +1,10 @@
# Tells MetalLB which IP addresses it's allowed to hand out to Services.
# Pick a small range on your LAN that nothing else will ever assign —
# especially your router's DHCP server. MetalLB doesn't coordinate with
# your router or DHCP server at all; it just answers network requests for
# whatever address a Service claims from this range. If it overlaps your
# DHCP range, you'll eventually get two devices claiming the same IP.
# Tells MetalLB which IP addresses it may assign to LoadBalancer Services.
# This is a dedicated routed service range, not the nodes' 10.10.10.0/24 LAN
# and not a DHCP pool. BGPAdvertisement announces only addresses actually in
# use, while the MikroTik accepts those routes from the two workers.
#
# >> Ask Claude to help you pick this range before editing it — it needs
# to know your LAN's subnet and DHCP range, which it can't guess.
# A typical home LAN answer looks like "my router hands out
# 192.168.1.100–192.168.1.199, so let's use 192.168.1.240–192.168.1.250."
# The first and last few addresses remain reserved for future network use.
# Argo CD takes .10 and the demo app takes .11; .12-.250 remain available.
#
# This file lives under apps/metallb/templates/ alongside MetalLB's own
# Helm-installed resources, rather than as a separate app, because it
@@ -26,4 +22,4 @@ metadata:
argocd.argoproj.io/sync-wave: "1"
spec:
addresses:
- 192.168.1.240-192.168.1.250 # EDIT ME — replace with your chosen range
- 10.10.60.10-10.10.60.250
apps/metallb/templates/l2advertisement.yaml
-19
View File
@@ -1,19 +0,0 @@
# L2Advertisement is what actually makes addresses from the pool reachable:
# it tells MetalLB to answer ARP requests for whichever address a Service
# was given, from whichever node currently hosts that Service. This is
# "Layer 2 mode" — the simplest of MetalLB's two modes, and the right
# choice here since it needs no cooperation from your router (the
# alternative, BGP mode, does — worth knowing exists if you outgrow a
# single L2 segment, not needed for this setup).
# Same sync-wave reasoning as ipaddresspool.yaml — deferred to wave "1" so
# it applies after MetalLB's own chart resources are up.
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
ipAddressPools:
- default-pool
argocd/README.md
+3 -4
View File
@@ -123,13 +123,12 @@ Argo CD's own install isn't managed by this repo's GitOps loop:
```sh
kubectl -n argocd patch svc argocd-server --type merge -p \
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.universe.tf/loadBalancerIPs":"<ARGOCD_IP>"}}}'
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}'
```
```sh
kubectl -n argocd get svc argocd-server
```
Once `EXTERNAL-IP` shows your chosen address, `https://<that-address>`
works directly, from any machine on your network — no port-forward
required.
Once `EXTERNAL-IP` shows `10.10.60.10`, `https://10.10.60.10` works
from routed networks — no port-forward required.
docs/04-app-of-apps.md
+5 -6
View File
@@ -94,10 +94,9 @@ and `apps/longhorn.yaml`.
```sh
kubectl -n demo get pods,svc
```
You'll notice the Service's `EXTERNAL-IP` column says `<pending>` —
that's expected. The demo app is a `LoadBalancer` Service, and nothing
hands those out real addresses on bare metal until MetalLB is
configured, which is exactly the next doc.
MetalLB is already configured for this cluster's BGP service network, so
the Service's `EXTERNAL-IP` should become `10.10.60.11`. The next doc
verifies the BGP sessions and explains how that route reaches the app.
If `app-of-apps`, `demo`, `metallb`, and `longhorn` all show
`Synced`/`Healthy`, the loop — Git → Argo CD → cluster — is already proven
@@ -116,8 +115,8 @@ end to end; you just can't see the demo app in a browser yet.
- [ ] `app-of-apps` applied and shows `Synced`/`Healthy`
- [ ] `demo`, `metallb`, and `longhorn` Applications appeared
automatically, all `Synced`/`Healthy`
- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running
(Service `EXTERNAL-IP` still `<pending>` — expected for now)
- [ ] `kubectl -n demo get pods,svc` shows the demo Pod running and
Service `EXTERNAL-IP` `10.10.60.11`
Continue to [`docs/05-metallb.md`](05-metallb.md) to give the demo app —
and Argo CD's own UI — a real address.
docs/05-metallb.md
+102 -116
View File
@@ -1,155 +1,141 @@
# 05 — MetalLB: getting real IP addresses on bare metal
# 05 — MetalLB: routing real service addresses with BGP
The app-of-apps you just applied ([`docs/04-app-of-apps.md`](04-app-of-apps.md))
already brought up `metallb` alongside `demo`. This doc explains what it
does, walks you through the one piece only you can decide (which IP
addresses to actually use), and finishes by giving Argo CD's own UI a real
address too.
The app-of-apps from [`docs/04-app-of-apps.md`](04-app-of-apps.md) installs
MetalLB alongside the demo. This cluster uses a dedicated service network,
`10.10.60.0/24`, instead of borrowing addresses from the nodes'
`10.10.10.0/24` LAN.
## Why this is needed at all
## Why this is needed
On a cloud provider, a Kubernetes Service of `type: LoadBalancer` triggers
the cloud to provision an actual load balancer and hand your Service a
real, routable IP automatically. Bare metal — your Talos node, sitting on
your own network — has no such thing behind it. Without help, a
`LoadBalancer` Service on bare metal just sits with its address stuck at
`<pending>` forever.
A Kubernetes Service of `type: LoadBalancer` asks for an address reachable
from outside the cluster. A cloud platform would allocate and route one, but
bare-metal Kubernetes has no external load-balancer service. Without one, the
Service remains at `<pending>`.
**MetalLB** is that help: software that runs inside your cluster,
watches for `LoadBalancer` Services, and hands them a real address from a
range *you* configure — then makes that address actually reachable on your
network. It's the standard, widely-used answer to this exact gap, which is
why it's installed as a pre-installed app in this repo rather than left as
a "next step."
**MetalLB** fills that gap. It allocates an address from a controlled pool and
advertises a route for it with **BGP (Border Gateway Protocol)**. BGP is how
the Kubernetes workers tell the MikroTik router, "this service address is
reachable through me."
`apps/demo/service.yaml` is already `type: LoadBalancer` — that's what you
saw sitting at `<pending>` if you looked closely in the last doc. This doc
is what turns that into a real, reachable address.
## This cluster's routing design
## What's already running
```text
MikroTik RB5009
address: 10.10.10.1
ASN: 64512
│
│ eBGP, two equal-cost sessions
├──────────────────────┐
▼ ▼
worker01 / Gimli worker02 / Titus
10.10.10.203 10.10.10.204
ASN 64520 ASN 64520
└──────────┬───────────┘
│ advertise allocated /32 routes
▼
MetalLB pool: 10.10.60.10-10.10.60.250
```
**`apps/metallb.yaml`** → one Application, pointing at `apps/metallb`, a
Helm chart that does two things at once (see
[`apps/metallb/Chart.yaml`](../apps/metallb/Chart.yaml)):
An **ASN (Autonomous System Number)** identifies one side of a BGP
relationship. These are private ASNs used only inside this network.
- Pulls in the official upstream MetalLB chart as a dependency — this is
also the real, working example of a Helm-based app mentioned in
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
- Carries two templates of its own
([`apps/metallb/templates/`](../apps/metallb/templates)): an
`IPAddressPool` and an `L2Advertisement` — MetalLB's own custom
resources, which tell it which addresses it's allowed to use and how to
advertise them. Right now these still contain **placeholder** IP
addresses that won't work on your network — that's what you're about to
fix.
The router has `multipath=2`, enabling **ECMP (Equal-Cost Multi-Path)**: when
both workers advertise the same service route with equal attributes, the
router can use both paths. If either physical Proxmox host fails, its route is
withdrawn and the surviving worker remains.
Both the upstream chart and our two extra resources sync together as one
Application. Look at the `sync-wave` annotation on
[`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) —
it's what keeps this safe despite being one sync: MetalLB's own resources
(including its CRDs) apply first, and Argo CD waits for them to be healthy
before applying our two resources, which depend on those CRDs existing.
## What the Git configuration contains
## Answer three questions before editing anything
[`apps/metallb/`](../apps/metallb/) is an umbrella Helm chart. It installs the
official MetalLB chart and three cluster-specific resources:
> **Ask Claude:** literally ask — "help me pick MetalLB's IP range and the
> addresses for Argo CD and the demo app." This is the one part of this
> whole repo that depends entirely on *your* network, which nothing in
> this repo can know in advance. Have your router's admin page or DHCP
> settings handy; Claude will ask what your LAN's subnet is and what range
> your DHCP server already hands out, so the pool it suggests doesn't
> collide with anything.
- [`ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) permits
allocation from `10.10.60.10-10.10.60.250`.
- [`bgppeer.yaml`](../apps/metallb/templates/bgppeer.yaml) creates sessions
from only the two labelled worker nodes to `10.10.10.1`.
- [`bgpadvertisement.yaml`](../apps/metallb/templates/bgpadvertisement.yaml)
advertises each allocated service address as an individual `/32` route.
You need three decisions:
Advertising `/32`s matters: the router learns only addresses currently owned
by Services, rather than sending the entire `/24` toward the cluster.
1. **The pool range** — a handful of addresses (5–10 is plenty for this
repo) on your LAN's subnet that your DHCP server will never hand to
anything else. Commonly a slice just above or below your DHCP range,
e.g. if your router hands out `.100`–`.199`, something like
`.240`–`.250` is a safe, unused slice.
2. **Argo CD's UI address** — one address from that pool.
3. **The demo app's address** — a different address from that pool.
The resources use Argo CD sync wave `1`; MetalLB's chart and CRDs apply in the
default wave first. A **CRD (Custom Resource Definition)** teaches Kubernetes
resource types such as `BGPPeer` before those objects are created.
## Edit the placeholders
## MikroTik configuration
1. [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml)
— replace the `addresses` range with your chosen pool.
2. [`apps/demo/service.yaml`](../apps/demo/service.yaml) — replace the
`metallb.universe.tf/loadBalancerIPs` annotation with the demo app's
chosen address (must be inside the pool from step 1).
3. [`docs/network.md`](network.md) — record the pool range and the demo
app's address in the tables there (Argo CD's address goes in once you've
picked it below).
4. Commit and push all three:
```sh
git add apps/metallb/templates/ipaddresspool.yaml apps/demo/service.yaml docs/network.md
git commit -m "Configure MetalLB's address pool for my network"
git push
```
5. Force a sync rather than waiting for the next poll:
```sh
argocd app sync metallb demo
```
The RB5009 already has two passive RouterOS 7.23.3 connections: `.203` and
`.204` initiate sessions from ASN `64520` to router ASN `64512`. Its input
filter accepts only `/32` routes inside the dedicated service prefix:
## Verify
```routeros
/routing/filter/rule
add chain=cilium-bgp-in rule="if (dst in 10.10.60.0/24 && dst-len==32) { accept }"
add chain=cilium-bgp-out rule="if (dst-len>=0) { reject }"
```
The existing names retain `cilium-` from the previous cluster, but names are
only labels; they do not couple the peers to a CNI implementation. No BGP
password is used. Never add one to these manifests because this repository is
public.
## Verify after the app-of-apps syncs
Check MetalLB's resources:
```sh
kubectl -n metallb-system get ipaddresspools,bgppeers,bgpadvertisements
```
On the MikroTik, both sessions should be established:
```routeros
/routing/bgp/session/print detail without-paging
```
The demo requests `10.10.60.11` in
[`apps/demo/service.yaml`](../apps/demo/service.yaml):
```sh
kubectl -n demo get svc demo
```
`EXTERNAL-IP` should now show the address you chose instead of `<pending>`.
Visit `http://<that-address>` in a browser — same "It works!" page as
before, now on its own real IP instead of a NodePort.
```sh
kubectl -n metallb-system get ipaddresspools,l2advertisements
```
Confirms your pool and advertisement were applied as expected.
Its `EXTERNAL-IP` should become `10.10.60.11`. The router should learn a host
route for it, and `http://10.10.60.11` should show the demo page.
## Give Argo CD's UI a real address too
## Give Argo CD a routed address
So far you've reached the Argo CD UI through `kubectl port-forward`
(see [`docs/03-installing-argocd.md`](03-installing-argocd.md)). Now that
MetalLB is running, you can give it a permanent address instead. Argo CD's
own install isn't managed through this repo's GitOps loop (it's the
imperative bootstrap step — see `docs/03`), so this one change is applied
directly rather than through Git:
Argo CD itself was installed imperatively rather than through this repository,
so its Service is patched directly. The address is already reserved in
[`docs/network.md`](network.md):
```sh
kubectl -n argocd patch svc argocd-server --type merge -p \
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.universe.tf/loadBalancerIPs":"<ARGOCD_IP>"}}}'
'{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}'
```
Replace `<ARGOCD_IP>` with the address you chose in step 2 above. Then:
Verify:
```sh
kubectl -n argocd get svc argocd-server
```
Once `EXTERNAL-IP` shows your chosen address, visit
`https://<that-address>` directly — no port-forward needed from now on.
(You'll still hit the same self-signed certificate warning as before; see
[`docs/03-installing-argocd.md`](03-installing-argocd.md) for why that's
expected here.)
Once `EXTERNAL-IP` is `10.10.60.10`, visit `https://10.10.60.10`. Argo CD's
default certificate is self-signed, so a browser warning is expected.
Update [`docs/network.md`](network.md) one more time with this address —
commit and push it. This one's a genuine exception to "everything's driven
by Git": the patch command above isn't a file in this repo, so this file is
the only place this address gets recorded at all.
> **Ask Claude:** "why did the demo app get its LoadBalancer IP through
> Git, but Argo CD's didn't?" — a good check that the boundary between
> "managed by this repo" and "the imperative bootstrap that starts it"
> has actually landed.
> **Ask Claude:** why does the demo's address live in Git while Argo CD's is
> applied by hand? The answer is the bootstrap boundary: Argo CD manages the
> demo, but its own initial installation remains outside its GitOps loop.
## Checklist
- [ ] Chosen a pool range, an Argo CD UI address, and a demo app address,
with Claude's help, that fit your actual network
- [ ] `apps/metallb/templates/ipaddresspool.yaml` edited to your real range
- [ ] `apps/demo/service.yaml` edited to your real demo address
- [ ] `docs/network.md` updated with the pool range and both addresses
- [ ] Committed, pushed, synced
- [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser
- [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding
- [ ] MetalLB Application is `Synced` and `Healthy`
- [ ] Both MikroTik BGP sessions are established
- [ ] Router accepts only `10.10.60.0/24` `/32` routes from the workers
- [ ] Demo Service has `EXTERNAL-IP` `10.10.60.11`
- [ ] `http://10.10.60.11` loads
- [ ] Argo CD Service has `EXTERNAL-IP` `10.10.60.10`
- [ ] `https://10.10.60.10` reaches Argo CD
Continue to [`docs/06-longhorn.md`](06-longhorn.md).
docs/network.md
+10 -5
View File
@@ -34,17 +34,22 @@ on every node.
## MetalLB pool
This dedicated service range is advertised to the MikroTik at
`10.10.10.1` with BGP by the two workers; it is not an L2 or DHCP pool.
| | |
|---|---|
| Range | `EDIT ME — e.g. 192.168.1.240-192.168.1.250` |
| Configured in | [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) |
| Range | `10.10.60.10-10.10.60.250` |
| Router / ASN | MikroTik RB5009 `10.10.10.1` / `64512` |
| Worker ASN | `64520` |
| Configured in | [`apps/metallb/templates/`](../apps/metallb/templates/) |
## Allocated addresses (from the MetalLB pool)
| Address | What | Namespace | Port(s) | URL | Source |
|---|---|---|---|---|---|
| `EDIT ME` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://EDIT ME` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) |
| `EDIT ME` | Demo app | `demo` | 80 | `http://EDIT ME` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) |
| `10.10.60.10` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://10.10.60.10` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) |
| `10.10.60.11` | Demo app | `demo` | 80 | `http://10.10.60.11` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) |
Add a row here every time you add a new app with a `LoadBalancer` Service
(see [`docs/08-adding-new-apps.md`](08-adding-new-apps.md)). Leave the URL
@@ -57,4 +62,4 @@ Addresses from the range above that aren't spoken for yet. Update this
alongside the table above so it's obvious at a glance what's safe to hand
to the next app.
- `EDIT ME`
- `10.10.60.12-10.10.60.250`