diff --git a/apps/demo/service.yaml b/apps/demo/service.yaml index aa5be09..9436eee 100644 --- a/apps/demo/service.yaml +++ b/apps/demo/service.yaml @@ -22,7 +22,7 @@ metadata: # Must fall inside the range in # apps/metallb/templates/ipaddresspool.yaml, or MetalLB will never # satisfy the request. - metallb.universe.tf/loadBalancerIPs: "192.168.1.241" # EDIT ME + metallb.io/loadBalancerIPs: "10.10.60.11" spec: type: LoadBalancer selector: diff --git a/apps/metallb.yaml b/apps/metallb.yaml index c9a56d6..b92a8ef 100644 --- a/apps/metallb.yaml +++ b/apps/metallb.yaml @@ -1,12 +1,9 @@ # MetalLB, installed like any other app: an Application pointing at -# apps/metallb, a Helm chart (see apps/metallb/Chart.yaml) that both pulls -# in the upstream MetalLB chart AND carries our own IPAddressPool and -# L2Advertisement as extra templates (apps/metallb/templates/). Once this -# syncs, the cluster can hand out real IP addresses to Services of type -# LoadBalancer — something bare-metal Kubernetes doesn't do on its own -# (that's normally a cloud provider's job). See docs/05-metallb.md for the -# full explanation and the setup questions you'll need to answer before -# this is actually useful. +# apps/metallb, a Helm chart (see apps/metallb/Chart.yaml) that pulls in the +# upstream chart and carries this cluster's IPAddressPool, BGPPeer, and +# BGPAdvertisement templates. Together they let the MikroTik route dedicated +# 10.10.60.x addresses to LoadBalancer Services—something bare-metal +# Kubernetes does not provide on its own. See docs/05-metallb.md. # # Everything — MetalLB's own resources (including its CRDs) and our two # custom resources that depend on them — comes from this one Application, diff --git a/apps/metallb/Chart.yaml b/apps/metallb/Chart.yaml index 6f22589..b259762 100644 --- a/apps/metallb/Chart.yaml +++ b/apps/metallb/Chart.yaml @@ -1,7 +1,7 @@ # An "umbrella" chart: it declares a dependency on the upstream MetalLB -# chart below, AND carries two templates of its own -# (apps/metallb/templates/) — the IPAddressPool and L2Advertisement that -# configure it. Argo CD detects this as a Helm source because a Chart.yaml +# chart below, AND carries three templates of its own +# (apps/metallb/templates/) — the IPAddressPool, BGPPeer, and +# BGPAdvertisement that configure it. Argo CD detects this as a Helm source because a Chart.yaml # exists at this path, resolves the dependency, and syncs the combined # rendered output as one Application — exactly like any other app in # apps/. This is the real, working example of the Helm-based app pattern diff --git a/apps/metallb/templates/bgpadvertisement.yaml b/apps/metallb/templates/bgpadvertisement.yaml new file mode 100644 index 0000000..7b1842f --- /dev/null +++ b/apps/metallb/templates/bgpadvertisement.yaml @@ -0,0 +1,23 @@ +# BGPAdvertisement tells MetalLB to announce allocated Service addresses as +# routes to the MikroTik peer defined in bgppeer.yaml. Unlike Layer 2 mode, +# the workers do not need an interface inside 10.10.60.0/24: the router learns +# that each allocated address is reachable through the Kubernetes nodes. +# +# `aggregationLength: 32` advertises only addresses that a Service actually +# owns, as individual IPv4 host routes. Advertising the whole /24 would make +# the router send every address in the pool toward the cluster, including +# unallocated ones. +# +# Sync wave 1 defers this custom resource until the MetalLB chart's CRDs and +# controllers from wave 0 are available. +apiVersion: metallb.io/v1beta1 +kind: BGPAdvertisement +metadata: + name: default + namespace: metallb-system + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + ipAddressPools: + - default-pool + aggregationLength: 32 diff --git a/apps/metallb/templates/bgppeer.yaml b/apps/metallb/templates/bgppeer.yaml new file mode 100644 index 0000000..4a925ab --- /dev/null +++ b/apps/metallb/templates/bgppeer.yaml @@ -0,0 +1,27 @@ +# Establishes one external BGP session from each labelled worker to the +# MikroTik RB5009. BGP (Border Gateway Protocol) lets the workers advertise +# service IP routes from the dedicated 10.10.60.0/24 range even though their +# own interfaces remain in 10.10.10.0/24. +# +# The node selector deliberately excludes the Raspberry Pi control planes. +# Both workers share private ASN 64520 and peer with the router's private ASN +# 64512. The router supports ECMP, so it can retain equal-cost routes through +# both workers when both advertise the same Service address. +# +# No BGP password is used on this trusted internal network. Never add one in +# plaintext here: this repository is public, so authentication would require +# a separately managed Kubernetes Secret. +apiVersion: metallb.io/v1beta2 +kind: BGPPeer +metadata: + name: mikrotik + namespace: metallb-system + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + myASN: 64520 + peerASN: 64512 + peerAddress: 10.10.10.1 + nodeSelectors: + - matchLabels: + node-role.kubernetes.io/worker: worker diff --git a/apps/metallb/templates/ipaddresspool.yaml b/apps/metallb/templates/ipaddresspool.yaml index a61c9b3..95d4737 100644 --- a/apps/metallb/templates/ipaddresspool.yaml +++ b/apps/metallb/templates/ipaddresspool.yaml @@ -1,14 +1,10 @@ -# Tells MetalLB which IP addresses it's allowed to hand out to Services. -# Pick a small range on your LAN that nothing else will ever assign — -# especially your router's DHCP server. MetalLB doesn't coordinate with -# your router or DHCP server at all; it just answers network requests for -# whatever address a Service claims from this range. If it overlaps your -# DHCP range, you'll eventually get two devices claiming the same IP. +# Tells MetalLB which IP addresses it may assign to LoadBalancer Services. +# This is a dedicated routed service range, not the nodes' 10.10.10.0/24 LAN +# and not a DHCP pool. BGPAdvertisement announces only addresses actually in +# use, while the MikroTik accepts those routes from the two workers. # -# >> Ask Claude to help you pick this range before editing it — it needs -# to know your LAN's subnet and DHCP range, which it can't guess. -# A typical home LAN answer looks like "my router hands out -# 192.168.1.100–192.168.1.199, so let's use 192.168.1.240–192.168.1.250." +# The first and last few addresses remain reserved for future network use. +# Argo CD takes .10 and the demo app takes .11; .12-.250 remain available. # # This file lives under apps/metallb/templates/ alongside MetalLB's own # Helm-installed resources, rather than as a separate app, because it @@ -26,4 +22,4 @@ metadata: argocd.argoproj.io/sync-wave: "1" spec: addresses: - - 192.168.1.240-192.168.1.250 # EDIT ME — replace with your chosen range + - 10.10.60.10-10.10.60.250 diff --git a/apps/metallb/templates/l2advertisement.yaml b/apps/metallb/templates/l2advertisement.yaml deleted file mode 100644 index c3aa374..0000000 --- a/apps/metallb/templates/l2advertisement.yaml +++ /dev/null @@ -1,19 +0,0 @@ -# L2Advertisement is what actually makes addresses from the pool reachable: -# it tells MetalLB to answer ARP requests for whichever address a Service -# was given, from whichever node currently hosts that Service. This is -# "Layer 2 mode" — the simplest of MetalLB's two modes, and the right -# choice here since it needs no cooperation from your router (the -# alternative, BGP mode, does — worth knowing exists if you outgrow a -# single L2 segment, not needed for this setup). -# Same sync-wave reasoning as ipaddresspool.yaml — deferred to wave "1" so -# it applies after MetalLB's own chart resources are up. -apiVersion: metallb.io/v1beta1 -kind: L2Advertisement -metadata: - name: default - namespace: metallb-system - annotations: - argocd.argoproj.io/sync-wave: "1" -spec: - ipAddressPools: - - default-pool diff --git a/argocd/README.md b/argocd/README.md index dbca6bf..cc86a68 100644 --- a/argocd/README.md +++ b/argocd/README.md @@ -123,13 +123,12 @@ Argo CD's own install isn't managed by this repo's GitOps loop: ```sh kubectl -n argocd patch svc argocd-server --type merge -p \ - '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.universe.tf/loadBalancerIPs":""}}}' + '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}' ``` ```sh kubectl -n argocd get svc argocd-server ``` -Once `EXTERNAL-IP` shows your chosen address, `https://` -works directly, from any machine on your network — no port-forward -required. +Once `EXTERNAL-IP` shows `10.10.60.10`, `https://10.10.60.10` works +from routed networks — no port-forward required. diff --git a/docs/04-app-of-apps.md b/docs/04-app-of-apps.md index b7e2ea7..de7ae14 100644 --- a/docs/04-app-of-apps.md +++ b/docs/04-app-of-apps.md @@ -94,10 +94,9 @@ and `apps/longhorn.yaml`. ```sh kubectl -n demo get pods,svc ``` - You'll notice the Service's `EXTERNAL-IP` column says `` — - that's expected. The demo app is a `LoadBalancer` Service, and nothing - hands those out real addresses on bare metal until MetalLB is - configured, which is exactly the next doc. + MetalLB is already configured for this cluster's BGP service network, so + the Service's `EXTERNAL-IP` should become `10.10.60.11`. The next doc + verifies the BGP sessions and explains how that route reaches the app. If `app-of-apps`, `demo`, `metallb`, and `longhorn` all show `Synced`/`Healthy`, the loop — Git → Argo CD → cluster — is already proven @@ -116,8 +115,8 @@ end to end; you just can't see the demo app in a browser yet. - [ ] `app-of-apps` applied and shows `Synced`/`Healthy` - [ ] `demo`, `metallb`, and `longhorn` Applications appeared automatically, all `Synced`/`Healthy` -- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running - (Service `EXTERNAL-IP` still `` — expected for now) +- [ ] `kubectl -n demo get pods,svc` shows the demo Pod running and + Service `EXTERNAL-IP` `10.10.60.11` Continue to [`docs/05-metallb.md`](05-metallb.md) to give the demo app — and Argo CD's own UI — a real address. diff --git a/docs/05-metallb.md b/docs/05-metallb.md index 3db1eb6..b8008c9 100644 --- a/docs/05-metallb.md +++ b/docs/05-metallb.md @@ -1,155 +1,141 @@ -# 05 — MetalLB: getting real IP addresses on bare metal +# 05 — MetalLB: routing real service addresses with BGP -The app-of-apps you just applied ([`docs/04-app-of-apps.md`](04-app-of-apps.md)) -already brought up `metallb` alongside `demo`. This doc explains what it -does, walks you through the one piece only you can decide (which IP -addresses to actually use), and finishes by giving Argo CD's own UI a real -address too. +The app-of-apps from [`docs/04-app-of-apps.md`](04-app-of-apps.md) installs +MetalLB alongside the demo. This cluster uses a dedicated service network, +`10.10.60.0/24`, instead of borrowing addresses from the nodes' +`10.10.10.0/24` LAN. -## Why this is needed at all +## Why this is needed -On a cloud provider, a Kubernetes Service of `type: LoadBalancer` triggers -the cloud to provision an actual load balancer and hand your Service a -real, routable IP automatically. Bare metal — your Talos node, sitting on -your own network — has no such thing behind it. Without help, a -`LoadBalancer` Service on bare metal just sits with its address stuck at -`` forever. +A Kubernetes Service of `type: LoadBalancer` asks for an address reachable +from outside the cluster. A cloud platform would allocate and route one, but +bare-metal Kubernetes has no external load-balancer service. Without one, the +Service remains at ``. -**MetalLB** is that help: software that runs inside your cluster, -watches for `LoadBalancer` Services, and hands them a real address from a -range *you* configure — then makes that address actually reachable on your -network. It's the standard, widely-used answer to this exact gap, which is -why it's installed as a pre-installed app in this repo rather than left as -a "next step." +**MetalLB** fills that gap. It allocates an address from a controlled pool and +advertises a route for it with **BGP (Border Gateway Protocol)**. BGP is how +the Kubernetes workers tell the MikroTik router, "this service address is +reachable through me." -`apps/demo/service.yaml` is already `type: LoadBalancer` — that's what you -saw sitting at `` if you looked closely in the last doc. This doc -is what turns that into a real, reachable address. +## This cluster's routing design -## What's already running +```text +MikroTik RB5009 + address: 10.10.10.1 + ASN: 64512 + │ + │ eBGP, two equal-cost sessions + ├──────────────────────┐ + ▼ ▼ +worker01 / Gimli worker02 / Titus +10.10.10.203 10.10.10.204 +ASN 64520 ASN 64520 + └──────────┬───────────┘ + │ advertise allocated /32 routes + ▼ + MetalLB pool: 10.10.60.10-10.10.60.250 +``` -**`apps/metallb.yaml`** → one Application, pointing at `apps/metallb`, a -Helm chart that does two things at once (see -[`apps/metallb/Chart.yaml`](../apps/metallb/Chart.yaml)): +An **ASN (Autonomous System Number)** identifies one side of a BGP +relationship. These are private ASNs used only inside this network. -- Pulls in the official upstream MetalLB chart as a dependency — this is - also the real, working example of a Helm-based app mentioned in - [`docs/08-adding-new-apps.md`](08-adding-new-apps.md). -- Carries two templates of its own - ([`apps/metallb/templates/`](../apps/metallb/templates)): an - `IPAddressPool` and an `L2Advertisement` — MetalLB's own custom - resources, which tell it which addresses it's allowed to use and how to - advertise them. Right now these still contain **placeholder** IP - addresses that won't work on your network — that's what you're about to - fix. +The router has `multipath=2`, enabling **ECMP (Equal-Cost Multi-Path)**: when +both workers advertise the same service route with equal attributes, the +router can use both paths. If either physical Proxmox host fails, its route is +withdrawn and the surviving worker remains. -Both the upstream chart and our two extra resources sync together as one -Application. Look at the `sync-wave` annotation on -[`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) — -it's what keeps this safe despite being one sync: MetalLB's own resources -(including its CRDs) apply first, and Argo CD waits for them to be healthy -before applying our two resources, which depend on those CRDs existing. +## What the Git configuration contains -## Answer three questions before editing anything +[`apps/metallb/`](../apps/metallb/) is an umbrella Helm chart. It installs the +official MetalLB chart and three cluster-specific resources: -> **Ask Claude:** literally ask — "help me pick MetalLB's IP range and the -> addresses for Argo CD and the demo app." This is the one part of this -> whole repo that depends entirely on *your* network, which nothing in -> this repo can know in advance. Have your router's admin page or DHCP -> settings handy; Claude will ask what your LAN's subnet is and what range -> your DHCP server already hands out, so the pool it suggests doesn't -> collide with anything. +- [`ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) permits + allocation from `10.10.60.10-10.10.60.250`. +- [`bgppeer.yaml`](../apps/metallb/templates/bgppeer.yaml) creates sessions + from only the two labelled worker nodes to `10.10.10.1`. +- [`bgpadvertisement.yaml`](../apps/metallb/templates/bgpadvertisement.yaml) + advertises each allocated service address as an individual `/32` route. -You need three decisions: +Advertising `/32`s matters: the router learns only addresses currently owned +by Services, rather than sending the entire `/24` toward the cluster. -1. **The pool range** — a handful of addresses (5–10 is plenty for this - repo) on your LAN's subnet that your DHCP server will never hand to - anything else. Commonly a slice just above or below your DHCP range, - e.g. if your router hands out `.100`–`.199`, something like - `.240`–`.250` is a safe, unused slice. -2. **Argo CD's UI address** — one address from that pool. -3. **The demo app's address** — a different address from that pool. +The resources use Argo CD sync wave `1`; MetalLB's chart and CRDs apply in the +default wave first. A **CRD (Custom Resource Definition)** teaches Kubernetes +resource types such as `BGPPeer` before those objects are created. -## Edit the placeholders +## MikroTik configuration -1. [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) - — replace the `addresses` range with your chosen pool. -2. [`apps/demo/service.yaml`](../apps/demo/service.yaml) — replace the - `metallb.universe.tf/loadBalancerIPs` annotation with the demo app's - chosen address (must be inside the pool from step 1). -3. [`docs/network.md`](network.md) — record the pool range and the demo - app's address in the tables there (Argo CD's address goes in once you've - picked it below). -4. Commit and push all three: - ```sh - git add apps/metallb/templates/ipaddresspool.yaml apps/demo/service.yaml docs/network.md - git commit -m "Configure MetalLB's address pool for my network" - git push - ``` -5. Force a sync rather than waiting for the next poll: - ```sh - argocd app sync metallb demo - ``` +The RB5009 already has two passive RouterOS 7.23.3 connections: `.203` and +`.204` initiate sessions from ASN `64520` to router ASN `64512`. Its input +filter accepts only `/32` routes inside the dedicated service prefix: -## Verify +```routeros +/routing/filter/rule +add chain=cilium-bgp-in rule="if (dst in 10.10.60.0/24 && dst-len==32) { accept }" +add chain=cilium-bgp-out rule="if (dst-len>=0) { reject }" +``` + +The existing names retain `cilium-` from the previous cluster, but names are +only labels; they do not couple the peers to a CNI implementation. No BGP +password is used. Never add one to these manifests because this repository is +public. + +## Verify after the app-of-apps syncs + +Check MetalLB's resources: + +```sh +kubectl -n metallb-system get ipaddresspools,bgppeers,bgpadvertisements +``` + +On the MikroTik, both sessions should be established: + +```routeros +/routing/bgp/session/print detail without-paging +``` + +The demo requests `10.10.60.11` in +[`apps/demo/service.yaml`](../apps/demo/service.yaml): ```sh kubectl -n demo get svc demo ``` -`EXTERNAL-IP` should now show the address you chose instead of ``. -Visit `http://` in a browser — same "It works!" page as -before, now on its own real IP instead of a NodePort. -```sh -kubectl -n metallb-system get ipaddresspools,l2advertisements -``` -Confirms your pool and advertisement were applied as expected. +Its `EXTERNAL-IP` should become `10.10.60.11`. The router should learn a host +route for it, and `http://10.10.60.11` should show the demo page. -## Give Argo CD's UI a real address too +## Give Argo CD a routed address -So far you've reached the Argo CD UI through `kubectl port-forward` -(see [`docs/03-installing-argocd.md`](03-installing-argocd.md)). Now that -MetalLB is running, you can give it a permanent address instead. Argo CD's -own install isn't managed through this repo's GitOps loop (it's the -imperative bootstrap step — see `docs/03`), so this one change is applied -directly rather than through Git: +Argo CD itself was installed imperatively rather than through this repository, +so its Service is patched directly. The address is already reserved in +[`docs/network.md`](network.md): ```sh kubectl -n argocd patch svc argocd-server --type merge -p \ - '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.universe.tf/loadBalancerIPs":""}}}' + '{"spec":{"type":"LoadBalancer"},"metadata":{"annotations":{"metallb.io/loadBalancerIPs":"10.10.60.10"}}}' ``` -Replace `` with the address you chose in step 2 above. Then: +Verify: ```sh kubectl -n argocd get svc argocd-server ``` -Once `EXTERNAL-IP` shows your chosen address, visit -`https://` directly — no port-forward needed from now on. -(You'll still hit the same self-signed certificate warning as before; see -[`docs/03-installing-argocd.md`](03-installing-argocd.md) for why that's -expected here.) +Once `EXTERNAL-IP` is `10.10.60.10`, visit `https://10.10.60.10`. Argo CD's +default certificate is self-signed, so a browser warning is expected. -Update [`docs/network.md`](network.md) one more time with this address — -commit and push it. This one's a genuine exception to "everything's driven -by Git": the patch command above isn't a file in this repo, so this file is -the only place this address gets recorded at all. - -> **Ask Claude:** "why did the demo app get its LoadBalancer IP through -> Git, but Argo CD's didn't?" — a good check that the boundary between -> "managed by this repo" and "the imperative bootstrap that starts it" -> has actually landed. +> **Ask Claude:** why does the demo's address live in Git while Argo CD's is +> applied by hand? The answer is the bootstrap boundary: Argo CD manages the +> demo, but its own initial installation remains outside its GitOps loop. ## Checklist -- [ ] Chosen a pool range, an Argo CD UI address, and a demo app address, - with Claude's help, that fit your actual network -- [ ] `apps/metallb/templates/ipaddresspool.yaml` edited to your real range -- [ ] `apps/demo/service.yaml` edited to your real demo address -- [ ] `docs/network.md` updated with the pool range and both addresses -- [ ] Committed, pushed, synced -- [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser -- [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding +- [ ] MetalLB Application is `Synced` and `Healthy` +- [ ] Both MikroTik BGP sessions are established +- [ ] Router accepts only `10.10.60.0/24` `/32` routes from the workers +- [ ] Demo Service has `EXTERNAL-IP` `10.10.60.11` +- [ ] `http://10.10.60.11` loads +- [ ] Argo CD Service has `EXTERNAL-IP` `10.10.60.10` +- [ ] `https://10.10.60.10` reaches Argo CD Continue to [`docs/06-longhorn.md`](06-longhorn.md). diff --git a/docs/network.md b/docs/network.md index c704b27..5d9c639 100644 --- a/docs/network.md +++ b/docs/network.md @@ -34,17 +34,22 @@ on every node. ## MetalLB pool +This dedicated service range is advertised to the MikroTik at +`10.10.10.1` with BGP by the two workers; it is not an L2 or DHCP pool. + | | | |---|---| -| Range | `EDIT ME — e.g. 192.168.1.240-192.168.1.250` | -| Configured in | [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) | +| Range | `10.10.60.10-10.10.60.250` | +| Router / ASN | MikroTik RB5009 `10.10.10.1` / `64512` | +| Worker ASN | `64520` | +| Configured in | [`apps/metallb/templates/`](../apps/metallb/templates/) | ## Allocated addresses (from the MetalLB pool) | Address | What | Namespace | Port(s) | URL | Source | |---|---|---|---|---|---| -| `EDIT ME` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://EDIT ME` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) | -| `EDIT ME` | Demo app | `demo` | 80 | `http://EDIT ME` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) | +| `10.10.60.10` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://10.10.60.10` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) | +| `10.10.60.11` | Demo app | `demo` | 80 | `http://10.10.60.11` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) | Add a row here every time you add a new app with a `LoadBalancer` Service (see [`docs/08-adding-new-apps.md`](08-adding-new-apps.md)). Leave the URL @@ -57,4 +62,4 @@ Addresses from the range above that aren't spoken for yet. Update this alongside the table above so it's obvious at a glance what's safe to hand to the next app. -- `EDIT ME` +- `10.10.60.12-10.10.60.250`