diff --git a/docs/network.md b/docs/network.md index d1170d4..a6b7e84 100644 --- a/docs/network.md +++ b/docs/network.md @@ -32,6 +32,21 @@ on every node. | `10.10.10.204` | `talos-vba-ood` — worker02 (Proxmox VM on Titus) | 50000 | `https://10.10.10.204:50000` | | `10.10.10.210` | Cluster VIP (use this one, not a node's own address) | 6443 | `https://10.10.10.210:6443` | +## NetBird overlay + +These addresses exist only inside the private NetBird overlay and are subject +to NetBird access policies. All Talos nodes use the official +`siderolabs/netbird` system extension. + +| Address | What | Port(s) | URL(s) | +|---|---|---|---| +| `100.71.24.9` | `prd-vps-hel-01` — external Grafana and VictoriaMetrics host | 3000, 8428 | `http://100.71.24.9:3000`, `http://100.71.24.9:8428` | +| `100.71.121.67` | `talos-455-i3b` — control plane NetBird address | 50000 | `https://100.71.121.67:50000` | +| `100.71.157.217` | `talos-9xy-iuk` — control plane NetBird address | 50000 | `https://100.71.157.217:50000` | +| `100.71.242.40` | `talos-k1b-7j5` — control plane NetBird address | 50000 | `https://100.71.242.40:50000` | +| `100.71.3.210` | `talos-rss-v5l` — worker01 NetBird address | 50000 | `https://100.71.3.210:50000` | +| `100.71.29.51` | `talos-vba-ood` — worker02 NetBird address | 50000 | `https://100.71.29.51:50000` | + ## MetalLB pool This dedicated service range is advertised to the MikroTik at diff --git a/talos/README.md b/talos/README.md index 91a26fa..083f9db 100644 --- a/talos/README.md +++ b/talos/README.md @@ -5,18 +5,14 @@ This is the detailed, copy-pasteable companion to first for the explanations — this one is here so you have every exact command in one place once you understand what they do. -Throughout, replace ``, ``, `` with your three -machines' real IP addresses, and `` with the shared address you -picked (with Claude's help) for the cluster's VIP — see -[`docs/02-talos-bootstrap.md`](../docs/02-talos-bootstrap.md#the-vip-one-address-for-a-cluster-that-moves). -Give each machine a DHCP reservation (a fixed IP tied to its MAC address) -in your router or hypervisor before you start, so none of these addresses -change under the cluster. Talos bakes them into certificates during setup, -and changing one later means starting over for that node. +Throughout, use the three control-plane and two worker addresses recorded in +[`docs/network.md`](../docs/network.md), plus the cluster's shared VIP. Give +each machine a DHCP reservation (a fixed IP tied to its MAC address) before +installation so its address remains stable. -Steps 1–2 below run **once, total**. Steps 3 runs **once per node** (three -times). Steps 4–6 run **once, total**, against whichever node you like -(or, once it exists, the VIP). +Steps 1–2 below run **once, total**. Step 3 runs **once per node** (five +times, using the matching role configuration). Steps 4–6 run **once, total**; +Talos API commands target a control-plane node, while Kubernetes uses the VIP. All commands assume you're running them from the root of this repo. @@ -26,7 +22,10 @@ All commands assume you're running them from the root of this repo. talosctl gen config gitops-starter https://:6443 \ --output-dir _out \ --config-patch-control-plane @talos/patch-controlplane.yaml \ - --config-patch-control-plane @talos/patch-longhorn.yaml + --config-patch-control-plane @talos/patch-netbird.yaml \ + --config-patch-worker @talos/patch-worker.yaml \ + --config-patch-worker @talos/patch-longhorn.yaml \ + --config-patch-worker @talos/patch-netbird.yaml ``` Note the cluster endpoint here is the **VIP**, not any one node's address @@ -38,13 +37,12 @@ This writes three files into `_out/` (already covered by this repo's `.gitignore` — they contain private keys and must never be committed): - `controlplane.yaml` — the machine configuration for a control-plane - node, with both patches baked in. Applied identically to all three - nodes — there's no separate worker config in this setup, since every - node is both. -- `worker.yaml` — a plain worker config, generated automatically but - unused here since every node in this repo's setup is both control plane - and worker. Kept around for [`docs/09-growing-up.md`](../docs/09-growing-up.md), - where a fourth-and-beyond node uses exactly this file. + node, with the Raspberry Pi/VIP and non-secret NetBird patches baked in. + It is applied identically to all three control-plane nodes. +- `worker.yaml` — the machine configuration for the two Proxmox workers, + with their role, Longhorn data-disk, and non-secret NetBird patches baked + in. The NetBird setup key is deliberately absent from both generated files; + enrollment uses a short-lived key applied separately and revoked afterward. - `talosconfig` — *your* client credentials for talking to the Talos API on this cluster (equivalent in spirit to a kubeconfig, but for Talos itself rather than Kubernetes). @@ -68,34 +66,39 @@ separate. ## 3. Apply the configuration — once per node -Boot each machine from your custom Talos ISO (see +Boot each machine from its architecture-appropriate custom Talos image (see [docs/01-prerequisites.md](../docs/01-prerequisites.md)). Each boots into -"maintenance mode" — alive enough to accept a configuration, with nothing -installed yet. For **each** node in turn: +"maintenance mode" — alive enough to accept configuration, with nothing +installed yet. Inspect its disks first: ```sh talosctl get disks --insecure --nodes ``` -Confirm you see two disks: the one Talos will install itself to, and the -≥900GB one `talos/patch-longhorn.yaml`'s `UserVolumeConfig` will claim for -Longhorn. Then: +For each Raspberry Pi control plane, confirm `/dev/mmcblk0` is the intended +system disk, then apply the control-plane configuration: ```sh talosctl apply-config --insecure \ - --nodes \ + --nodes \ --file _out/controlplane.yaml ``` -`--insecure` is required for this one command only: the node has no -certificates yet, so talosctl can't authenticate to it normally. This is -the one moment during setup where you're trusting the network — do this on -a network you control. The node will reboot, install Talos to its system -disk, format and mount its second disk for Longhorn, and come back up -running the configuration you gave it — now *with* certificates, so every -command after this uses the secure path. +For each Proxmox worker, confirm `/dev/sda` is the Talos system disk and +`/dev/sdb` is the dedicated Longhorn disk selected by +`talos/patch-longhorn.yaml`, then apply the worker configuration: -Repeat this for ``, ``, and `` before moving on. +```sh +talosctl apply-config --insecure \ + --nodes \ + --file _out/worker.yaml +``` + +`--insecure` is required for this one command only: the node has no +certificates yet, so talosctl cannot authenticate to it normally. This is +the one moment during setup where you trust the local network. After applying +configuration, the node installs Talos and returns with authenticated API +access. Configure all three control planes and both workers before continuing. ## 4. Bootstrap etcd — once, total @@ -115,12 +118,14 @@ existing one. Running it again against an existing cluster can corrupt it. ## 5. Get your kubeconfig ```sh -talosctl kubeconfig --nodes . +talosctl kubeconfig --nodes . ``` -Downloads a kubeconfig into the current directory (also gitignored — -it grants full cluster access), pointed at the VIP rather than any one -node. Merge it with `~/.kube/config` or point `KUBECONFIG` at it directly: +Downloads a kubeconfig through a control-plane node's Talos API into the +current directory (also gitignored — it grants full cluster access). The +resulting Kubernetes configuration points at the cluster VIP rather than one +specific control plane. Merge it with `~/.kube/config` or point `KUBECONFIG` +at it directly: ```sh export KUBECONFIG=$PWD/kubeconfig @@ -132,15 +137,17 @@ export KUBECONFIG=$PWD/kubeconfig kubectl get nodes ``` -Expect to see **three** nodes, `Ready`, within a couple of minutes (it -takes a moment for the default CNI — Talos ships Flannel out of the box — -to come up and clear each node's networking checks). +Expect to see **five** nodes, `Ready`, within a couple of minutes (it takes +a moment for the default CNI — Talos ships Flannel out of the box — to clear +each node's networking checks). -``` +```text NAME STATUS ROLES AGE VERSION talos-... Ready control-plane 90s v1.3x.x talos-... Ready control-plane 85s v1.3x.x talos-... Ready control-plane 80s v1.3x.x +talos-... Ready worker 75s v1.3x.x +talos-... Ready worker 70s v1.3x.x ``` If any node is stuck on `NotReady`, give it another minute before diff --git a/talos/patch-controlplane.yaml b/talos/patch-controlplane.yaml index e574b14..d5f4adc 100644 --- a/talos/patch-controlplane.yaml +++ b/talos/patch-controlplane.yaml @@ -12,9 +12,10 @@ # --- install image --- # All three control planes are Raspberry Pi 4 systems currently installed on # their microSD (`mmcblk0`). The Image Factory schematic contains the -# `rpi_generic` ARM64 overlay needed to boot this hardware. Keep this custom -# installer reference during upgrades or the board-specific boot support may -# be lost. +# `rpi_generic` ARM64 overlay needed to boot this hardware and the NetBird +# system extension used for private overlay connectivity. Keep this custom +# installer reference during upgrades or the board-specific boot support and +# NetBird binary may be lost. # # --- vip --- # With three control-plane nodes, "the cluster's API server" isn't one @@ -38,7 +39,8 @@ cluster: machine: install: disk: /dev/mmcblk0 - image: factory.talos.dev/metal-installer/ee21ef4a5ef808a9b7484cc0dda0f25075021691c8c09a276591eedb638ea1f9:v1.13.8 + image: "factory.talos.dev/metal-installer/\ + d093bb68c464b87cc637765535dde5f68fbb040da44280617dd11a42d6c03988:v1.13.8" network: interfaces: # Raspberry Pi 4's built-in Ethernet interface is named `end0` by diff --git a/talos/patch-netbird.yaml b/talos/patch-netbird.yaml new file mode 100644 index 0000000..889613e --- /dev/null +++ b/talos/patch-netbird.yaml @@ -0,0 +1,20 @@ +# Persistent, non-secret configuration for the official SideroLabs NetBird +# system extension. Apply this patch to every Talos node alongside its normal +# role-specific patches. +# +# Enrollment still requires NB_SETUP_KEY, but that short-lived credential must +# never be committed to this public repository. Supply it separately from a +# silent shell prompt during first enrollment, revoke it after all nodes join, +# then remove it from the live ExtensionServiceConfig. +apiVersion: v1alpha1 +kind: ExtensionServiceConfig +name: netbird +environment: + # Both endpoints use this installation's combined self-hosted NetBird URL. + - NB_MANAGEMENT_URL=https://netbird.ahmza.com + - NB_ADMIN_URL=https://netbird.ahmza.com + # Talos mounts /etc read-only. Prevent NetBird from repeatedly trying to + # rewrite resolv.conf and the OpenSSH client configuration there. Overlay + # networking remains active; NetBird-managed DNS is deliberately disabled. + - NB_DISABLE_DNS=true + - NB_DISABLE_SSH_CONFIG=true diff --git a/talos/patch-worker.yaml b/talos/patch-worker.yaml index 9c5df66..280a024 100644 --- a/talos/patch-worker.yaml +++ b/talos/patch-worker.yaml @@ -8,9 +8,9 @@ # claiming the wrong virtual disk. # # The custom Image Factory installer contains Longhorn's `iscsi-tools` and -# `util-linux-tools` extensions plus `qemu-guest-agent`. Keep this schematic -# reference during future upgrades; switching to the stock installer would -# silently remove those extensions. +# `util-linux-tools` extensions plus `qemu-guest-agent` and NetBird. Keep this +# schematic reference during future upgrades; switching to the stock installer +# would silently remove those extensions. # # Both VMs use `ens18`. Their DHCP-reserved MAC addresses keep worker01 at # 10.10.10.203 and worker02 at 10.10.10.204 while DHCP supplies their gateway @@ -26,7 +26,8 @@ machine: node.longhorn.io/create-default-disk: "true" install: disk: /dev/sda - image: factory.talos.dev/metal-installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.13.8 + image: "factory.talos.dev/metal-installer/\ + 81fb1af7c03952593ba459c6c8b7fcc1268b6fe0cc80e6035ac629a741c183cc:v1.13.8" network: interfaces: - interface: ens18