ahmza.com/vps-k8s
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move to a three-node cluster with a Talos VIP; add Longhorn

Browse Source 
Every node is now both control plane and worker, giving real tolerance
for losing any single machine (etcd quorum needs 3, not 1). A Talos VIP
gives the cluster one stable address regardless of which node answers.

Also pre-installs Longhorn for replicated storage, backed by a Talos
UserVolume on each node's second disk (>=900GB, non-system) rather than
the default path — mirroring the pattern from the homescale repo.

Docs renumbered to make room for 06-longhorn.md; checklist, glossary,
network map, and CLAUDE.md's guided-bootstrap instructions updated to
match, including asking the user for node IPs, VIP, and disk info before
generating any Talos config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Maxwell Morris
2026-08-22 18:25:38 +01:00
co-authored by Claude Sonnet 5
parent 92c3ad3191
commit 110bd97cca
26 changed files with 777 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+25 -13
View File
@@ -59,7 +59,16 @@ end to end" — drive [`docs/checklist.md`](docs/checklist.md) interactively:
6. Periodically remind them of `docs/glossary.md` and the "Ask Claude"
checkpoints along the way — this mode shouldn't feel like a script being
read at them.
7. **Phase 5 (MetalLB) needs real information from the user, not
7. **The Talos phase (docs/02) needs real information too, before any
config is generated.** Ask: what are the three nodes' IP addresses (or
DHCP reservations), and — separately — what unused address should the
cluster's VIP use? The VIP is a fourth address, distinct from all three
node IPs and from the MetalLB pool (below); it needs the same "ask about
their LAN and DHCP range" treatment as MetalLB. Also confirm each node
actually has a second disk ≥900GB before they generate config — if not,
that's a real constraint on `talos/patch-longhorn.yaml`'s `minSize`, not
something to paper over with a placeholder.
8. **The MetalLB phase (docs/05) needs real information from the user, not
placeholders.** Before editing `apps/metallb/templates/ipaddresspool.yaml`
or `apps/demo/service.yaml`, actually ask: what does their LAN's subnet
look like, and what IP range does their router's DHCP server already
@@ -80,10 +89,12 @@ useful if it's actually current, so treat updating it as part of the
change itself, not an afterthought:
- Whenever you (Claude) assign, learn, or change a real IP address for
something — walking the user through MetalLB's pool
([`docs/05-metallb.md`](docs/05-metallb.md)), patching `argocd-server`,
helping add a new app with a `LoadBalancer` Service — update the table in
`docs/network.md` in the same breath. Don't wait to be asked.
something — the three node addresses and VIP in
[`docs/02-talos-bootstrap.md`](docs/02-talos-bootstrap.md), walking the
user through MetalLB's pool ([`docs/05-metallb.md`](docs/05-metallb.md)),
patching `argocd-server`, helping add a new app with a `LoadBalancer`
Service — update the table in `docs/network.md` in the same breath.
Don't wait to be asked.
- Every row needs a full, correct URL — scheme, IP, and port — not just a
bare address. Include the port number unless it's the protocol's default
(80 for `http://`, 443 for `https://`). If you're not sure what port a
@@ -101,7 +112,7 @@ Point new users here first if they seem unsure where to start:
1. [`README.md`](README.md) — orientation
2. [`docs/00-what-is-all-this.md`](docs/00-what-is-all-this.md) — mental model, read before touching a terminal
3. [`docs/01-prerequisites.md`](docs/01-prerequisites.md) → [`08-growing-up.md`](docs/08-growing-up.md) — in order
3. [`docs/01-prerequisites.md`](docs/01-prerequisites.md) → [`docs/09-growing-up.md`](docs/09-growing-up.md) — in order
4. [`docs/checklist.md`](docs/checklist.md) — condensed step-by-step reference once concepts are understood, or to drive Guided Bootstrap Mode
5. [`docs/glossary.md`](docs/glossary.md) — reference throughout, not read start to finish
@@ -116,17 +127,18 @@ If asked to change manifests or docs rather than explain them:
`argocd/`.
- Keep the moving parts minimal. Ingress, cert-manager, and secrets
management are intentionally **documented, not shipped** (see
[`docs/08-growing-up.md`](docs/08-growing-up.md)) — don't add them to
make the repo "more complete" without being asked. MetalLB is the one
exception, shipped deliberately because bare metal has no other way to
give a Service a real IP address — don't take that as license to add
other addons the same way without being asked.
[`docs/09-growing-up.md`](docs/09-growing-up.md)) — don't add them to
make the repo "more complete" without being asked. MetalLB and Longhorn
are the two exceptions, shipped deliberately because bare metal has no
other way to give a Service a real IP address or give an app real
persistent storage — don't take that as license to add other addons the
same way without being asked.
- This repo is **public**. Never write a plaintext secret, credential, or
private key into any file here. If a change would need one, stop and
point to the Sealed Secrets / SOPS guidance in
`docs/08-growing-up.md` instead of improvising.
`docs/09-growing-up.md` instead of improvising.
- New apps follow the existing `apps/<name>.yaml` + `apps/<name>/` pattern
— see [`docs/07-adding-new-apps.md`](docs/07-adding-new-apps.md) before
— see [`docs/08-adding-new-apps.md`](docs/08-adding-new-apps.md) before
adding one. `apps/metallb/` is the reference example for a Helm-based
one.
- Every `apps/*.yaml` Application manifest needs its own `repoURL` fixed to
README.md
+15 -10
View File
@@ -1,7 +1,7 @@
# GitOps Starter Pack
A minimal, real, working GitOps setup for a single Kubernetes cluster —
built to teach, not just to run. If you've never touched Kubernetes,
A minimal, real, working GitOps setup for a three-node Kubernetes cluster
— built to teach, not just to run. If you've never touched Kubernetes,
GitOps, Talos, or Argo CD before, this is meant to get you from a bare
machine to a working cluster you actually understand, not just one you
copy-pasted into existence.
@@ -35,6 +35,9 @@ shell history. This repo takes a different approach:
5. **MetalLB** gives apps in this cluster real IP addresses on your own
network — something a cloud provider would normally do for you, and
bare metal doesn't have on its own.
6. **Longhorn** does the same for storage — turning a spare disk on each
node into one pool of storage that's replicated across all three
nodes, so losing a machine doesn't mean losing its data.
Git becomes the one place that's actually true. If you want the full
picture with diagrams before doing anything, start with
@@ -46,13 +49,14 @@ picture with diagrams before doing anything, start with
talos/ → Talos machine config + bootstrap guide (the OS layer)
argocd/ → Argo CD install + the one manifest you apply by hand
apps/ → Everything Argo CD manages from here on — demo app, MetalLB,
and wherever you add new apps
Longhorn, and wherever you add new apps
docs/ → The numbered, progressive guide you should actually read
```
## Prerequisites
- One machine (physical or VM) to become your cluster node
- Three machines (physical or VM), each with a second disk ≥900GB, to
become your cluster's three nodes
- `talosctl`, `kubectl`, and the `argocd` CLI installed locally
- A GitHub account, and your own fork of this repo
@@ -68,11 +72,12 @@ Read these in order. Each is short — one concept per file.
4. [`docs/03-installing-argocd.md`](docs/03-installing-argocd.md) — install Argo CD into the cluster
5. [`docs/04-app-of-apps.md`](docs/04-app-of-apps.md) — apply the one manifest that hands control to Git
6. [`docs/05-metallb.md`](docs/05-metallb.md) — give apps real IP addresses on your network
7. [`docs/06-your-first-change.md`](docs/06-your-first-change.md) — edit a file, push, watch it deploy itself
8. [`docs/07-adding-new-apps.md`](docs/07-adding-new-apps.md) — add something of your own
9. [`docs/08-growing-up.md`](docs/08-growing-up.md) — next steps: more nodes, ingress, secrets management
10. [`docs/glossary.md`](docs/glossary.md) — every term, plain English, for reference throughout
11. [`docs/network.md`](docs/network.md) — the running log of every real IP address in your cluster, kept current as you go
7. [`docs/06-longhorn.md`](docs/06-longhorn.md) — give apps real, replicated storage
8. [`docs/07-your-first-change.md`](docs/07-your-first-change.md) — edit a file, push, watch it deploy itself
9. [`docs/08-adding-new-apps.md`](docs/08-adding-new-apps.md) — add something of your own
10. [`docs/09-growing-up.md`](docs/09-growing-up.md) — next steps: more nodes, ingress, secrets management
11. [`docs/glossary.md`](docs/glossary.md) — every term, plain English, for reference throughout
12. [`docs/network.md`](docs/network.md) — the running log of every real IP address in your cluster, kept current as you go
Prefer a single condensed list you can work through step by step with
Claude, checking items off as you go? Use
@@ -101,7 +106,7 @@ moving on.
This repo deliberately does **not** ship ingress, TLS/cert-manager, or
secrets management in v1 — they're real next steps, documented in
[`docs/08-growing-up.md`](docs/08-growing-up.md), left out so the core
[`docs/09-growing-up.md`](docs/09-growing-up.md), left out so the core
loop stays small enough to fully understand first. This repo is also
**public** — never commit plaintext secrets, credentials, or private keys
to it; see that same doc for how to handle secrets properly once you need
apps/demo.yaml
+1 -1
View File
@@ -16,7 +16,7 @@
# (e.g. a database or ingress controller from a public chart repo). Argo CD
# detects a Helm chart automatically when it finds a Chart.yaml at the
# source path, resolves its dependencies, and syncs the rendered output just
# like any other app. See docs/07-adding-new-apps.md for a worked example —
# like any other app. See docs/08-adding-new-apps.md for a worked example —
# the demo app deliberately stays plain manifests so the GitOps loop is easy
# to read end to end with no templating in the way.
apiVersion: argoproj.io/v1alpha1
apps/demo/configmap.yaml
+1 -1
View File
@@ -4,7 +4,7 @@
# baking the page into a custom container image.
#
# Why this matters for the GitOps story: this is the file you'll edit in
# docs/06-your-first-change.md. Change the text here, commit, push, and
# docs/07-your-first-change.md. Change the text here, commit, push, and
# watch Argo CD notice the difference in Git and push it into the cluster
# for you — no `kubectl` command involved. That's the whole loop, proven
# with one line of HTML.
apps/demo/deployment.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
# serving this ConfigMap's index.html") and hands it to Kubernetes, whose
# job is to continuously make reality match that description. If the Pod
# crashes, Kubernetes restarts it. If the node dies, Kubernetes (given more
# than one node — see docs/08-growing-up.md) reschedules it elsewhere.
# than one node — see docs/09-growing-up.md) reschedules it elsewhere.
# You never tell Kubernetes "start a container" directly; you describe the
# end state and it does the work. This is what "declarative" means.
apiVersion: apps/v1
apps/longhorn.yaml
+36
View File
@@ -0,0 +1,36 @@
# Longhorn — this repo's pre-installed storage layer. Same pattern as
# apps/metallb.yaml: an Application pointing at a Helm chart
# (apps/longhorn/) that pulls in the official upstream chart. Once this
# syncs, any app in this cluster can request persistent storage (a
# PersistentVolumeClaim) and actually get real, replicated storage backed
# by the dedicated disk each node prepared in
# talos/patch-longhorn.yaml — see docs/06-longhorn.md.
#
# Longhorn needs privileged Pods (it manages block devices directly) and
# the `open-iscsi`/`util-linux-tools` Talos system extensions baked into
# your node image — both covered in docs/01-prerequisites.md and
# docs/02-talos-bootstrap.md. Skipping either means Longhorn's Pods will
# sit unhealthy no matter how correctly this Application syncs.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: longhorn
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/YOUR_GITHUB_USERNAME/k8s-starter-pack.git
targetRevision: main
path: apps/longhorn
destination:
server: https://kubernetes.default.svc
namespace: longhorn-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
apps/longhorn/Chart.yaml
+18
View File
@@ -0,0 +1,18 @@
# An umbrella chart, same pattern as apps/metallb/Chart.yaml: it declares a
# dependency on the upstream Longhorn chart and carries no templates of its
# own — everything Longhorn needs (its CRDs, controllers, UI, default
# StorageClass) comes from the dependency. See
# docs/08-adding-new-apps.md for more on this pattern.
apiVersion: v2
name: longhorn-app
description: Installs Longhorn, this repo's pre-installed distributed block storage
version: 0.1.0
dependencies:
- name: longhorn
# A version range rather than an exact pin — see the same note in
# apps/metallb/Chart.yaml. Check
# https://artifacthub.io/packages/helm/longhorn/longhorn for the
# current release and pin an exact version for a fully reproducible
# install.
version: ">=1.7.0"
repository: https://charts.longhorn.io
apps/longhorn/values.yaml
+22
View File
@@ -0,0 +1,22 @@
# Values for the upstream Longhorn chart, nested under `longhorn:` (the
# dependency's name in Chart.yaml).
longhorn:
defaultSettings:
# Must match the mount path from the UserVolumeConfig in
# talos/patch-longhorn.yaml (/var/mnt/<name>) — this is what points
# Longhorn at the dedicated data disk instead of its own default path
# on the root filesystem, which on Talos is ephemeral/unsuited for
# real data anyway.
defaultDataPath: /var/mnt/longhorn
# Longhorn's own headroom settings, kept conservative: stop scheduling
# new replicas onto a disk once free space drops below these
# thresholds, so the cluster degrades gracefully instead of actually
# filling a disk.
storageMinimalAvailablePercentage: 25
storageReservedPercentageForDefaultDisk: 30
preUpgradeChecker:
# Longhorn's own pre-upgrade check runs as a Job that expects to reach
# an upgrade-responder service on the internet by default. Harmless to
# leave on, but pointless for a learning cluster and one less thing to
# explain — off by default here.
jobEnabled: false
apps/metallb/Chart.yaml
+1 -1
View File
@@ -5,7 +5,7 @@
# exists at this path, resolves the dependency, and syncs the combined
# rendered output as one Application — exactly like any other app in
# apps/. This is the real, working example of the Helm-based app pattern
# mentioned in docs/07-adding-new-apps.md: use it as a template if you want
# mentioned in docs/08-adding-new-apps.md: use it as a template if you want
# to bring in another third-party chart later.
apiVersion: v2
name: metallb-app
argocd/README.md
+2 -2
View File
@@ -105,8 +105,8 @@ argocd app get app-of-apps
argocd app list
```
You should see `app-of-apps` sync, then `demo` and `metallb` Applications
appear underneath it, then sync themselves. See
You should see `app-of-apps` sync, then `demo`, `metallb`, and `longhorn`
Applications appear underneath it, then sync themselves. See
[`docs/04-app-of-apps.md`](../docs/04-app-of-apps.md) for what's happening
here and [`docs/checklist.md`](../docs/checklist.md) for the full guided
path.
docs/00-what-is-all-this.md
+46 -33
View File
@@ -19,17 +19,20 @@ the real cluster match it. You stop making changes by hand. You edit a file,
push it, and a robot applies it for you — the same way every time, with a
record of every change in `git log`.
## The five pieces, in plain terms
## The six pieces, in plain terms
**Kubernetes** is software that runs on one or more machines and manages
containers for you: starts them, restarts them if they crash, and moves
them to a different machine if one dies (once you have more than one — see
[`docs/08-growing-up.md`](08-growing-up.md)). You describe the end state you
want ("run one copy of this"); Kubernetes' job is to make that true and keep
it true. This is what people mean by **declarative**: you state the
destination, not the steps to get there.
them to a different machine if one dies. You describe the end state you
want ("run one copy of this"); Kubernetes' job is to make that true and
keep it true. This is what people mean by **declarative**: you state the
destination, not the steps to get there. This repo runs three machines
from the start, specifically so that "moves them to a different machine
if one dies" is something you actually get, not just a fact about
Kubernetes in general — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md) for why three.
**Talos Linux** is the operating system your machine runs. It's built for
**Talos Linux** is the operating system each machine runs. It's built for
exactly one purpose — running Kubernetes — and nothing else. There's no SSH,
no shell, no package manager to fiddle with by hand. You configure it by
sending it a config file over the network (a "machine config"), the same
@@ -47,15 +50,23 @@ manages. Rather than one Application resource pointing at everything, one
Application — the app-of-apps — watches a folder (`apps/`) full of *other*
Application definitions, each pointing at its own small piece. Add a new
app by adding a new file — see
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md).
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
**MetalLB** solves a specific problem: on a cloud provider, asking for a
public IP address for a service is one API call away; on your own bare
metal machine, nothing hands those out automatically. MetalLB runs inside
metal machines, nothing hands those out automatically. MetalLB runs inside
the cluster and fills that gap, so apps here can get a real, dedicated IP
address on your network instead of an awkward fixed port. It's pre-installed
in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
**Longhorn** solves the equivalent problem for storage: a cloud provider
gives you durable, replicated disks on demand; bare metal just gives you
whatever's physically bolted into each machine. Longhorn turns a plain
disk on each of your three nodes into one pool of replicated storage any
app can request a slice of — and because it's replicated *across* nodes,
losing one machine doesn't lose the data either. Also pre-installed, not a
"next step" — see [`docs/06-longhorn.md`](06-longhorn.md).
## Architecture — the shape of what you'll build
```
@@ -64,9 +75,11 @@ in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
│ repo, on GitHub) │
│ │
│ apps/demo.yaml │
│ apps/demo/*.yaml │──┐
│ apps/metallb.yaml │ │ watched by
│ apps/metallb/* │ │
│ apps/demo/*.yaml │
│ apps/metallb.yaml │──┐
│ apps/metallb/* │ │ watched by
│ apps/longhorn.yaml │ │
│ apps/longhorn/* │ │
└─────────────────────────┘ │
▼
┌───────────────────┐
@@ -76,29 +89,29 @@ in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
└─────────┬──────────┘
│ applies manifests
▼
┌─────────────────────────────┐
│ Talos Linux node │
│ (control plane + worker, │
│ single node) │
│ │
│ ┌────────────┐ ┌──────────┐ │
│ │demo │ metallb- │ │
│ │namespace: │ system: │ │
│ │nginx Pod ⇄ │ hands out │ │
│ │Service │ a real IP │ │
│ │(LoadBalancer)◄─address to │ │
│ │ │ that Service│ │
│ └────────────┘ └──────────┘ │
└─────────────────────────────┘
┌───────────────────────────────────────────────┐
│ Three Talos Linux nodes (VIP: <VIP_IP>) │
│ each: control plane + worker + etcd peer │
│ │
│ demo namespace: metallb-system: │
│ nginx Pod ⇄ Service hands out a real IP │
│ (LoadBalancer) ◄───────address to that Service │
│ │
│ longhorn-system: each node's 2nd disk │
│ replicates data (≥900GB, UserVolume) │
│ across all 3 nodes ◄──mounted at │
│ /var/mnt/longhorn │
└───────────────────────────────────────────────┘
```
Everything below the Git repo is a *consequence* of what's in Git. If the
box labelled "Talos Linux node" burned to the ground and you rebuilt it from
scratch, replaying this repo against it would recreate everything except the
raw OS bootstrap (docs/02), the one-time Argo CD install (docs/03), and the
handful of network-specific addresses you choose in docs/05 — all
deliberately kept outside the loop, since something has to start it and
nothing in Git can know your LAN's IP layout for you.
Everything below the Git repo is a *consequence* of what's in Git. If any
one of the three node boxes burned to the ground and you replaced that
machine, replaying this repo against it would recreate everything on it
except the raw OS bootstrap (docs/02) and the one-time Argo CD install
(docs/03) — both deliberately kept outside the loop, since something has
to start it. The handful of network-specific addresses you choose along
the way (the VIP, the MetalLB pool) live in `docs/network.md`, not in Git
history alone, since nothing in Git can know your LAN's IP layout for you.
## The reconciliation loop
docs/01-prerequisites.md
+56 -21
View File
@@ -5,31 +5,62 @@ PC, or an amd64 VM). If your hardware is arm64 (Raspberry Pi, Apple Silicon
VM), the tools below all publish arm64 builds too; swap the architecture in
whichever install command you use and the rest of this guide is unchanged.
## 1. A target machine
## 1. Three target machines
You need one machine to become your cluster node — a spare physical PC, a
mini PC, or a VM (VirtualBox, UTM, Proxmox, or any hypervisor that can boot
from an ISO). Minimum realistic spec: 2 CPU cores, 4GB RAM, 20GB disk.
This repo builds a **three-node cluster**, with every node acting as both
control plane (the brain: API server, scheduler, the cluster's own
database) and worker (runs your actual apps) at once. Three, not one,
because it's the smallest number where losing any single machine doesn't
take the whole cluster down — see
[`docs/00-what-is-all-this.md`](00-what-is-all-this.md) for why three
specifically, not two or four.
It needs to be reachable over the network from your everyday machine (the
one you'll run `talosctl` and `kubectl` from), and it needs a stable IP
address — see the note about DHCP reservations in
[`talos/README.md`](../talos/README.md).
You need three machines — spare physical PCs, mini PCs, or VMs (VirtualBox,
UTM, Proxmox, or any hypervisor that can boot from an ISO). Minimum
realistic spec per machine: 2 CPU cores, 4GB RAM, plus **two disks each**:
one for Talos itself (20GB+ is plenty), and a second, separate disk of
**at least 900GB** that will be handed entirely to Longhorn
(see [`docs/06-longhorn.md`](06-longhorn.md)) for storage. If your spare
hardware doesn't have a second disk that large, that's a real constraint —
ask Claude about adjusting the size threshold in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) to fit what you
actually have.
You do **not** need to install an OS on it yet. That's what the next doc
covers — Talos gets installed by booting a special ISO and sending it
configuration over the network, not through a normal installer.
Download the Talos amd64 ISO from the
[Talos releases page](https://github.com/siderolabs/talos/releases) (look
for `metal-amd64.iso` under the latest release) and boot your target machine
from it. Leave it sitting at the boot screen — you'll configure it in
Each machine needs to be reachable over the network from your everyday
machine (the one you'll run `talosctl` and `kubectl` from), and each needs
its own stable IP address — see the note about DHCP reservations in
[`talos/README.md`](../talos/README.md). You'll also need one *additional*
unused address for the cluster's shared VIP (Virtual IP) — more on that in
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
> **Ask Claude:** if you're not sure whether to use a VM or spare physical
You do **not** need to install an OS on any of them yet. That's what the
next doc covers — Talos gets installed by booting a special ISO and
sending it configuration over the network, not through a normal installer.
### Building your Talos image
Stock Talos doesn't include everything Longhorn needs — specifically, the
`iscsi-tools` and `util-linux-tools` **system extensions** (optional
add-ons baked into the boot image itself, since Talos has no package
manager to install them onto a running node). You build a custom image,
called a **schematic**, once, and use it for all three nodes:
1. Go to [factory.talos.dev](https://factory.talos.dev).
2. Under system extensions, select `siderolabs/iscsi-tools` and
`siderolabs/util-linux-tools`.
3. Pick the current stable Talos version and `metal-amd64` as the platform.
4. Download the resulting ISO (the site calls this a "schematic" —
your own custom Talos image, identified by a schematic ID).
Boot **all three** machines from this same ISO, and leave each sitting at
its boot screen — you'll configure them in
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
> **Ask Claude:** if you're not sure whether to use VMs or spare physical
> hardware, ask — the tradeoffs (snapshots and easy resets vs. more
> realistic networking) are worth five minutes of discussion before you
> commit.
> realistic networking, and whether three VMs on one physical host really
> gives you the resilience three nodes are supposed to buy you) are worth
> five minutes of discussion before you commit.
## 2. talosctl
@@ -101,8 +132,12 @@ that references a repo URL means your fork's URL.
## Checklist
- [ ] Target machine chosen, network-reachable, IP address noted
- [ ] Talos ISO downloaded, machine booted from it and idle at the boot screen
- [ ] Three target machines chosen, each network-reachable, each with a
stable IP address noted, each with a second disk ≥900GB for Longhorn
- [ ] One additional unused IP address set aside for the cluster's VIP
- [ ] Custom Talos schematic built at factory.talos.dev with `iscsi-tools`
and `util-linux-tools`, ISO downloaded
- [ ] All three machines booted from that ISO, idle at the boot screen
- [ ] `talosctl version --client` works
- [ ] `kubectl version --client` works
- [ ] `argocd version --client` works
docs/02-talos-bootstrap.md
+83 -31
View File
@@ -1,7 +1,8 @@
# 02 — Bootstrapping Talos
At the end of this doc, you'll have a single-node Kubernetes cluster,
running nothing but Kubernetes itself. No Argo CD yet — that's next.
At the end of this doc, you'll have a three-node Kubernetes cluster,
running nothing but Kubernetes itself (plus the storage prep Longhorn
needs later). No Argo CD yet — that's next.
## Why Talos, specifically
@@ -17,44 +18,92 @@ same declarative way Kubernetes applies your app manifests. Less surface
area, less to secure, less to keep in sync by hand — the same philosophy as
the rest of this stack, one layer lower.
## Single node, on purpose
## Three nodes, each both control plane and worker
A "real" Kubernetes cluster usually has multiple nodes — some running the
**control plane** (the brain: API server, scheduler, the cluster's
database) and others as **workers** (running your actual apps), so that
losing one machine doesn't take the whole cluster down. This repo starts
with exactly one node acting as both, because that's the cheapest way to
learn the concepts that matter for GitOps without needing multiple
machines. [`docs/08-growing-up.md`](08-growing-up.md) covers adding more
nodes once you're comfortable — the config here is deliberately compatible
with that path, not a dead end.
A "real" Kubernetes cluster has multiple nodes — some running the
**control plane** (the brain: API server, scheduler, the cluster's own
database, called etcd) and others as **workers** (running your actual
apps) — so that losing one machine doesn't take the whole cluster down.
This repo starts with three nodes, each acting as *both* control plane and
worker, which gets you real resilience — survive losing any one machine —
without needing more than three.
Why three specifically, and not two: etcd (the control plane's database)
uses **quorum** — a majority vote among its members — to agree on cluster
state, and a majority only means something with an odd number of voters.
With one control-plane node, losing it loses the cluster outright. Two
doesn't help: losing either one leaves one survivor, which is not a
majority of two. Three is the smallest number where losing any *one*
still leaves two — a majority — able to keep agreeing on state. This is
also why [`docs/09-growing-up.md`](09-growing-up.md) — once you're past
this doc — specifically warns against going to four control-plane nodes
later: four doesn't improve on three's fault tolerance, it just makes
quorum math worse.
## The VIP: one address for a cluster that moves
With three control-plane nodes, "where's the API server?" no longer has
one fixed answer — it's whichever nodes happen to be up right now. Talos's
**VIP** (Virtual IP) feature solves this: the three nodes continuously
elect one of themselves to answer on one shared IP address you choose, and
re-elect within seconds if that node goes down. Everything that talks to
this cluster — `talosctl`, `kubectl`, the Application manifests in this
repo — points at the VIP, never at one node's own address. See the
`vip` block in
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml).
> **Ask Claude:** "help me pick the VIP address for my cluster" before you
> generate any config — same kind of decision as the MetalLB pool in
> [`docs/05-metallb.md`](05-metallb.md): a real, unused address on your
> LAN, outside your DHCP range, that Claude can't guess on its own.
## Preparing storage for Longhorn
Each node also gets a second disk reserved for
[Longhorn](06-longhorn.md), this repo's pre-installed storage layer —
handled by [`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml),
applied at the same time as the control-plane patch below. Read that
file's comments; the short version is that Talos finds, formats, and
mounts a disk for Longhorn on its own, using rules you set (not the system
disk, at least 900GB), so there's no manual partitioning step here.
## Steps
Full exact commands live in [`talos/README.md`](../talos/README.md) — read
that alongside this doc; this section explains what's happening at each
stage, that one has copy-paste blocks.
stage, that one has copy-paste blocks. You'll repeat most of these steps
**three times**, once per node — `talosctl README.md` calls out exactly
which commands are "once" vs. "once per node."
1. **Generate machine configuration.** `talosctl gen config` produces
Talos's config files for your cluster, including
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml) —
read that file, it's short and explains itself — which lets your one
node run workloads as well as the control plane.
Talos's config files for your cluster, with both
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml)
(scheduling + the VIP) and
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) (storage
prep) baked in — read both files, they explain themselves. This
produces one `controlplane.yaml`, applied identically to all three
nodes; there's no separate worker config in this setup, since every
node is both.
2. **Apply the config to your node.** The node, freshly booted from the
Talos ISO, is sitting in "maintenance mode" with no configuration.
`talosctl apply-config` sends it the config generated in step 1. The
node reboots, installs Talos to its disk, and comes up running for real.
2. **Apply the config to each node.** Each node, freshly booted from your
custom Talos ISO, is sitting in "maintenance mode" with no
configuration. `talosctl apply-config` sends it the config from step 1.
Run this once per node, pointed at that node's own address. Each node
reboots, installs Talos to its system disk, formats and mounts its
second disk for Longhorn, and comes up running for real.
3. **Bootstrap etcd.** `talosctl bootstrap` initializes the cluster's data
store. This is a one-time, one-node-ever operation — even in a
multi-node cluster you only ever bootstrap once.
store. This is a one-time, one-node-ever operation, run against exactly
one of your three nodes — even though all three will end up running
etcd, only one of them needs telling to *start* the cluster; the other
two join the existing one once it exists.
4. **Retrieve your kubeconfig.** `talosctl kubeconfig` downloads the
credentials you'll use to talk to Kubernetes with `kubectl` — the
Kubernetes equivalent of the `talosconfig` you got in step 1.
Kubernetes equivalent of the `talosconfig` you got in step 1. Point it
at the VIP, not at any one node.
5. **Verify.** `kubectl get nodes` should show one node, `Ready`.
5. **Verify.** `kubectl get nodes` should show all three nodes, `Ready`.
> **Ask Claude:** "what's the difference between a kubeconfig and a
> talosconfig, and why are they separate files?" is a good one to ask once
@@ -62,11 +111,14 @@ stage, that one has copy-paste blocks.
## Checklist
- [ ] `_out/controlplane.yaml`, `worker.yaml`, and `talosconfig` generated
- [ ] Config applied to the node, node has rebooted and installed Talos
- [ ] `talosctl bootstrap` run exactly once
- [ ] kubeconfig retrieved
- [ ] `kubectl get nodes` shows one `Ready` node
- [ ] VIP address chosen, with Claude's help, for your LAN
- [ ] `_out/controlplane.yaml` and `talosconfig` generated, with both
patches applied
- [ ] Config applied to **all three** nodes; each has rebooted, installed
Talos, and mounted its Longhorn disk
- [ ] `talosctl bootstrap` run exactly once, against exactly one node
- [ ] kubeconfig retrieved, pointed at the VIP
- [ ] `kubectl get nodes` shows all three nodes `Ready`
Once your node is `Ready`, continue to
Once all three nodes are `Ready`, continue to
[`docs/03-installing-argocd.md`](03-installing-argocd.md).
docs/03-installing-argocd.md
+1 -1
View File
@@ -46,7 +46,7 @@ each stage does:
> **Ask Claude:** "why is a port-forward good enough for now, and what
> would I need instead to reach this Argo CD from somewhere other than my
> own machine?" — a preview of the Ingress topic in
> [`docs/08-growing-up.md`](08-growing-up.md).
> [`docs/09-growing-up.md`](09-growing-up.md).
## Checklist
docs/04-app-of-apps.md
+41 -36
View File
@@ -22,30 +22,33 @@ new app.
argocd/app-of-apps/app-of-apps.yaml (you apply this ONE file, by hand, once)
│
│ watches apps/, non-recursively
├──────────────┬───────────────┐
▼ ▼ ▼
apps/demo.yaml apps/metallb.yaml (more, later — just add a file)
│ │
▼ ▼
apps/demo/*.yaml apps/metallb/* (the actual manifests / Helm chart)
├───────────────┬────────────────┬───────────────┐
▼ ▼ ▼ ▼
apps/demo.yaml apps/metallb.yaml apps/longhorn.yaml (more, later)
│ │ │
▼ ▼ ▼
apps/demo/*.yaml apps/metallb/* apps/longhorn/* (manifests / Helm charts)
```
Three layers, each watching the next. You only ever touch the bottom two —
the app-of-apps is set-and-forget. Notice `apps/metallb.yaml` is already
here — see [`docs/05-metallb.md`](05-metallb.md) right after this one.
Four layers wide, each watching the next. You only ever touch the bottom
two — the app-of-apps is set-and-forget. Notice `apps/metallb.yaml` and
`apps/longhorn.yaml` are already here — see
[`docs/05-metallb.md`](05-metallb.md) and
[`docs/06-longhorn.md`](06-longhorn.md) right after this one.
## Why `apps/` isn't recursive
Look at [`argocd/app-of-apps/app-of-apps.yaml`](../argocd/app-of-apps/app-of-apps.yaml)
— specifically `directory.recurse: false`. The `apps/` folder contains both
Application manifests (`apps/demo.yaml`, `apps/metallb.yaml`) *and*
subfolders holding the real workload manifests those Applications point to
(`apps/demo/*.yaml`, `apps/metallb/*`). Non-recursive means the
app-of-apps only ever looks at files directly inside `apps/`, never inside
its subfolders — so it only ever sees Application definitions, and leaves
the actual workload manifests to the Application that owns them. Without
this, the app-of-apps would try to apply raw Deployments and Helm chart
files itself, duplicating and conflicting with what those Applications are
Application manifests (`apps/demo.yaml`, `apps/metallb.yaml`,
`apps/longhorn.yaml`) *and* subfolders holding the real workload manifests
those Applications point to (`apps/demo/*.yaml`, `apps/metallb/*`,
`apps/longhorn/*`). Non-recursive means the app-of-apps only ever looks at
files directly inside `apps/`, never inside its subfolders — so it only
ever sees Application definitions, and leaves the actual workload
manifests to the Application that owns them. Without this, the
app-of-apps would try to apply raw Deployments and Helm chart files
itself, duplicating and conflicting with what those Applications are
already doing.
## Every Application needs its own `repoURL`
@@ -59,19 +62,19 @@ that has it, not just the one you apply by hand:
grep -rl YOUR_GITHUB_USERNAME .
```
You should see three files: `argocd/app-of-apps/app-of-apps.yaml`,
`apps/demo.yaml`, and `apps/metallb.yaml`. Edit `spec.source.repoURL` in
each to point at your fork.
You should see four files: `argocd/app-of-apps/app-of-apps.yaml`,
`apps/demo.yaml`, `apps/metallb.yaml`, and `apps/longhorn.yaml`. Edit
`spec.source.repoURL` in each to point at your fork.
## Steps
1. Fix `repoURL` in all three files (above).
1. Fix `repoURL` in all four files (above).
2. Commit and push — this part matters more than it looks: the
app-of-apps reads `apps/demo.yaml` and `apps/metallb.yaml` *from Git*,
not from your local edits, so they have to be pushed before you apply
anything.
app-of-apps reads `apps/demo.yaml`, `apps/metallb.yaml`, and
`apps/longhorn.yaml` *from Git*, not from your local edits, so they
have to be pushed before you apply anything.
```sh
git add argocd/app-of-apps/app-of-apps.yaml apps/demo.yaml apps/metallb.yaml
git add argocd/app-of-apps/app-of-apps.yaml apps/demo.yaml apps/metallb.yaml apps/longhorn.yaml
git commit -m "Point Applications at my fork"
git push
```
@@ -79,13 +82,15 @@ each to point at your fork.
```sh
kubectl apply -f argocd/app-of-apps/app-of-apps.yaml
```
4. Watch it sync, then watch `demo` and `metallb` appear underneath it:
4. Watch it sync, then watch `demo`, `metallb`, and `longhorn` appear
underneath it:
```sh
argocd app list
```
You should see `app-of-apps`, `demo`, and `metallb`, all `Synced` and
`Healthy` within a minute or two (MetalLB's own install takes a little
longer than the demo app).
You should see `app-of-apps`, `demo`, `metallb`, and `longhorn`, all
`Synced` and `Healthy` within a few minutes (MetalLB and Longhorn's own
installs take longer than the demo app — Longhorn especially, since it
has to find and prepare a disk on every node).
5. Confirm the demo app is actually running:
```sh
kubectl -n demo get pods,svc
@@ -95,23 +100,23 @@ each to point at your fork.
hands those out real addresses on bare metal until MetalLB is
configured, which is exactly the next doc.
If `app-of-apps`, `demo`, and `metallb` all show `Synced`/`Healthy`, the
loop — Git → Argo CD → cluster — is already proven end to end; you just
can't see the demo app in a browser yet.
If `app-of-apps`, `demo`, `metallb`, and `longhorn` all show
`Synced`/`Healthy`, the loop — Git → Argo CD → cluster — is already proven
end to end; you just can't see the demo app in a browser yet.
> **Ask Claude:** "walk me through exactly what happened, in order, from the
> moment I ran `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml` to
> these three Applications existing." Making sure you can narrate that
> these four Applications existing." Making sure you can narrate that
> chain yourself is the real checkpoint for this doc — everything from
> here on is a variation on it.
## Checklist
- [ ] `repoURL` fixed in all three files listed above
- [ ] `repoURL` fixed in all four files listed above
- [ ] Fix committed and pushed
- [ ] `app-of-apps` applied and shows `Synced`/`Healthy`
- [ ] `demo` and `metallb` Applications appeared automatically, both
`Synced`/`Healthy`
- [ ] `demo`, `metallb`, and `longhorn` Applications appeared
automatically, all `Synced`/`Healthy`
- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running
(Service `EXTERNAL-IP` still `<pending>` — expected for now)
docs/05-metallb.md
+2 -2
View File
@@ -34,7 +34,7 @@ Helm chart that does two things at once (see
- Pulls in the official upstream MetalLB chart as a dependency — this is
also the real, working example of a Helm-based app mentioned in
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md).
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
- Carries two templates of its own
([`apps/metallb/templates/`](../apps/metallb/templates)): an
`IPAddressPool` and an `L2Advertisement` — MetalLB's own custom
@@ -152,4 +152,4 @@ the only place this address gets recorded at all.
- [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser
- [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding
Continue to [`docs/06-your-first-change.md`](06-your-first-change.md).
Continue to [`docs/06-longhorn.md`](06-longhorn.md).
docs/06-longhorn.md
+100
View File
@@ -0,0 +1,100 @@
# 06 — Longhorn: real, replicated storage
The app-of-apps you applied in [`docs/04-app-of-apps.md`](04-app-of-apps.md)
already brought up `longhorn` alongside `demo` and `metallb`. This doc
explains what it does and how to actually use it — there's no address to
pick this time, unlike MetalLB; the interesting decisions already happened
back in [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
## Why this is needed at all
A Kubernetes Pod's own filesystem disappears the moment that Pod does —
fine for a stateless web server, useless for anything that needs to keep
data around (a database, uploaded files, anything you'd be upset to lose).
Kubernetes' answer is a **PersistentVolumeClaim (PVC)**: an app asks for
some amount of storage, and something has to actually provide it. On a
cloud provider, that something is the cloud's own disk service. On bare
metal — your three Talos nodes — nothing provides that on its own, the
same gap MetalLB fills for IP addresses rather than storage.
**Longhorn** is that provider: it turns a plain disk on each node into one
pool of storage, and — because your cluster has three nodes — it
replicates each volume across more than one of them. Lose a node, and the
data backing any volume that had a replica elsewhere is still there. This
is also why Longhorn is genuinely a **three-node** story: with one node,
"replicated storage" has nothing to replicate *to*.
## What's already running
**`apps/longhorn.yaml`** → one Application, pointing at `apps/longhorn`, a
Helm chart that pulls in the official upstream Longhorn chart (see
[`apps/longhorn/Chart.yaml`](../apps/longhorn/Chart.yaml)) — the same
umbrella-chart pattern as MetalLB, just without any extra templates of its
own this time.
The interesting configuration already happened at the Talos layer, back in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml): each node
found, formatted, and mounted its own ≥900GB disk at
`/var/mnt/longhorn`. [`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml)
points Longhorn's `defaultDataPath` at that exact path — this is the thing
that makes Longhorn use *that* disk instead of carving space out of each
node's root filesystem (small, and on Talos, not meant for this kind of
data anyway).
## Verify it's actually working
```sh
kubectl -n longhorn-system get pods
```
Expect a `longhorn-manager` and `longhorn-csi-plugin` Pod on each of your
three nodes, plus one `instance-manager` per node once a volume exists,
all `Running`.
```sh
kubectl get storageclass
```
Longhorn's chart creates a default `StorageClass` called `longhorn`
automatically — this is what makes it the thing Kubernetes reaches for
whenever an app asks for storage without naming one specifically.
## Try it: give the demo app real storage
The demo app's HTML currently lives in a ConfigMap (see
[`docs/07-your-first-change.md`](07-your-first-change.md)) — fine for a
few lines of text, but not how a real app would store real data. As a
hands-on check that Longhorn actually works, ask Claude to help you:
1. Add a small `PersistentVolumeClaim` to `apps/demo/`, requesting a
couple of gigabytes with no `storageClassName` set (so it uses
Longhorn's default).
2. Mount it into the demo Deployment somewhere harmless, like
`/usr/share/nginx/html/data`.
3. Push it, let Argo CD sync, and confirm:
```sh
kubectl -n demo get pvc
```
should show your claim `Bound`, and
```sh
kubectl -n longhorn-system get volumes
```
should show a matching volume, `healthy`, with more than one replica.
This isn't part of the checklist below — treat it as an optional exercise
once the pre-installed pieces are confirmed working, not a requirement to
move on.
> **Ask Claude:** "what would happen to that PVC's data if I lost one of
> my three nodes right now?" — worth being able to answer before you trust
> this with anything real.
## Checklist
- [ ] `kubectl -n longhorn-system get pods` shows manager/CSI Pods
`Running` on all three nodes
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
with a healthy, replicated Longhorn volume behind it
Continue to [`docs/07-your-first-change.md`](07-your-first-change.md).
docs/06-your-first-change.md → docs/07-your-first-change.md
+2 -2
View File
@@ -1,4 +1,4 @@
# 06 — Your first change
# 07 — Your first change
Everything so far proved Argo CD can deploy what's in Git. This doc proves
the part that actually matters day to day: changing what's in Git and
@@ -90,5 +90,5 @@ for the loop that just ran.
- [ ] Saw the new text in the browser
- [ ] (Optional) Watched self-healing revert a hand-made change
Continue to [`docs/07-adding-new-apps.md`](07-adding-new-apps.md) to add
Continue to [`docs/08-adding-new-apps.md`](08-adding-new-apps.md) to add
something of your own.
docs/07-adding-new-apps.md → docs/08-adding-new-apps.md
+9 -4
View File
@@ -1,4 +1,4 @@
# 07 — Adding a new app
# 08 — Adding a new app
This is the payoff for the app-of-apps structure: adding something new is
"drop a file here," not a special procedure.
@@ -32,8 +32,13 @@ To add a new app called, say, `whoami`:
`whoami.yaml`, creates the `whoami` Application, which in turn deploys
everything in `apps/whoami/`.
4. Confirm: `argocd app list` should show `whoami` alongside `demo` and
`metallb`.
4. Confirm: `argocd app list` should show `whoami` alongside `demo`,
`metallb`, and `longhorn`.
If your new app needs storage, request it with a plain
`PersistentVolumeClaim` and no `storageClassName` — Longhorn's chart set
itself as the cluster's default StorageClass, so it's used automatically.
See [`docs/06-longhorn.md`](06-longhorn.md).
## Manifest-based vs. Helm-based apps
@@ -83,5 +88,5 @@ address shows up in it.
- [ ] `argocd app list` shows the new app, `Synced`/`Healthy`
- [ ] If it has a `LoadBalancer` Service, its address is recorded in `docs/network.md`
Continue to [`docs/08-growing-up.md`](08-growing-up.md) for what to add
Continue to [`docs/09-growing-up.md`](09-growing-up.md) for what to add
next as this cluster grows beyond a learning setup.
docs/08-growing-up.md → docs/09-growing-up.md
+36 -19
View File
@@ -1,4 +1,4 @@
# 08 — Growing up: what to add next
# 09 — Growing up: what to add next
Everything up to here is deliberately minimal — enough to understand the
whole loop, nothing more. This doc is a map of the natural next steps, each
@@ -7,26 +7,43 @@ for you; each is its own project once you're ready.
## More nodes
Right now, one machine is both control plane and worker
(`allowSchedulingOnControlPlanes: true` in
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml)). That's
a single point of failure: if that machine dies, the cluster is gone.
Your three nodes already give you real resilience — losing any one still
leaves a majority of control-plane voters (see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker))
and, once Longhorn has real volumes, a replica of your data likely
survives too. Growing further is about *capacity*, not fault tolerance you
don't already have — so beyond this point, add plain **worker** nodes, not
more control-plane ones:
To grow beyond it:
1. Boot a second machine from the Talos ISO, same as the first.
2. Generate a **worker** config instead of a control-plane one — you
1. Boot a new machine from the same custom Talos ISO as the first three
(see [`docs/01-prerequisites.md`](01-prerequisites.md)).
2. Apply the **worker** config instead of the control-plane one — you
already have `_out/worker.yaml` from
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now.
3. `talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml`
4. `kubectl get nodes` should show two nodes shortly after.
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now:
```sh
talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
```
3. `kubectl get nodes` should show a fourth node shortly after, without
the `control-plane` role.
For real resilience against a control-plane node failing (not just adding
worker capacity), you'd want an odd number of control-plane nodes — 3 is
the normal minimum — because etcd (the cluster's data store) uses majority
voting to agree on state, and majority voting needs an odd number to avoid
ties. That's a bigger step than this doc covers in detail; ask Claude when
you're actually ready to take it.
### Why not a fourth control-plane node
It's tempting to reach for a fourth control-plane node when you're growing
a cluster you already made resilient with three — but it actually makes
things worse, not better. etcd (the control-plane database) needs a
**majority** of its members to agree before it can do anything, and
majority only means something clean with an odd number: with 3 members, 1
can fail and 2 (a majority) still agree — you can lose one. With 4, you
still need 3 to agree, so you can *still* only lose one — you paid for an
extra machine and gained nothing. Worse, 4 is *more* likely to have
exactly a 2-2 split during a network hiccup, which stalls the cluster
until it's resolved. If you ever do want more control-plane fault
tolerance than three nodes gives you, the next meaningful number is 5 —
not 4.
> **Ask Claude:** "walk me through the quorum math for 3 vs. 4 vs. 5
> control-plane nodes" if you want to see this worked out concretely
> rather than taking it on faith.
## Ingress
@@ -40,7 +57,7 @@ so big, and there's no way to route by hostname (`app1.example.com` vs
An **Ingress controller** is itself just another app you'd deploy the same
way as anything else in this repo (see
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md)) — commonly as a
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md)) — commonly as a
Helm-based app, since most ingress controllers are distributed that way,
same as MetalLB. It typically sits behind one single MetalLB
`LoadBalancer` Service (one IP, total, for every app), and you add an
docs/checklist.md
+37 -21
View File
@@ -1,25 +1,27 @@
# Guided bootstrap checklist
A single-page, ordered checklist covering the entire path from bare
machine to a working GitOps loop. It's meant to be worked through **with
machines to a working GitOps loop. It's meant to be worked through **with
Claude**, one item at a time — ask Claude to walk you through this list,
and it will run each step with you, wait for your output, confirm it
worked, and only then move to the next one. See `CLAUDE.md` at the repo
root for how Claude is primed to do this.
Each item names the doc it's drawn from, in case you want the full
explanation before or after running it. Replace `<NODE_IP>`,
`YOUR_GITHUB_USERNAME`, and the MetalLB IP addresses with your real values
throughout — Claude will remind you if you paste a command still
containing a placeholder, and will ask you for the IP addresses directly
in Phase 5 rather than guessing them.
explanation before or after running it. Replace `<NODE1_IP>`/`<NODE2_IP>`/`<NODE3_IP>`,
`<VIP_IP>`, `YOUR_GITHUB_USERNAME`, and the MetalLB IP addresses with your
real values throughout — Claude will remind you if you paste a command
still containing a placeholder, and will ask you for all of these directly
in Phases 2 and 5 rather than guessing them.
## Phase 1 — Prerequisites ([`01`](01-prerequisites.md))
- [ ] Target machine chosen, reachable on the network, with a stable
(ideally DHCP-reserved) IP address
- [ ] Talos amd64 ISO downloaded, machine booted from it, sitting at the
boot screen
- [ ] Three target machines chosen, each network-reachable, each with a
stable (ideally DHCP-reserved) IP address, each with a second disk
≥900GB set aside for Longhorn
- [ ] Custom Talos schematic built at factory.talos.dev with `iscsi-tools`
and `util-linux-tools`, ISO downloaded
- [ ] All three machines booted from that ISO, sitting at the boot screen
- [ ] `talosctl version --client` runs successfully
- [ ] `kubectl version --client` runs successfully
- [ ] `argocd version --client` runs successfully
@@ -27,12 +29,16 @@ in Phase 5 rather than guessing them.
## Phase 2 — Talos bootstrap ([`02`](02-talos-bootstrap.md), [`talos/README.md`](../talos/README.md))
- [ ] `talosctl gen config gitops-starter https://<NODE_IP>:6443 --output-dir _out --config-patch-control-plane @talos/patch-controlplane.yaml` run — `_out/controlplane.yaml`, `worker.yaml`, `talosconfig` exist
- [ ] **Ask Claude** to help pick: your three nodes' IP addresses (or
confirm DHCP reservations) and a separate VIP address — all based on
your actual LAN/DHCP setup
- [ ] `talosctl gen config gitops-starter https://<VIP_IP>:6443 --output-dir _out --config-patch-control-plane @talos/patch-controlplane.yaml --config-patch-control-plane @talos/patch-longhorn.yaml` run — `_out/controlplane.yaml`, `worker.yaml`, `talosconfig` exist
- [ ] `TALOSCONFIG` exported (or `talosctl config endpoint`/`node` set) to `_out/talosconfig`
- [ ] `talosctl apply-config --insecure --nodes <NODE_IP> --file _out/controlplane.yaml` run, node rebooted
- [ ] `talosctl bootstrap --nodes <NODE_IP>` run — **exactly once**
- [ ] `talosctl kubeconfig --nodes <NODE_IP> .` run, `KUBECONFIG` pointed at the result
- [ ] `kubectl get nodes` shows one node, `Ready`
- [ ] For **each** of the three nodes: `talosctl get disks --insecure --nodes <NODE_IP>` confirms two disks, then `talosctl apply-config --insecure --nodes <NODE_IP> --file _out/controlplane.yaml` run, node rebooted
- [ ] `talosctl bootstrap --nodes <NODE1_IP>` run — **exactly once**, against **exactly one** node
- [ ] `talosctl kubeconfig --nodes <VIP_IP> .` run, `KUBECONFIG` pointed at the result
- [ ] `kubectl get nodes` shows **all three** nodes, `Ready`
- [ ] `docs/network.md` updated with all three node addresses and the VIP
## Phase 3 — Argo CD install ([`03`](03-installing-argocd.md), [`argocd/README.md`](../argocd/README.md))
@@ -45,13 +51,15 @@ in Phase 5 rather than guessing them.
## Phase 4 — Hand the cluster to Git ([`04`](04-app-of-apps.md))
- [ ] `spec.source.repoURL` fixed in all three placeholder files: run
- [ ] `spec.source.repoURL` fixed in all four placeholder files: run
`grep -rl YOUR_GITHUB_USERNAME .` to find them
(`argocd/app-of-apps/app-of-apps.yaml`, `apps/demo.yaml`, `apps/metallb.yaml`)
(`argocd/app-of-apps/app-of-apps.yaml`, `apps/demo.yaml`,
`apps/metallb.yaml`, `apps/longhorn.yaml`)
- [ ] Edit committed and pushed to your fork's `main` branch
- [ ] `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml`
- [ ] `argocd app list` shows `app-of-apps`, `Synced`/`Healthy`
- [ ] `argocd app list` shows `demo` and `metallb`, appeared automatically, both `Synced`/`Healthy`
- [ ] `argocd app list` shows `demo`, `metallb`, and `longhorn`, appeared
automatically, all `Synced`/`Healthy`
- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running
(Service `EXTERNAL-IP` still `<pending>` — expected for now)
@@ -68,7 +76,15 @@ in Phase 5 rather than guessing them.
- [ ] `argocd-server` Service patched to `LoadBalancer` with your chosen Argo CD address (`kubectl -n argocd patch svc argocd-server ...`)
- [ ] Argo CD UI reachable at `https://<that-address>` with no port-forward running
## Phase 6 — Prove the loop ([`06`](06-your-first-change.md))
## Phase 6 — Longhorn: real storage ([`06`](06-longhorn.md))
- [ ] `kubectl -n longhorn-system get pods` shows manager/CSI Pods
`Running` on all three nodes
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
with a healthy, replicated Longhorn volume behind it
## Phase 7 — Prove the loop ([`07`](07-your-first-change.md))
- [ ] Edited the `<h1>` text in `apps/demo/configmap.yaml`
- [ ] Committed and pushed
@@ -85,6 +101,6 @@ push" and the page updating. If you can't yet, that's the sign to ask
Claude for a walkthrough of whichever phase felt shaky before calling this
finished.
From here: [`07-adding-new-apps.md`](07-adding-new-apps.md) to add
something of your own, or [`08-growing-up.md`](08-growing-up.md) for the
From here: [`08-adding-new-apps.md`](08-adding-new-apps.md) to add
something of your own, or [`09-growing-up.md`](09-growing-up.md) for the
bigger next steps (more nodes, ingress, secrets management).
docs/glossary.md
+61 -7
View File
@@ -29,7 +29,7 @@ steps — installing Argo CD, applying the app-of-apps — that get the
GitOps loop started before Git takes over.
**cluster** — A group of one or more machines (nodes) working together as
one Kubernetes system. This repo's cluster starts as a single node.
one Kubernetes system. This repo's cluster starts as three nodes.
**CNI (Container Network Interface)** — The plugin responsible for giving
Pods network connectivity and IP addresses. Talos ships Flannel, a CNI, by
@@ -42,9 +42,9 @@ demo app's HTML page.
**control plane** — The set of components that run the cluster itself
(API server, scheduler, etcd) as opposed to your applications. Normally
kept on separate machines from your workloads; this repo's single-node
setup deliberately blurs that line for simplicity — see
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml).
kept on separate machines from your workloads; this repo's three nodes
deliberately blur that line, each acting as control plane and worker at
once — see [`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml).
**CRD (Custom Resource Definition)** — A way to teach Kubernetes about a
new *kind* of object beyond its built-ins (Pod, Service, etc.). Argo CD's
@@ -74,13 +74,19 @@ the real system to match it. The core idea this whole repo demonstrates.
**Ingress** — A Kubernetes object that routes external HTTP(S) traffic to
services inside the cluster by hostname/path, via an Ingress controller.
Not used in this repo's v1 — see
[`docs/08-growing-up.md`](08-growing-up.md).
[`docs/09-growing-up.md`](09-growing-up.md).
**IPAddressPool** — A MetalLB custom resource (see CRD) declaring which IP
addresses it's allowed to hand out to `LoadBalancer` Services. Configured
in [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml)
— see [`docs/05-metallb.md`](05-metallb.md).
**iSCSI** — A protocol for accessing block storage over a network as if it
were a local disk. Longhorn's underlying engine uses it, which is why
nodes need the `iscsi-tools` Talos system extension (see below) baked
into their image before Longhorn will work — see
[`docs/01-prerequisites.md`](01-prerequisites.md).
**kubeconfig** — A file holding the credentials and connection info
`kubectl` (and other Kubernetes tools) use to talk to a specific cluster.
Retrieved with `talosctl kubeconfig` in this setup.
@@ -96,6 +102,12 @@ cloud provider this provisions an actual cloud load balancer; on bare
metal (this repo) MetalLB fills that role instead — see
[`docs/05-metallb.md`](05-metallb.md).
**Longhorn** — Software installed in this repo (`apps/longhorn.yaml`) that
turns a plain disk on each node into one pool of storage, replicated
across nodes, so apps can request real persistent storage on bare metal —
a job a cloud provider's disk service would normally do. See
[`docs/06-longhorn.md`](06-longhorn.md).
**machine config** — Talos's equivalent of a Kubernetes manifest, but for
the operating system itself: a declarative file describing how a node
should be configured, generated with `talosctl gen config` and sent to the
@@ -116,19 +128,37 @@ independently.
**node** — A single machine (physical or virtual) that's part of a
Kubernetes cluster.
**PersistentVolumeClaim (PVC)** — How an app asks Kubernetes for storage:
"I need this much space," without caring which disk or which node
provides it. Longhorn is what actually fulfills that request in this
repo — see [`docs/06-longhorn.md`](06-longhorn.md).
**Pod** — The smallest deployable unit in Kubernetes: one or more
containers that run together, sharing network and storage. You'll rarely
create Pods directly — a Deployment creates and manages them for you.
**quorum** — The minimum number of etcd members (a majority) that have to
agree before the cluster's control plane can do anything. This is why
control-plane node counts matter, and why 3 is a meaningfully different
number than 2 or 4 — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker)
and [`docs/09-growing-up.md`](09-growing-up.md#why-not-a-fourth-control-plane-node).
**reconciliation** — The repeating process of comparing desired state
(Git, for Argo CD) against actual state (the live cluster) and correcting
any difference. See the diagram in
[`docs/00-what-is-all-this.md`](00-what-is-all-this.md#the-reconciliation-loop).
**schematic** — A custom Talos boot image, built at
[factory.talos.dev](https://factory.talos.dev) by selecting which system
extensions (see below) it should include. This repo's nodes need one with
`iscsi-tools` and `util-linux-tools` for Longhorn to work — see
[`docs/01-prerequisites.md`](01-prerequisites.md).
**Secret** — A Kubernetes object for holding sensitive data. Its values are
only base64-*encoded*, not encrypted — never commit a raw Secret manifest
containing real credentials to Git. See
[`docs/08-growing-up.md`](08-growing-up.md) for how to do this safely.
[`docs/09-growing-up.md`](09-growing-up.md) for how to do this safely.
**selfHeal** — An Argo CD sync policy setting: if the live cluster drifts
from what's in Git (e.g. someone runs `kubectl edit`), Argo CD reverts it
@@ -141,6 +171,12 @@ uses a `LoadBalancer` Service for the demo app, backed by MetalLB — see
[`apps/demo/service.yaml`](../apps/demo/service.yaml) and
[`docs/05-metallb.md`](05-metallb.md).
**StorageClass** — A Kubernetes object describing a "kind" of storage
that's available (who provides it, how). Longhorn's chart registers itself
as the cluster's default StorageClass, so a plain
`PersistentVolumeClaim` with no class named gets Longhorn storage
automatically — see [`docs/06-longhorn.md`](06-longhorn.md).
**sync** — Argo CD's term for actually applying a change (a reconciliation
pass that found and fixed a difference). "Synced" means the cluster
currently matches Git.
@@ -153,9 +189,27 @@ waits for a wave to be healthy before starting the next. Used in
apply before the `IPAddressPool`/`L2Advertisement` that depend on them —
see [`docs/05-metallb.md`](05-metallb.md).
**Talos Linux** — The operating system this repo's node(s) run: built only
**system extension** — An optional add-on baked into a Talos boot image at
build time (see "schematic," above), since Talos has no package manager to
install one onto an already-running node. This repo's nodes need
`iscsi-tools` and `util-linux-tools` for Longhorn.
**Talos Linux** — The operating system this repo's nodes run: built only
to run Kubernetes, configured entirely declaratively over the network, with
no SSH or shell. See [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
**talosconfig** — The credentials file `talosctl` uses to talk to a Talos
node's API — the Talos-level equivalent of a kubeconfig.
**UserVolumeConfig** — A standalone Talos configuration resource (not part
of the normal `machine:`/`cluster:` structure) that tells Talos to find,
format, and mount a disk on its own, based on rules you give it (which
disk to pick, minimum size). Used in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) to prepare
each node's storage for Longhorn.
**VIP (Virtual IP)** — One IP address that Talos keeps pointed at whichever
of several control-plane nodes is currently healthy, re-electing within
seconds if that node goes down. Everything that talks to this cluster
should use the VIP, not any one node's own address — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#the-vip-one-address-for-a-cluster-that-moves).
docs/network.md
+17 -3
View File
@@ -13,6 +13,21 @@ assigned or changed, so you (or Claude) never have to reverse-engineer
> instruction that tells Claude to do this on its own when it makes such a
> change itself.
## The three nodes and the VIP
None of these are from the MetalLB pool below — they're the machines
themselves and the shared address Talos elects between them. Set in
[`talos/README.md`](../talos/README.md); see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md) for what the VIP is
and why it's separate from any one node's address.
| Address | What | Port | URL |
|---|---|---|---|
| `EDIT ME` | Node 1 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Node 2 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Node 3 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Cluster VIP (use this one, not a node's own address) | 6443 | `https://EDIT ME:6443` |
## MetalLB pool
| | |
@@ -20,16 +35,15 @@ assigned or changed, so you (or Claude) never have to reverse-engineer
| Range | `EDIT ME — e.g. 192.168.1.240-192.168.1.250` |
| Configured in | [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) |
## Allocated addresses
## Allocated addresses (from the MetalLB pool)
| Address | What | Namespace | Port(s) | URL | Source |
|---|---|---|---|---|---|
| `<NODE_IP>` | Talos node — Kubernetes API server | — | 6443 | `https://<NODE_IP>:6443` | Not from the MetalLB pool — the node's own address, set in [`talos/README.md`](../talos/README.md) |
| `EDIT ME` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://EDIT ME` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) |
| `EDIT ME` | Demo app | `demo` | 80 | `http://EDIT ME` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) |
Add a row here every time you add a new app with a `LoadBalancer` Service
(see [`docs/07-adding-new-apps.md`](07-adding-new-apps.md)). Leave the URL
(see [`docs/08-adding-new-apps.md`](08-adding-new-apps.md)). Leave the URL
column's port number off only when it's the protocol default (80 for
`http://`, 443 for `https://`) — include it otherwise.
talos/README.md
+68 -44
View File
@@ -5,64 +5,81 @@ This is the detailed, copy-pasteable companion to
first for the explanations — this one is here so you have every exact
command in one place once you understand what they do.
Throughout, replace `<NODE_IP>` with your machine's real IP address. If it's
a VM, give it a DHCP reservation (a fixed IP tied to its MAC address) in your
router or hypervisor before you start, so the address never changes under
the cluster. Talos bakes the IP into certificates during setup, and changing
it later means starting over.
Throughout, replace `<NODE1_IP>`, `<NODE2_IP>`, `<NODE3_IP>` with your three
machines' real IP addresses, and `<VIP_IP>` with the shared address you
picked (with Claude's help) for the cluster's VIP — see
[`docs/02-talos-bootstrap.md`](../docs/02-talos-bootstrap.md#the-vip-one-address-for-a-cluster-that-moves).
Give each machine a DHCP reservation (a fixed IP tied to its MAC address)
in your router or hypervisor before you start, so none of these addresses
change under the cluster. Talos bakes them into certificates during setup,
and changing one later means starting over for that node.
Steps 1–2 below run **once, total**. Steps 3 runs **once per node** (three
times). Steps 4–6 run **once, total**, against whichever node you like
(or, once it exists, the VIP).
All commands assume you're running them from the root of this repo.
## 1. Generate machine configuration
```sh
talosctl gen config gitops-starter https://<NODE_IP>:6443 \
talosctl gen config gitops-starter https://<VIP_IP>:6443 \
--output-dir _out \
--config-patch-control-plane @talos/patch-controlplane.yaml
--config-patch-control-plane @talos/patch-controlplane.yaml \
--config-patch-control-plane @talos/patch-longhorn.yaml
```
Note the cluster endpoint here is the **VIP**, not any one node's address
— every node needs to agree in advance on where the cluster "lives," and
that address has to be the one that keeps working no matter which node is
currently answering.
This writes three files into `_out/` (already covered by this repo's
`.gitignore` — they contain private keys and must never be committed):
- `controlplane.yaml` — the full machine configuration for a control-plane
node, with our single-node patch baked in.
- `worker.yaml` — a worker machine configuration. Unused for now (single
node), kept for when you follow docs/08-growing-up.md.
- `controlplane.yaml` — the machine configuration for a control-plane
node, with both patches baked in. Applied identically to all three
nodes — there's no separate worker config in this setup, since every
node is both.
- `worker.yaml` — a plain worker config, generated automatically but
unused here since every node in this repo's setup is both control plane
and worker. Kept around for [`docs/09-growing-up.md`](../docs/09-growing-up.md),
where a fourth-and-beyond node uses exactly this file.
- `talosconfig` — *your* client credentials for talking to the Talos API on
this cluster (equivalent in spirit to a kubeconfig, but for Talos itself
rather than Kubernetes).
`gitops-starter` is the cluster name — an arbitrary label baked into the
generated certificates and config, used to tell clusters apart if you ever
manage more than one. `https://<NODE_IP>:6443` is the cluster's Kubernetes
API endpoint: port 6443 is the standard Kubernetes API server port, and this
same address gets written into the certificates and your future kubeconfig.
manage more than one. Port 6443 is the standard Kubernetes API server port.
## 2. Point talosctl at this cluster
```sh
export TALOSCONFIG=_out/talosconfig
talosctl config endpoint <NODE_IP>
talosctl config node <NODE_IP>
talosctl config endpoint <NODE1_IP> <NODE2_IP> <NODE3_IP>
```
`endpoint` is which machine talosctl *talks to*. `node` is which machine a
command *applies to*. On a single-node cluster they're the same address, but
Talos always keeps the two concepts separate, because on a multi-node
cluster you often want to ask one node about another.
`endpoint` is which machine(s) talosctl is willing to *talk to* — giving
it all three means it can still reach the cluster if any one of them is
down. You'll pass `--nodes` per-command below for which machine a given
command actually *applies to* — Talos always keeps those two concepts
separate.
## 3. Apply the configuration
## 3. Apply the configuration — once per node
Boot the target machine from the Talos ISO (see
[docs/01-prerequisites.md](../docs/01-prerequisites.md) for where to get it).
It boots into "maintenance mode" — alive enough to accept a configuration,
with nothing installed yet. Confirm it's reachable, then apply:
Boot each machine from your custom Talos ISO (see
[docs/01-prerequisites.md](../docs/01-prerequisites.md)). Each boots into
"maintenance mode" — alive enough to accept a configuration, with nothing
installed yet. For **each** node in turn:
```sh
talosctl get disks --insecure --nodes <NODE_IP>
```
Note the disk name you want Talos installed to (e.g. `/dev/sda`). Then:
Confirm you see two disks: the one Talos will install itself to, and the
≥900GB one `talos/patch-longhorn.yaml`'s `UserVolumeConfig` will claim for
Longhorn. Then:
```sh
talosctl apply-config --insecure \
@@ -73,32 +90,37 @@ talosctl apply-config --insecure \
`--insecure` is required for this one command only: the node has no
certificates yet, so talosctl can't authenticate to it normally. This is
the one moment during setup where you're trusting the network — do this on
a network you control. The node will reboot, install Talos to disk, and
come back up running the configuration you gave it — now *with*
certificates, so every command after this uses the secure path.
a network you control. The node will reboot, install Talos to its system
disk, format and mount its second disk for Longhorn, and come back up
running the configuration you gave it — now *with* certificates, so every
command after this uses the secure path.
## 4. Bootstrap etcd
Repeat this for `<NODE1_IP>`, `<NODE2_IP>`, and `<NODE3_IP>` before moving on.
Talos is installed and Kubernetes components are starting, but the cluster
has no data store yet. This one-time command initializes it:
## 4. Bootstrap etcd — once, total
Talos is installed on all three nodes and Kubernetes components are
starting, but the cluster has no data store yet. This one-time command
initializes it:
```sh
talosctl bootstrap --nodes <NODE_IP>
talosctl bootstrap --nodes <NODE1_IP>
```
Run this exactly once per cluster, against exactly one control-plane node,
even if you later add more. Running it again on an existing cluster can
corrupt it.
Run this **exactly once**, against **exactly one** of your three nodes —
even though all three will end up running etcd as peers, only one needs
telling to *start* the cluster; the other two discover and join the
existing one. Running it again against an existing cluster can corrupt it.
## 5. Get your kubeconfig
```sh
talosctl kubeconfig --nodes <NODE_IP> .
talosctl kubeconfig --nodes <VIP_IP> .
```
Downloads a kubeconfig into the current directory (also gitignored —
it grants full cluster access). Merge it with `~/.kube/config` or point
`KUBECONFIG` at it directly:
it grants full cluster access), pointed at the VIP rather than any one
node. Merge it with `~/.kube/config` or point `KUBECONFIG` at it directly:
```sh
export KUBECONFIG=$PWD/kubeconfig
@@ -110,14 +132,16 @@ export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes
```
Expect to see one node, `Ready`, within a couple of minutes (it takes a
moment for the default CNI — Talos ships Flannel out of the box — to come
up and clear the node's networking checks).
Expect to see **three** nodes, `Ready`, within a couple of minutes (it
takes a moment for the default CNI — Talos ships Flannel out of the box —
to come up and clear each node's networking checks).
```
NAME STATUS ROLES AGE VERSION
talos-... Ready control-plane 90s v1.3x.x
talos-... Ready control-plane 85s v1.3x.x
talos-... Ready control-plane 80s v1.3x.x
```
If it's stuck on `NotReady`, give it another minute before troubleshooting —
first boot does a lot of one-time work.
If any node is stuck on `NotReady`, give it another minute before
troubleshooting — first boot does a lot of one-time work.
talos/patch-controlplane.yaml
+36 -8
View File
@@ -1,15 +1,43 @@
# A patch applied on top of the controlplane.yaml that `talosctl gen config`
# produces (see talos/README.md and docs/02-talos-bootstrap.md for the full
# command). Talos's generated defaults assume a multi-node cluster where
# control-plane nodes only run the Kubernetes control plane itself (API
# server, scheduler, etcd, etc.) and never your actual workloads. For a
# single-node learning cluster, that would mean nothing can ever be
# scheduled — there's no other node to run it on.
# command). Applied identically to all three control-plane nodes.
#
# This one field undoes that restriction, letting the single node be both
# the control plane AND a worker.
# --- allowSchedulingOnControlPlanes ---
# Talos's generated defaults assume control-plane nodes only run the
# Kubernetes control plane itself (API server, scheduler, etcd, etc.) and
# never your actual workloads — normal practice, so a workload crashing
# never has a chance to take the control plane down with it. This repo's
# three nodes are deliberately both control plane AND worker at once (no
# separate worker nodes), to get real multi-node resilience without needing
# more than three machines. This one field undoes the default restriction.
#
# >> Ask Claude: "why does Kubernetes normally keep workloads off control
# plane nodes?" if you want the reasoning before you turn it off.
# plane nodes, and what am I giving up by turning that off here?" if you
# want the reasoning before you turn it off.
#
# --- vip ---
# With three control-plane nodes, "the cluster's API server" isn't one
# fixed machine anymore — it's whichever of the three happen to be up.
# Talos's VIP (Virtual IP) feature solves this: the three nodes
# continuously elect one of themselves to answer on this one shared
# address, and re-elect within seconds if that node goes down. Point
# everything that talks to the cluster (talosctl, kubectl, this repo's
# Application manifests) at the VIP, never at one node's own address —
# that's what makes losing any single node a non-event instead of an
# outage.
#
# Replace both the interface name and the IP below with your real values
# before generating config — see docs/02-talos-bootstrap.md.
#
# >> Ask Claude to help you pick the VIP address — like the MetalLB pool in
# docs/05-metallb.md, it has to be a real, unused address on your LAN,
# outside your DHCP range.
cluster:
allowSchedulingOnControlPlanes: true
machine:
network:
interfaces:
- interface: eth0 # EDIT ME — match your node's real NIC name (`talosctl get links --insecure --nodes <NODE_IP>` from maintenance mode will show it)
dhcp: true
vip:
ip: 192.168.1.239 # EDIT ME — replace with your chosen VIP address
talos/patch-longhorn.yaml
+60
View File
@@ -0,0 +1,60 @@
# Prepares each node for Longhorn (apps/longhorn.yaml) — the storage layer
# this repo ships pre-installed. Applied identically to all three
# control-plane nodes, alongside patch-controlplane.yaml.
#
# This file has TWO documents (separated by `---`), because it's making two
# different kinds of change: a patch to the normal machine config, and a
# standalone Talos resource that doesn't live inside `machine:`/`cluster:`
# at all.
#
# --- Document 1: mount the volume into the kubelet's view ---
# Talos runs the kubelet in its own isolated mount namespace. Without this,
# a volume mounted at /var/mnt/longhorn on the node itself would be
# invisible to any Pod using a hostPath-style mount there — which is
# exactly how Longhorn's own manager and engine Pods access node storage.
# This bind-mounts that same path into the kubelet's namespace so Pods can
# actually see it. `rshared` matters specifically: it lets mounts Longhorn
# creates later *underneath* this path (one per volume) propagate back out
# to the host, instead of being invisible outside the container that made
# them.
#
# --- Document 2: UserVolumeConfig ---
# A Talos UserVolumeConfig tells Talos itself to find, format, and mount a
# disk — no manual partitioning on your part. `diskSelector.match:
# '!system_disk'` means "any disk that isn't the one Talos is installed
# on" (never touches your OS disk), and `minSize: 900GiB` means "only
# consider disks at least that large" — so if a node has more than one
# extra disk, Talos won't accidentally pick a small one. Adjust `minSize`
# if your nodes' data disks are smaller than 900GiB; just make sure it's
# still large enough to exclude anything that *isn't* meant for Longhorn.
# `grow: false` keeps the volume at its initial size even if the
# underlying disk is later resized — deliberate here, so growing storage
# is always a conscious decision (see docs/09-growing-up.md).
#
# The `name: longhorn` below is what determines the mount path Talos uses
# — user volumes are always mounted at /var/mnt/<name>, which is why
# Document 1 above and apps/longhorn/values.yaml both hardcode
# /var/mnt/longhorn. Renaming this would mean updating both.
#
# >> Ask Claude: "how do I find out which of my disks Talos will actually
# pick for this?" before you boot for real — worth confirming on your
# specific hardware rather than assuming.
machine:
kubelet:
extraMounts:
- destination: /var/mnt/longhorn
type: bind
source: /var/mnt/longhorn
options:
- bind
- rshared
- rw
---
apiVersion: v1alpha1
kind: UserVolumeConfig
name: longhorn
provisioning:
diskSelector:
match: "!system_disk"
minSize: 900GiB
grow: false