Move to a three-node cluster with a Talos VIP; add Longhorn

Every node is now both control plane and worker, giving real tolerance
for losing any single machine (etcd quorum needs 3, not 1). A Talos VIP
gives the cluster one stable address regardless of which node answers.

Also pre-installs Longhorn for replicated storage, backed by a Talos
UserVolume on each node's second disk (>=900GB, non-system) rather than
the default path — mirroring the pattern from the homescale repo.

Docs renumbered to make room for 06-longhorn.md; checklist, glossary,
network map, and CLAUDE.md's guided-bootstrap instructions updated to
match, including asking the user for node IPs, VIP, and disk info before
generating any Talos config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Maxwell Morris
2026-08-22 18:25:38 +01:00
co-authored by Claude Sonnet 5
parent 92c3ad3191
commit 110bd97cca
26 changed files with 777 additions and 261 deletions
+25 -13
View File
@@ -59,7 +59,16 @@ end to end" — drive [`docs/checklist.md`](docs/checklist.md) interactively:
6. Periodically remind them of `docs/glossary.md` and the "Ask Claude" 6. Periodically remind them of `docs/glossary.md` and the "Ask Claude"
checkpoints along the way — this mode shouldn't feel like a script being checkpoints along the way — this mode shouldn't feel like a script being
read at them. read at them.
7. **Phase 5 (MetalLB) needs real information from the user, not 7. **The Talos phase (docs/02) needs real information too, before any
config is generated.** Ask: what are the three nodes' IP addresses (or
DHCP reservations), and — separately — what unused address should the
cluster's VIP use? The VIP is a fourth address, distinct from all three
node IPs and from the MetalLB pool (below); it needs the same "ask about
their LAN and DHCP range" treatment as MetalLB. Also confirm each node
actually has a second disk ≥900GB before they generate config — if not,
that's a real constraint on `talos/patch-longhorn.yaml`'s `minSize`, not
something to paper over with a placeholder.
8. **The MetalLB phase (docs/05) needs real information from the user, not
placeholders.** Before editing `apps/metallb/templates/ipaddresspool.yaml` placeholders.** Before editing `apps/metallb/templates/ipaddresspool.yaml`
or `apps/demo/service.yaml`, actually ask: what does their LAN's subnet or `apps/demo/service.yaml`, actually ask: what does their LAN's subnet
look like, and what IP range does their router's DHCP server already look like, and what IP range does their router's DHCP server already
@@ -80,10 +89,12 @@ useful if it's actually current, so treat updating it as part of the
change itself, not an afterthought: change itself, not an afterthought:
- Whenever you (Claude) assign, learn, or change a real IP address for - Whenever you (Claude) assign, learn, or change a real IP address for
something — walking the user through MetalLB's pool something — the three node addresses and VIP in
([`docs/05-metallb.md`](docs/05-metallb.md)), patching `argocd-server`, [`docs/02-talos-bootstrap.md`](docs/02-talos-bootstrap.md), walking the
helping add a new app with a `LoadBalancer` Service — update the table in user through MetalLB's pool ([`docs/05-metallb.md`](docs/05-metallb.md)),
`docs/network.md` in the same breath. Don't wait to be asked. patching `argocd-server`, helping add a new app with a `LoadBalancer`
Service — update the table in `docs/network.md` in the same breath.
Don't wait to be asked.
- Every row needs a full, correct URL — scheme, IP, and port — not just a - Every row needs a full, correct URL — scheme, IP, and port — not just a
bare address. Include the port number unless it's the protocol's default bare address. Include the port number unless it's the protocol's default
(80 for `http://`, 443 for `https://`). If you're not sure what port a (80 for `http://`, 443 for `https://`). If you're not sure what port a
@@ -101,7 +112,7 @@ Point new users here first if they seem unsure where to start:
1. [`README.md`](README.md) — orientation 1. [`README.md`](README.md) — orientation
2. [`docs/00-what-is-all-this.md`](docs/00-what-is-all-this.md) — mental model, read before touching a terminal 2. [`docs/00-what-is-all-this.md`](docs/00-what-is-all-this.md) — mental model, read before touching a terminal
3. [`docs/01-prerequisites.md`](docs/01-prerequisites.md) → [`08-growing-up.md`](docs/08-growing-up.md) — in order 3. [`docs/01-prerequisites.md`](docs/01-prerequisites.md) → [`docs/09-growing-up.md`](docs/09-growing-up.md) — in order
4. [`docs/checklist.md`](docs/checklist.md) — condensed step-by-step reference once concepts are understood, or to drive Guided Bootstrap Mode 4. [`docs/checklist.md`](docs/checklist.md) — condensed step-by-step reference once concepts are understood, or to drive Guided Bootstrap Mode
5. [`docs/glossary.md`](docs/glossary.md) — reference throughout, not read start to finish 5. [`docs/glossary.md`](docs/glossary.md) — reference throughout, not read start to finish
@@ -116,17 +127,18 @@ If asked to change manifests or docs rather than explain them:
`argocd/`. `argocd/`.
- Keep the moving parts minimal. Ingress, cert-manager, and secrets - Keep the moving parts minimal. Ingress, cert-manager, and secrets
management are intentionally **documented, not shipped** (see management are intentionally **documented, not shipped** (see
[`docs/08-growing-up.md`](docs/08-growing-up.md)) — don't add them to [`docs/09-growing-up.md`](docs/09-growing-up.md)) — don't add them to
make the repo "more complete" without being asked. MetalLB is the one make the repo "more complete" without being asked. MetalLB and Longhorn
exception, shipped deliberately because bare metal has no other way to are the two exceptions, shipped deliberately because bare metal has no
give a Service a real IP address — don't take that as license to add other way to give a Service a real IP address or give an app real
other addons the same way without being asked. persistent storage — don't take that as license to add other addons the
same way without being asked.
- This repo is **public**. Never write a plaintext secret, credential, or - This repo is **public**. Never write a plaintext secret, credential, or
private key into any file here. If a change would need one, stop and private key into any file here. If a change would need one, stop and
point to the Sealed Secrets / SOPS guidance in point to the Sealed Secrets / SOPS guidance in
`docs/08-growing-up.md` instead of improvising. `docs/09-growing-up.md` instead of improvising.
- New apps follow the existing `apps/<name>.yaml` + `apps/<name>/` pattern - New apps follow the existing `apps/<name>.yaml` + `apps/<name>/` pattern
— see [`docs/07-adding-new-apps.md`](docs/07-adding-new-apps.md) before — see [`docs/08-adding-new-apps.md`](docs/08-adding-new-apps.md) before
adding one. `apps/metallb/` is the reference example for a Helm-based adding one. `apps/metallb/` is the reference example for a Helm-based
one. one.
- Every `apps/*.yaml` Application manifest needs its own `repoURL` fixed to - Every `apps/*.yaml` Application manifest needs its own `repoURL` fixed to
+15 -10
View File
@@ -1,7 +1,7 @@
# GitOps Starter Pack # GitOps Starter Pack
A minimal, real, working GitOps setup for a single Kubernetes cluster — A minimal, real, working GitOps setup for a three-node Kubernetes cluster
built to teach, not just to run. If you've never touched Kubernetes, — built to teach, not just to run. If you've never touched Kubernetes,
GitOps, Talos, or Argo CD before, this is meant to get you from a bare GitOps, Talos, or Argo CD before, this is meant to get you from a bare
machine to a working cluster you actually understand, not just one you machine to a working cluster you actually understand, not just one you
copy-pasted into existence. copy-pasted into existence.
@@ -35,6 +35,9 @@ shell history. This repo takes a different approach:
5. **MetalLB** gives apps in this cluster real IP addresses on your own 5. **MetalLB** gives apps in this cluster real IP addresses on your own
network — something a cloud provider would normally do for you, and network — something a cloud provider would normally do for you, and
bare metal doesn't have on its own. bare metal doesn't have on its own.
6. **Longhorn** does the same for storage — turning a spare disk on each
node into one pool of storage that's replicated across all three
nodes, so losing a machine doesn't mean losing its data.
Git becomes the one place that's actually true. If you want the full Git becomes the one place that's actually true. If you want the full
picture with diagrams before doing anything, start with picture with diagrams before doing anything, start with
@@ -46,13 +49,14 @@ picture with diagrams before doing anything, start with
talos/ → Talos machine config + bootstrap guide (the OS layer) talos/ → Talos machine config + bootstrap guide (the OS layer)
argocd/ → Argo CD install + the one manifest you apply by hand argocd/ → Argo CD install + the one manifest you apply by hand
apps/ → Everything Argo CD manages from here on — demo app, MetalLB, apps/ → Everything Argo CD manages from here on — demo app, MetalLB,
and wherever you add new apps Longhorn, and wherever you add new apps
docs/ → The numbered, progressive guide you should actually read docs/ → The numbered, progressive guide you should actually read
``` ```
## Prerequisites ## Prerequisites
- One machine (physical or VM) to become your cluster node - Three machines (physical or VM), each with a second disk ≥900GB, to
become your cluster's three nodes
- `talosctl`, `kubectl`, and the `argocd` CLI installed locally - `talosctl`, `kubectl`, and the `argocd` CLI installed locally
- A GitHub account, and your own fork of this repo - A GitHub account, and your own fork of this repo
@@ -68,11 +72,12 @@ Read these in order. Each is short — one concept per file.
4. [`docs/03-installing-argocd.md`](docs/03-installing-argocd.md) — install Argo CD into the cluster 4. [`docs/03-installing-argocd.md`](docs/03-installing-argocd.md) — install Argo CD into the cluster
5. [`docs/04-app-of-apps.md`](docs/04-app-of-apps.md) — apply the one manifest that hands control to Git 5. [`docs/04-app-of-apps.md`](docs/04-app-of-apps.md) — apply the one manifest that hands control to Git
6. [`docs/05-metallb.md`](docs/05-metallb.md) — give apps real IP addresses on your network 6. [`docs/05-metallb.md`](docs/05-metallb.md) — give apps real IP addresses on your network
7. [`docs/06-your-first-change.md`](docs/06-your-first-change.md) — edit a file, push, watch it deploy itself 7. [`docs/06-longhorn.md`](docs/06-longhorn.md) — give apps real, replicated storage
8. [`docs/07-adding-new-apps.md`](docs/07-adding-new-apps.md) — add something of your own 8. [`docs/07-your-first-change.md`](docs/07-your-first-change.md) — edit a file, push, watch it deploy itself
9. [`docs/08-growing-up.md`](docs/08-growing-up.md) — next steps: more nodes, ingress, secrets management 9. [`docs/08-adding-new-apps.md`](docs/08-adding-new-apps.md) — add something of your own
10. [`docs/glossary.md`](docs/glossary.md) — every term, plain English, for reference throughout 10. [`docs/09-growing-up.md`](docs/09-growing-up.md) — next steps: more nodes, ingress, secrets management
11. [`docs/network.md`](docs/network.md) — the running log of every real IP address in your cluster, kept current as you go 11. [`docs/glossary.md`](docs/glossary.md) — every term, plain English, for reference throughout
12. [`docs/network.md`](docs/network.md) — the running log of every real IP address in your cluster, kept current as you go
Prefer a single condensed list you can work through step by step with Prefer a single condensed list you can work through step by step with
Claude, checking items off as you go? Use Claude, checking items off as you go? Use
@@ -101,7 +106,7 @@ moving on.
This repo deliberately does **not** ship ingress, TLS/cert-manager, or This repo deliberately does **not** ship ingress, TLS/cert-manager, or
secrets management in v1 — they're real next steps, documented in secrets management in v1 — they're real next steps, documented in
[`docs/08-growing-up.md`](docs/08-growing-up.md), left out so the core [`docs/09-growing-up.md`](docs/09-growing-up.md), left out so the core
loop stays small enough to fully understand first. This repo is also loop stays small enough to fully understand first. This repo is also
**public** — never commit plaintext secrets, credentials, or private keys **public** — never commit plaintext secrets, credentials, or private keys
to it; see that same doc for how to handle secrets properly once you need to it; see that same doc for how to handle secrets properly once you need
+1 -1
View File
@@ -16,7 +16,7 @@
# (e.g. a database or ingress controller from a public chart repo). Argo CD # (e.g. a database or ingress controller from a public chart repo). Argo CD
# detects a Helm chart automatically when it finds a Chart.yaml at the # detects a Helm chart automatically when it finds a Chart.yaml at the
# source path, resolves its dependencies, and syncs the rendered output just # source path, resolves its dependencies, and syncs the rendered output just
# like any other app. See docs/07-adding-new-apps.md for a worked example — # like any other app. See docs/08-adding-new-apps.md for a worked example —
# the demo app deliberately stays plain manifests so the GitOps loop is easy # the demo app deliberately stays plain manifests so the GitOps loop is easy
# to read end to end with no templating in the way. # to read end to end with no templating in the way.
apiVersion: argoproj.io/v1alpha1 apiVersion: argoproj.io/v1alpha1
+1 -1
View File
@@ -4,7 +4,7 @@
# baking the page into a custom container image. # baking the page into a custom container image.
# #
# Why this matters for the GitOps story: this is the file you'll edit in # Why this matters for the GitOps story: this is the file you'll edit in
# docs/06-your-first-change.md. Change the text here, commit, push, and # docs/07-your-first-change.md. Change the text here, commit, push, and
# watch Argo CD notice the difference in Git and push it into the cluster # watch Argo CD notice the difference in Git and push it into the cluster
# for you — no `kubectl` command involved. That's the whole loop, proven # for you — no `kubectl` command involved. That's the whole loop, proven
# with one line of HTML. # with one line of HTML.
+1 -1
View File
@@ -2,7 +2,7 @@
# serving this ConfigMap's index.html") and hands it to Kubernetes, whose # serving this ConfigMap's index.html") and hands it to Kubernetes, whose
# job is to continuously make reality match that description. If the Pod # job is to continuously make reality match that description. If the Pod
# crashes, Kubernetes restarts it. If the node dies, Kubernetes (given more # crashes, Kubernetes restarts it. If the node dies, Kubernetes (given more
# than one node — see docs/08-growing-up.md) reschedules it elsewhere. # than one node — see docs/09-growing-up.md) reschedules it elsewhere.
# You never tell Kubernetes "start a container" directly; you describe the # You never tell Kubernetes "start a container" directly; you describe the
# end state and it does the work. This is what "declarative" means. # end state and it does the work. This is what "declarative" means.
apiVersion: apps/v1 apiVersion: apps/v1
+36
View File
@@ -0,0 +1,36 @@
# Longhorn — this repo's pre-installed storage layer. Same pattern as
# apps/metallb.yaml: an Application pointing at a Helm chart
# (apps/longhorn/) that pulls in the official upstream chart. Once this
# syncs, any app in this cluster can request persistent storage (a
# PersistentVolumeClaim) and actually get real, replicated storage backed
# by the dedicated disk each node prepared in
# talos/patch-longhorn.yaml — see docs/06-longhorn.md.
#
# Longhorn needs privileged Pods (it manages block devices directly) and
# the `open-iscsi`/`util-linux-tools` Talos system extensions baked into
# your node image — both covered in docs/01-prerequisites.md and
# docs/02-talos-bootstrap.md. Skipping either means Longhorn's Pods will
# sit unhealthy no matter how correctly this Application syncs.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: longhorn
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/YOUR_GITHUB_USERNAME/k8s-starter-pack.git
targetRevision: main
path: apps/longhorn
destination:
server: https://kubernetes.default.svc
namespace: longhorn-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
+18
View File
@@ -0,0 +1,18 @@
# An umbrella chart, same pattern as apps/metallb/Chart.yaml: it declares a
# dependency on the upstream Longhorn chart and carries no templates of its
# own — everything Longhorn needs (its CRDs, controllers, UI, default
# StorageClass) comes from the dependency. See
# docs/08-adding-new-apps.md for more on this pattern.
apiVersion: v2
name: longhorn-app
description: Installs Longhorn, this repo's pre-installed distributed block storage
version: 0.1.0
dependencies:
- name: longhorn
# A version range rather than an exact pin — see the same note in
# apps/metallb/Chart.yaml. Check
# https://artifacthub.io/packages/helm/longhorn/longhorn for the
# current release and pin an exact version for a fully reproducible
# install.
version: ">=1.7.0"
repository: https://charts.longhorn.io
+22
View File
@@ -0,0 +1,22 @@
# Values for the upstream Longhorn chart, nested under `longhorn:` (the
# dependency's name in Chart.yaml).
longhorn:
defaultSettings:
# Must match the mount path from the UserVolumeConfig in
# talos/patch-longhorn.yaml (/var/mnt/<name>) — this is what points
# Longhorn at the dedicated data disk instead of its own default path
# on the root filesystem, which on Talos is ephemeral/unsuited for
# real data anyway.
defaultDataPath: /var/mnt/longhorn
# Longhorn's own headroom settings, kept conservative: stop scheduling
# new replicas onto a disk once free space drops below these
# thresholds, so the cluster degrades gracefully instead of actually
# filling a disk.
storageMinimalAvailablePercentage: 25
storageReservedPercentageForDefaultDisk: 30
preUpgradeChecker:
# Longhorn's own pre-upgrade check runs as a Job that expects to reach
# an upgrade-responder service on the internet by default. Harmless to
# leave on, but pointless for a learning cluster and one less thing to
# explain — off by default here.
jobEnabled: false
+1 -1
View File
@@ -5,7 +5,7 @@
# exists at this path, resolves the dependency, and syncs the combined # exists at this path, resolves the dependency, and syncs the combined
# rendered output as one Application — exactly like any other app in # rendered output as one Application — exactly like any other app in
# apps/. This is the real, working example of the Helm-based app pattern # apps/. This is the real, working example of the Helm-based app pattern
# mentioned in docs/07-adding-new-apps.md: use it as a template if you want # mentioned in docs/08-adding-new-apps.md: use it as a template if you want
# to bring in another third-party chart later. # to bring in another third-party chart later.
apiVersion: v2 apiVersion: v2
name: metallb-app name: metallb-app
+2 -2
View File
@@ -105,8 +105,8 @@ argocd app get app-of-apps
argocd app list argocd app list
``` ```
You should see `app-of-apps` sync, then `demo` and `metallb` Applications You should see `app-of-apps` sync, then `demo`, `metallb`, and `longhorn`
appear underneath it, then sync themselves. See Applications appear underneath it, then sync themselves. See
[`docs/04-app-of-apps.md`](../docs/04-app-of-apps.md) for what's happening [`docs/04-app-of-apps.md`](../docs/04-app-of-apps.md) for what's happening
here and [`docs/checklist.md`](../docs/checklist.md) for the full guided here and [`docs/checklist.md`](../docs/checklist.md) for the full guided
path. path.
+46 -33
View File
@@ -19,17 +19,20 @@ the real cluster match it. You stop making changes by hand. You edit a file,
push it, and a robot applies it for you — the same way every time, with a push it, and a robot applies it for you — the same way every time, with a
record of every change in `git log`. record of every change in `git log`.
## The five pieces, in plain terms ## The six pieces, in plain terms
**Kubernetes** is software that runs on one or more machines and manages **Kubernetes** is software that runs on one or more machines and manages
containers for you: starts them, restarts them if they crash, and moves containers for you: starts them, restarts them if they crash, and moves
them to a different machine if one dies (once you have more than one — see them to a different machine if one dies. You describe the end state you
[`docs/08-growing-up.md`](08-growing-up.md)). You describe the end state you want ("run one copy of this"); Kubernetes' job is to make that true and
want ("run one copy of this"); Kubernetes' job is to make that true and keep keep it true. This is what people mean by **declarative**: you state the
it true. This is what people mean by **declarative**: you state the destination, not the steps to get there. This repo runs three machines
destination, not the steps to get there. from the start, specifically so that "moves them to a different machine
if one dies" is something you actually get, not just a fact about
Kubernetes in general — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md) for why three.
**Talos Linux** is the operating system your machine runs. It's built for **Talos Linux** is the operating system each machine runs. It's built for
exactly one purpose — running Kubernetes — and nothing else. There's no SSH, exactly one purpose — running Kubernetes — and nothing else. There's no SSH,
no shell, no package manager to fiddle with by hand. You configure it by no shell, no package manager to fiddle with by hand. You configure it by
sending it a config file over the network (a "machine config"), the same sending it a config file over the network (a "machine config"), the same
@@ -47,15 +50,23 @@ manages. Rather than one Application resource pointing at everything, one
Application — the app-of-apps — watches a folder (`apps/`) full of *other* Application — the app-of-apps — watches a folder (`apps/`) full of *other*
Application definitions, each pointing at its own small piece. Add a new Application definitions, each pointing at its own small piece. Add a new
app by adding a new file — see app by adding a new file — see
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md). [`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
**MetalLB** solves a specific problem: on a cloud provider, asking for a **MetalLB** solves a specific problem: on a cloud provider, asking for a
public IP address for a service is one API call away; on your own bare public IP address for a service is one API call away; on your own bare
metal machine, nothing hands those out automatically. MetalLB runs inside metal machines, nothing hands those out automatically. MetalLB runs inside
the cluster and fills that gap, so apps here can get a real, dedicated IP the cluster and fills that gap, so apps here can get a real, dedicated IP
address on your network instead of an awkward fixed port. It's pre-installed address on your network instead of an awkward fixed port. It's pre-installed
in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md). in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
**Longhorn** solves the equivalent problem for storage: a cloud provider
gives you durable, replicated disks on demand; bare metal just gives you
whatever's physically bolted into each machine. Longhorn turns a plain
disk on each of your three nodes into one pool of replicated storage any
app can request a slice of — and because it's replicated *across* nodes,
losing one machine doesn't lose the data either. Also pre-installed, not a
"next step" — see [`docs/06-longhorn.md`](06-longhorn.md).
## Architecture — the shape of what you'll build ## Architecture — the shape of what you'll build
``` ```
@@ -64,9 +75,11 @@ in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
│ repo, on GitHub) │ │ repo, on GitHub) │
│ │ │ │
│ apps/demo.yaml │ │ apps/demo.yaml │
│ apps/demo/*.yaml │──┐ │ apps/demo/*.yaml │
│ apps/metallb.yaml │ │ watched by │ apps/metallb.yaml │──┐
│ apps/metallb/* │ │ │ apps/metallb/* │ │ watched by
│ apps/longhorn.yaml │ │
│ apps/longhorn/* │ │
└─────────────────────────┘ │ └─────────────────────────┘ │
▼ ▼
┌───────────────────┐ ┌───────────────────┐
@@ -76,29 +89,29 @@ in this repo, not a "next step" — see [`docs/05-metallb.md`](05-metallb.md).
└─────────┬──────────┘ └─────────┬──────────┘
│ applies manifests │ applies manifests
▼ ▼
┌─────────────────────────────┐ ┌───────────────────────────────────────────────┐
│ Talos Linux node │ │ Three Talos Linux nodes (VIP: <VIP_IP>) │
│ (control plane + worker, │ │ each: control plane + worker + etcd peer │
│ single node) │ │ │
│ │ │ demo namespace: metallb-system: │
│ ┌────────────┐ ┌──────────┐ │ │ nginx Pod ⇄ Service hands out a real IP │
│ │demo │ metallb- │ │ │ (LoadBalancer) ◄───────address to that Service │
│ │namespace: │ system: │ │ │ │
│ │nginx Pod ⇄ │ hands out │ │ │ longhorn-system: each node's 2nd disk │
│ │Service │ a real IP │ │ │ replicates data (≥900GB, UserVolume) │
│ │(LoadBalancer)◄─address to │ │ │ across all 3 nodes ◄──mounted at │
│ │ │ that Service│ │ │ /var/mnt/longhorn │
│ └────────────┘ └──────────┘ │ └───────────────────────────────────────────────┘
└─────────────────────────────┘
``` ```
Everything below the Git repo is a *consequence* of what's in Git. If the Everything below the Git repo is a *consequence* of what's in Git. If any
box labelled "Talos Linux node" burned to the ground and you rebuilt it from one of the three node boxes burned to the ground and you replaced that
scratch, replaying this repo against it would recreate everything except the machine, replaying this repo against it would recreate everything on it
raw OS bootstrap (docs/02), the one-time Argo CD install (docs/03), and the except the raw OS bootstrap (docs/02) and the one-time Argo CD install
handful of network-specific addresses you choose in docs/05 — all (docs/03) — both deliberately kept outside the loop, since something has
deliberately kept outside the loop, since something has to start it and to start it. The handful of network-specific addresses you choose along
nothing in Git can know your LAN's IP layout for you. the way (the VIP, the MetalLB pool) live in `docs/network.md`, not in Git
history alone, since nothing in Git can know your LAN's IP layout for you.
## The reconciliation loop ## The reconciliation loop
+56 -21
View File
@@ -5,31 +5,62 @@ PC, or an amd64 VM). If your hardware is arm64 (Raspberry Pi, Apple Silicon
VM), the tools below all publish arm64 builds too; swap the architecture in VM), the tools below all publish arm64 builds too; swap the architecture in
whichever install command you use and the rest of this guide is unchanged. whichever install command you use and the rest of this guide is unchanged.
## 1. A target machine ## 1. Three target machines
You need one machine to become your cluster node — a spare physical PC, a This repo builds a **three-node cluster**, with every node acting as both
mini PC, or a VM (VirtualBox, UTM, Proxmox, or any hypervisor that can boot control plane (the brain: API server, scheduler, the cluster's own
from an ISO). Minimum realistic spec: 2 CPU cores, 4GB RAM, 20GB disk. database) and worker (runs your actual apps) at once. Three, not one,
because it's the smallest number where losing any single machine doesn't
take the whole cluster down — see
[`docs/00-what-is-all-this.md`](00-what-is-all-this.md) for why three
specifically, not two or four.
It needs to be reachable over the network from your everyday machine (the You need three machines — spare physical PCs, mini PCs, or VMs (VirtualBox,
one you'll run `talosctl` and `kubectl` from), and it needs a stable IP UTM, Proxmox, or any hypervisor that can boot from an ISO). Minimum
address — see the note about DHCP reservations in realistic spec per machine: 2 CPU cores, 4GB RAM, plus **two disks each**:
[`talos/README.md`](../talos/README.md). one for Talos itself (20GB+ is plenty), and a second, separate disk of
**at least 900GB** that will be handed entirely to Longhorn
(see [`docs/06-longhorn.md`](06-longhorn.md)) for storage. If your spare
hardware doesn't have a second disk that large, that's a real constraint —
ask Claude about adjusting the size threshold in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) to fit what you
actually have.
You do **not** need to install an OS on it yet. That's what the next doc Each machine needs to be reachable over the network from your everyday
covers — Talos gets installed by booting a special ISO and sending it machine (the one you'll run `talosctl` and `kubectl` from), and each needs
configuration over the network, not through a normal installer. its own stable IP address — see the note about DHCP reservations in
[`talos/README.md`](../talos/README.md). You'll also need one *additional*
Download the Talos amd64 ISO from the unused address for the cluster's shared VIP (Virtual IP) — more on that in
[Talos releases page](https://github.com/siderolabs/talos/releases) (look
for `metal-amd64.iso` under the latest release) and boot your target machine
from it. Leave it sitting at the boot screen — you'll configure it in
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md). [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
> **Ask Claude:** if you're not sure whether to use a VM or spare physical You do **not** need to install an OS on any of them yet. That's what the
next doc covers — Talos gets installed by booting a special ISO and
sending it configuration over the network, not through a normal installer.
### Building your Talos image
Stock Talos doesn't include everything Longhorn needs — specifically, the
`iscsi-tools` and `util-linux-tools` **system extensions** (optional
add-ons baked into the boot image itself, since Talos has no package
manager to install them onto a running node). You build a custom image,
called a **schematic**, once, and use it for all three nodes:
1. Go to [factory.talos.dev](https://factory.talos.dev).
2. Under system extensions, select `siderolabs/iscsi-tools` and
`siderolabs/util-linux-tools`.
3. Pick the current stable Talos version and `metal-amd64` as the platform.
4. Download the resulting ISO (the site calls this a "schematic" —
your own custom Talos image, identified by a schematic ID).
Boot **all three** machines from this same ISO, and leave each sitting at
its boot screen — you'll configure them in
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
> **Ask Claude:** if you're not sure whether to use VMs or spare physical
> hardware, ask — the tradeoffs (snapshots and easy resets vs. more > hardware, ask — the tradeoffs (snapshots and easy resets vs. more
> realistic networking) are worth five minutes of discussion before you > realistic networking, and whether three VMs on one physical host really
> commit. > gives you the resilience three nodes are supposed to buy you) are worth
> five minutes of discussion before you commit.
## 2. talosctl ## 2. talosctl
@@ -101,8 +132,12 @@ that references a repo URL means your fork's URL.
## Checklist ## Checklist
- [ ] Target machine chosen, network-reachable, IP address noted - [ ] Three target machines chosen, each network-reachable, each with a
- [ ] Talos ISO downloaded, machine booted from it and idle at the boot screen stable IP address noted, each with a second disk ≥900GB for Longhorn
- [ ] One additional unused IP address set aside for the cluster's VIP
- [ ] Custom Talos schematic built at factory.talos.dev with `iscsi-tools`
and `util-linux-tools`, ISO downloaded
- [ ] All three machines booted from that ISO, idle at the boot screen
- [ ] `talosctl version --client` works - [ ] `talosctl version --client` works
- [ ] `kubectl version --client` works - [ ] `kubectl version --client` works
- [ ] `argocd version --client` works - [ ] `argocd version --client` works
+83 -31
View File
@@ -1,7 +1,8 @@
# 02 — Bootstrapping Talos # 02 — Bootstrapping Talos
At the end of this doc, you'll have a single-node Kubernetes cluster, At the end of this doc, you'll have a three-node Kubernetes cluster,
running nothing but Kubernetes itself. No Argo CD yet — that's next. running nothing but Kubernetes itself (plus the storage prep Longhorn
needs later). No Argo CD yet — that's next.
## Why Talos, specifically ## Why Talos, specifically
@@ -17,44 +18,92 @@ same declarative way Kubernetes applies your app manifests. Less surface
area, less to secure, less to keep in sync by hand — the same philosophy as area, less to secure, less to keep in sync by hand — the same philosophy as
the rest of this stack, one layer lower. the rest of this stack, one layer lower.
## Single node, on purpose ## Three nodes, each both control plane and worker
A "real" Kubernetes cluster usually has multiple nodes — some running the A "real" Kubernetes cluster has multiple nodes — some running the
**control plane** (the brain: API server, scheduler, the cluster's **control plane** (the brain: API server, scheduler, the cluster's own
database) and others as **workers** (running your actual apps), so that database, called etcd) and others as **workers** (running your actual
losing one machine doesn't take the whole cluster down. This repo starts apps) — so that losing one machine doesn't take the whole cluster down.
with exactly one node acting as both, because that's the cheapest way to This repo starts with three nodes, each acting as *both* control plane and
learn the concepts that matter for GitOps without needing multiple worker, which gets you real resilience — survive losing any one machine —
machines. [`docs/08-growing-up.md`](08-growing-up.md) covers adding more without needing more than three.
nodes once you're comfortable — the config here is deliberately compatible
with that path, not a dead end. Why three specifically, and not two: etcd (the control plane's database)
uses **quorum** — a majority vote among its members — to agree on cluster
state, and a majority only means something with an odd number of voters.
With one control-plane node, losing it loses the cluster outright. Two
doesn't help: losing either one leaves one survivor, which is not a
majority of two. Three is the smallest number where losing any *one*
still leaves two — a majority — able to keep agreeing on state. This is
also why [`docs/09-growing-up.md`](09-growing-up.md) — once you're past
this doc — specifically warns against going to four control-plane nodes
later: four doesn't improve on three's fault tolerance, it just makes
quorum math worse.
## The VIP: one address for a cluster that moves
With three control-plane nodes, "where's the API server?" no longer has
one fixed answer — it's whichever nodes happen to be up right now. Talos's
**VIP** (Virtual IP) feature solves this: the three nodes continuously
elect one of themselves to answer on one shared IP address you choose, and
re-elect within seconds if that node goes down. Everything that talks to
this cluster — `talosctl`, `kubectl`, the Application manifests in this
repo — points at the VIP, never at one node's own address. See the
`vip` block in
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml).
> **Ask Claude:** "help me pick the VIP address for my cluster" before you
> generate any config — same kind of decision as the MetalLB pool in
> [`docs/05-metallb.md`](05-metallb.md): a real, unused address on your
> LAN, outside your DHCP range, that Claude can't guess on its own.
## Preparing storage for Longhorn
Each node also gets a second disk reserved for
[Longhorn](06-longhorn.md), this repo's pre-installed storage layer —
handled by [`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml),
applied at the same time as the control-plane patch below. Read that
file's comments; the short version is that Talos finds, formats, and
mounts a disk for Longhorn on its own, using rules you set (not the system
disk, at least 900GB), so there's no manual partitioning step here.
## Steps ## Steps
Full exact commands live in [`talos/README.md`](../talos/README.md) — read Full exact commands live in [`talos/README.md`](../talos/README.md) — read
that alongside this doc; this section explains what's happening at each that alongside this doc; this section explains what's happening at each
stage, that one has copy-paste blocks. stage, that one has copy-paste blocks. You'll repeat most of these steps
**three times**, once per node — `talosctl README.md` calls out exactly
which commands are "once" vs. "once per node."
1. **Generate machine configuration.** `talosctl gen config` produces 1. **Generate machine configuration.** `talosctl gen config` produces
Talos's config files for your cluster, including Talos's config files for your cluster, with both
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml) — [`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml)
read that file, it's short and explains itself — which lets your one (scheduling + the VIP) and
node run workloads as well as the control plane. [`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) (storage
prep) baked in — read both files, they explain themselves. This
produces one `controlplane.yaml`, applied identically to all three
nodes; there's no separate worker config in this setup, since every
node is both.
2. **Apply the config to your node.** The node, freshly booted from the 2. **Apply the config to each node.** Each node, freshly booted from your
Talos ISO, is sitting in "maintenance mode" with no configuration. custom Talos ISO, is sitting in "maintenance mode" with no
`talosctl apply-config` sends it the config generated in step 1. The configuration. `talosctl apply-config` sends it the config from step 1.
node reboots, installs Talos to its disk, and comes up running for real. Run this once per node, pointed at that node's own address. Each node
reboots, installs Talos to its system disk, formats and mounts its
second disk for Longhorn, and comes up running for real.
3. **Bootstrap etcd.** `talosctl bootstrap` initializes the cluster's data 3. **Bootstrap etcd.** `talosctl bootstrap` initializes the cluster's data
store. This is a one-time, one-node-ever operation — even in a store. This is a one-time, one-node-ever operation, run against exactly
multi-node cluster you only ever bootstrap once. one of your three nodes — even though all three will end up running
etcd, only one of them needs telling to *start* the cluster; the other
two join the existing one once it exists.
4. **Retrieve your kubeconfig.** `talosctl kubeconfig` downloads the 4. **Retrieve your kubeconfig.** `talosctl kubeconfig` downloads the
credentials you'll use to talk to Kubernetes with `kubectl` — the credentials you'll use to talk to Kubernetes with `kubectl` — the
Kubernetes equivalent of the `talosconfig` you got in step 1. Kubernetes equivalent of the `talosconfig` you got in step 1. Point it
at the VIP, not at any one node.
5. **Verify.** `kubectl get nodes` should show one node, `Ready`. 5. **Verify.** `kubectl get nodes` should show all three nodes, `Ready`.
> **Ask Claude:** "what's the difference between a kubeconfig and a > **Ask Claude:** "what's the difference between a kubeconfig and a
> talosconfig, and why are they separate files?" is a good one to ask once > talosconfig, and why are they separate files?" is a good one to ask once
@@ -62,11 +111,14 @@ stage, that one has copy-paste blocks.
## Checklist ## Checklist
- [ ] `_out/controlplane.yaml`, `worker.yaml`, and `talosconfig` generated - [ ] VIP address chosen, with Claude's help, for your LAN
- [ ] Config applied to the node, node has rebooted and installed Talos - [ ] `_out/controlplane.yaml` and `talosconfig` generated, with both
- [ ] `talosctl bootstrap` run exactly once patches applied
- [ ] kubeconfig retrieved - [ ] Config applied to **all three** nodes; each has rebooted, installed
- [ ] `kubectl get nodes` shows one `Ready` node Talos, and mounted its Longhorn disk
- [ ] `talosctl bootstrap` run exactly once, against exactly one node
- [ ] kubeconfig retrieved, pointed at the VIP
- [ ] `kubectl get nodes` shows all three nodes `Ready`
Once your node is `Ready`, continue to Once all three nodes are `Ready`, continue to
[`docs/03-installing-argocd.md`](03-installing-argocd.md). [`docs/03-installing-argocd.md`](03-installing-argocd.md).
+1 -1
View File
@@ -46,7 +46,7 @@ each stage does:
> **Ask Claude:** "why is a port-forward good enough for now, and what > **Ask Claude:** "why is a port-forward good enough for now, and what
> would I need instead to reach this Argo CD from somewhere other than my > would I need instead to reach this Argo CD from somewhere other than my
> own machine?" — a preview of the Ingress topic in > own machine?" — a preview of the Ingress topic in
> [`docs/08-growing-up.md`](08-growing-up.md). > [`docs/09-growing-up.md`](09-growing-up.md).
## Checklist ## Checklist
+41 -36
View File
@@ -22,30 +22,33 @@ new app.
argocd/app-of-apps/app-of-apps.yaml (you apply this ONE file, by hand, once) argocd/app-of-apps/app-of-apps.yaml (you apply this ONE file, by hand, once)
│ │
│ watches apps/, non-recursively │ watches apps/, non-recursively
├──────────────┬───────────────┐ ├───────────────┬────────────────┬───────────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼
apps/demo.yaml apps/metallb.yaml (more, later — just add a file) apps/demo.yaml apps/metallb.yaml apps/longhorn.yaml (more, later)
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
apps/demo/*.yaml apps/metallb/* (the actual manifests / Helm chart) apps/demo/*.yaml apps/metallb/* apps/longhorn/* (manifests / Helm charts)
``` ```
Three layers, each watching the next. You only ever touch the bottom two — Four layers wide, each watching the next. You only ever touch the bottom
the app-of-apps is set-and-forget. Notice `apps/metallb.yaml` is already two — the app-of-apps is set-and-forget. Notice `apps/metallb.yaml` and
here — see [`docs/05-metallb.md`](05-metallb.md) right after this one. `apps/longhorn.yaml` are already here — see
[`docs/05-metallb.md`](05-metallb.md) and
[`docs/06-longhorn.md`](06-longhorn.md) right after this one.
## Why `apps/` isn't recursive ## Why `apps/` isn't recursive
Look at [`argocd/app-of-apps/app-of-apps.yaml`](../argocd/app-of-apps/app-of-apps.yaml) Look at [`argocd/app-of-apps/app-of-apps.yaml`](../argocd/app-of-apps/app-of-apps.yaml)
— specifically `directory.recurse: false`. The `apps/` folder contains both — specifically `directory.recurse: false`. The `apps/` folder contains both
Application manifests (`apps/demo.yaml`, `apps/metallb.yaml`) *and* Application manifests (`apps/demo.yaml`, `apps/metallb.yaml`,
subfolders holding the real workload manifests those Applications point to `apps/longhorn.yaml`) *and* subfolders holding the real workload manifests
(`apps/demo/*.yaml`, `apps/metallb/*`). Non-recursive means the those Applications point to (`apps/demo/*.yaml`, `apps/metallb/*`,
app-of-apps only ever looks at files directly inside `apps/`, never inside `apps/longhorn/*`). Non-recursive means the app-of-apps only ever looks at
its subfolders — so it only ever sees Application definitions, and leaves files directly inside `apps/`, never inside its subfolders — so it only
the actual workload manifests to the Application that owns them. Without ever sees Application definitions, and leaves the actual workload
this, the app-of-apps would try to apply raw Deployments and Helm chart manifests to the Application that owns them. Without this, the
files itself, duplicating and conflicting with what those Applications are app-of-apps would try to apply raw Deployments and Helm chart files
itself, duplicating and conflicting with what those Applications are
already doing. already doing.
## Every Application needs its own `repoURL` ## Every Application needs its own `repoURL`
@@ -59,19 +62,19 @@ that has it, not just the one you apply by hand:
grep -rl YOUR_GITHUB_USERNAME . grep -rl YOUR_GITHUB_USERNAME .
``` ```
You should see three files: `argocd/app-of-apps/app-of-apps.yaml`, You should see four files: `argocd/app-of-apps/app-of-apps.yaml`,
`apps/demo.yaml`, and `apps/metallb.yaml`. Edit `spec.source.repoURL` in `apps/demo.yaml`, `apps/metallb.yaml`, and `apps/longhorn.yaml`. Edit
each to point at your fork. `spec.source.repoURL` in each to point at your fork.
## Steps ## Steps
1. Fix `repoURL` in all three files (above). 1. Fix `repoURL` in all four files (above).
2. Commit and push — this part matters more than it looks: the 2. Commit and push — this part matters more than it looks: the
app-of-apps reads `apps/demo.yaml` and `apps/metallb.yaml` *from Git*, app-of-apps reads `apps/demo.yaml`, `apps/metallb.yaml`, and
not from your local edits, so they have to be pushed before you apply `apps/longhorn.yaml` *from Git*, not from your local edits, so they
anything. have to be pushed before you apply anything.
```sh ```sh
git add argocd/app-of-apps/app-of-apps.yaml apps/demo.yaml apps/metallb.yaml git add argocd/app-of-apps/app-of-apps.yaml apps/demo.yaml apps/metallb.yaml apps/longhorn.yaml
git commit -m "Point Applications at my fork" git commit -m "Point Applications at my fork"
git push git push
``` ```
@@ -79,13 +82,15 @@ each to point at your fork.
```sh ```sh
kubectl apply -f argocd/app-of-apps/app-of-apps.yaml kubectl apply -f argocd/app-of-apps/app-of-apps.yaml
``` ```
4. Watch it sync, then watch `demo` and `metallb` appear underneath it: 4. Watch it sync, then watch `demo`, `metallb`, and `longhorn` appear
underneath it:
```sh ```sh
argocd app list argocd app list
``` ```
You should see `app-of-apps`, `demo`, and `metallb`, all `Synced` and You should see `app-of-apps`, `demo`, `metallb`, and `longhorn`, all
`Healthy` within a minute or two (MetalLB's own install takes a little `Synced` and `Healthy` within a few minutes (MetalLB and Longhorn's own
longer than the demo app). installs take longer than the demo app — Longhorn especially, since it
has to find and prepare a disk on every node).
5. Confirm the demo app is actually running: 5. Confirm the demo app is actually running:
```sh ```sh
kubectl -n demo get pods,svc kubectl -n demo get pods,svc
@@ -95,23 +100,23 @@ each to point at your fork.
hands those out real addresses on bare metal until MetalLB is hands those out real addresses on bare metal until MetalLB is
configured, which is exactly the next doc. configured, which is exactly the next doc.
If `app-of-apps`, `demo`, and `metallb` all show `Synced`/`Healthy`, the If `app-of-apps`, `demo`, `metallb`, and `longhorn` all show
loop — Git → Argo CD → cluster — is already proven end to end; you just `Synced`/`Healthy`, the loop — Git → Argo CD → cluster — is already proven
can't see the demo app in a browser yet. end to end; you just can't see the demo app in a browser yet.
> **Ask Claude:** "walk me through exactly what happened, in order, from the > **Ask Claude:** "walk me through exactly what happened, in order, from the
> moment I ran `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml` to > moment I ran `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml` to
> these three Applications existing." Making sure you can narrate that > these four Applications existing." Making sure you can narrate that
> chain yourself is the real checkpoint for this doc — everything from > chain yourself is the real checkpoint for this doc — everything from
> here on is a variation on it. > here on is a variation on it.
## Checklist ## Checklist
- [ ] `repoURL` fixed in all three files listed above - [ ] `repoURL` fixed in all four files listed above
- [ ] Fix committed and pushed - [ ] Fix committed and pushed
- [ ] `app-of-apps` applied and shows `Synced`/`Healthy` - [ ] `app-of-apps` applied and shows `Synced`/`Healthy`
- [ ] `demo` and `metallb` Applications appeared automatically, both - [ ] `demo`, `metallb`, and `longhorn` Applications appeared
`Synced`/`Healthy` automatically, all `Synced`/`Healthy`
- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running - [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running
(Service `EXTERNAL-IP` still `<pending>` — expected for now) (Service `EXTERNAL-IP` still `<pending>` — expected for now)
+2 -2
View File
@@ -34,7 +34,7 @@ Helm chart that does two things at once (see
- Pulls in the official upstream MetalLB chart as a dependency — this is - Pulls in the official upstream MetalLB chart as a dependency — this is
also the real, working example of a Helm-based app mentioned in also the real, working example of a Helm-based app mentioned in
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md). [`docs/08-adding-new-apps.md`](08-adding-new-apps.md).
- Carries two templates of its own - Carries two templates of its own
([`apps/metallb/templates/`](../apps/metallb/templates)): an ([`apps/metallb/templates/`](../apps/metallb/templates)): an
`IPAddressPool` and an `L2Advertisement` — MetalLB's own custom `IPAddressPool` and an `L2Advertisement` — MetalLB's own custom
@@ -152,4 +152,4 @@ the only place this address gets recorded at all.
- [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser - [ ] `kubectl -n demo get svc demo` shows a real `EXTERNAL-IP`, page loads in a browser
- [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding - [ ] `argocd-server` patched to `LoadBalancer`, reachable at its own address without port-forwarding
Continue to [`docs/06-your-first-change.md`](06-your-first-change.md). Continue to [`docs/06-longhorn.md`](06-longhorn.md).
+100
View File
@@ -0,0 +1,100 @@
# 06 — Longhorn: real, replicated storage
The app-of-apps you applied in [`docs/04-app-of-apps.md`](04-app-of-apps.md)
already brought up `longhorn` alongside `demo` and `metallb`. This doc
explains what it does and how to actually use it — there's no address to
pick this time, unlike MetalLB; the interesting decisions already happened
back in [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
## Why this is needed at all
A Kubernetes Pod's own filesystem disappears the moment that Pod does —
fine for a stateless web server, useless for anything that needs to keep
data around (a database, uploaded files, anything you'd be upset to lose).
Kubernetes' answer is a **PersistentVolumeClaim (PVC)**: an app asks for
some amount of storage, and something has to actually provide it. On a
cloud provider, that something is the cloud's own disk service. On bare
metal — your three Talos nodes — nothing provides that on its own, the
same gap MetalLB fills for IP addresses rather than storage.
**Longhorn** is that provider: it turns a plain disk on each node into one
pool of storage, and — because your cluster has three nodes — it
replicates each volume across more than one of them. Lose a node, and the
data backing any volume that had a replica elsewhere is still there. This
is also why Longhorn is genuinely a **three-node** story: with one node,
"replicated storage" has nothing to replicate *to*.
## What's already running
**`apps/longhorn.yaml`** → one Application, pointing at `apps/longhorn`, a
Helm chart that pulls in the official upstream Longhorn chart (see
[`apps/longhorn/Chart.yaml`](../apps/longhorn/Chart.yaml)) — the same
umbrella-chart pattern as MetalLB, just without any extra templates of its
own this time.
The interesting configuration already happened at the Talos layer, back in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml): each node
found, formatted, and mounted its own ≥900GB disk at
`/var/mnt/longhorn`. [`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml)
points Longhorn's `defaultDataPath` at that exact path — this is the thing
that makes Longhorn use *that* disk instead of carving space out of each
node's root filesystem (small, and on Talos, not meant for this kind of
data anyway).
## Verify it's actually working
```sh
kubectl -n longhorn-system get pods
```
Expect a `longhorn-manager` and `longhorn-csi-plugin` Pod on each of your
three nodes, plus one `instance-manager` per node once a volume exists,
all `Running`.
```sh
kubectl get storageclass
```
Longhorn's chart creates a default `StorageClass` called `longhorn`
automatically — this is what makes it the thing Kubernetes reaches for
whenever an app asks for storage without naming one specifically.
## Try it: give the demo app real storage
The demo app's HTML currently lives in a ConfigMap (see
[`docs/07-your-first-change.md`](07-your-first-change.md)) — fine for a
few lines of text, but not how a real app would store real data. As a
hands-on check that Longhorn actually works, ask Claude to help you:
1. Add a small `PersistentVolumeClaim` to `apps/demo/`, requesting a
couple of gigabytes with no `storageClassName` set (so it uses
Longhorn's default).
2. Mount it into the demo Deployment somewhere harmless, like
`/usr/share/nginx/html/data`.
3. Push it, let Argo CD sync, and confirm:
```sh
kubectl -n demo get pvc
```
should show your claim `Bound`, and
```sh
kubectl -n longhorn-system get volumes
```
should show a matching volume, `healthy`, with more than one replica.
This isn't part of the checklist below — treat it as an optional exercise
once the pre-installed pieces are confirmed working, not a requirement to
move on.
> **Ask Claude:** "what would happen to that PVC's data if I lost one of
> my three nodes right now?" — worth being able to answer before you trust
> this with anything real.
## Checklist
- [ ] `kubectl -n longhorn-system get pods` shows manager/CSI Pods
`Running` on all three nodes
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
with a healthy, replicated Longhorn volume behind it
Continue to [`docs/07-your-first-change.md`](07-your-first-change.md).
@@ -1,4 +1,4 @@
# 06 — Your first change # 07 — Your first change
Everything so far proved Argo CD can deploy what's in Git. This doc proves Everything so far proved Argo CD can deploy what's in Git. This doc proves
the part that actually matters day to day: changing what's in Git and the part that actually matters day to day: changing what's in Git and
@@ -90,5 +90,5 @@ for the loop that just ran.
- [ ] Saw the new text in the browser - [ ] Saw the new text in the browser
- [ ] (Optional) Watched self-healing revert a hand-made change - [ ] (Optional) Watched self-healing revert a hand-made change
Continue to [`docs/07-adding-new-apps.md`](07-adding-new-apps.md) to add Continue to [`docs/08-adding-new-apps.md`](08-adding-new-apps.md) to add
something of your own. something of your own.
@@ -1,4 +1,4 @@
# 07 — Adding a new app # 08 — Adding a new app
This is the payoff for the app-of-apps structure: adding something new is This is the payoff for the app-of-apps structure: adding something new is
"drop a file here," not a special procedure. "drop a file here," not a special procedure.
@@ -32,8 +32,13 @@ To add a new app called, say, `whoami`:
`whoami.yaml`, creates the `whoami` Application, which in turn deploys `whoami.yaml`, creates the `whoami` Application, which in turn deploys
everything in `apps/whoami/`. everything in `apps/whoami/`.
4. Confirm: `argocd app list` should show `whoami` alongside `demo` and 4. Confirm: `argocd app list` should show `whoami` alongside `demo`,
`metallb`. `metallb`, and `longhorn`.
If your new app needs storage, request it with a plain
`PersistentVolumeClaim` and no `storageClassName` — Longhorn's chart set
itself as the cluster's default StorageClass, so it's used automatically.
See [`docs/06-longhorn.md`](06-longhorn.md).
## Manifest-based vs. Helm-based apps ## Manifest-based vs. Helm-based apps
@@ -83,5 +88,5 @@ address shows up in it.
- [ ] `argocd app list` shows the new app, `Synced`/`Healthy` - [ ] `argocd app list` shows the new app, `Synced`/`Healthy`
- [ ] If it has a `LoadBalancer` Service, its address is recorded in `docs/network.md` - [ ] If it has a `LoadBalancer` Service, its address is recorded in `docs/network.md`
Continue to [`docs/08-growing-up.md`](08-growing-up.md) for what to add Continue to [`docs/09-growing-up.md`](09-growing-up.md) for what to add
next as this cluster grows beyond a learning setup. next as this cluster grows beyond a learning setup.
+36 -19
View File
@@ -1,4 +1,4 @@
# 08 — Growing up: what to add next # 09 — Growing up: what to add next
Everything up to here is deliberately minimal — enough to understand the Everything up to here is deliberately minimal — enough to understand the
whole loop, nothing more. This doc is a map of the natural next steps, each whole loop, nothing more. This doc is a map of the natural next steps, each
@@ -7,26 +7,43 @@ for you; each is its own project once you're ready.
## More nodes ## More nodes
Right now, one machine is both control plane and worker Your three nodes already give you real resilience — losing any one still
(`allowSchedulingOnControlPlanes: true` in leaves a majority of control-plane voters (see
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml)). That's [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker))
a single point of failure: if that machine dies, the cluster is gone. and, once Longhorn has real volumes, a replica of your data likely
survives too. Growing further is about *capacity*, not fault tolerance you
don't already have — so beyond this point, add plain **worker** nodes, not
more control-plane ones:
To grow beyond it: 1. Boot a new machine from the same custom Talos ISO as the first three
(see [`docs/01-prerequisites.md`](01-prerequisites.md)).
1. Boot a second machine from the Talos ISO, same as the first. 2. Apply the **worker** config instead of the control-plane one — you
2. Generate a **worker** config instead of a control-plane one — you
already have `_out/worker.yaml` from already have `_out/worker.yaml` from
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now. [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now:
3. `talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml` ```sh
4. `kubectl get nodes` should show two nodes shortly after. talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
```
3. `kubectl get nodes` should show a fourth node shortly after, without
the `control-plane` role.
For real resilience against a control-plane node failing (not just adding ### Why not a fourth control-plane node
worker capacity), you'd want an odd number of control-plane nodes — 3 is
the normal minimum — because etcd (the cluster's data store) uses majority It's tempting to reach for a fourth control-plane node when you're growing
voting to agree on state, and majority voting needs an odd number to avoid a cluster you already made resilient with three — but it actually makes
ties. That's a bigger step than this doc covers in detail; ask Claude when things worse, not better. etcd (the control-plane database) needs a
you're actually ready to take it. **majority** of its members to agree before it can do anything, and
majority only means something clean with an odd number: with 3 members, 1
can fail and 2 (a majority) still agree — you can lose one. With 4, you
still need 3 to agree, so you can *still* only lose one — you paid for an
extra machine and gained nothing. Worse, 4 is *more* likely to have
exactly a 2-2 split during a network hiccup, which stalls the cluster
until it's resolved. If you ever do want more control-plane fault
tolerance than three nodes gives you, the next meaningful number is 5 —
not 4.
> **Ask Claude:** "walk me through the quorum math for 3 vs. 4 vs. 5
> control-plane nodes" if you want to see this worked out concretely
> rather than taking it on faith.
## Ingress ## Ingress
@@ -40,7 +57,7 @@ so big, and there's no way to route by hostname (`app1.example.com` vs
An **Ingress controller** is itself just another app you'd deploy the same An **Ingress controller** is itself just another app you'd deploy the same
way as anything else in this repo (see way as anything else in this repo (see
[`docs/07-adding-new-apps.md`](07-adding-new-apps.md)) — commonly as a [`docs/08-adding-new-apps.md`](08-adding-new-apps.md)) — commonly as a
Helm-based app, since most ingress controllers are distributed that way, Helm-based app, since most ingress controllers are distributed that way,
same as MetalLB. It typically sits behind one single MetalLB same as MetalLB. It typically sits behind one single MetalLB
`LoadBalancer` Service (one IP, total, for every app), and you add an `LoadBalancer` Service (one IP, total, for every app), and you add an
+37 -21
View File
@@ -1,25 +1,27 @@
# Guided bootstrap checklist # Guided bootstrap checklist
A single-page, ordered checklist covering the entire path from bare A single-page, ordered checklist covering the entire path from bare
machine to a working GitOps loop. It's meant to be worked through **with machines to a working GitOps loop. It's meant to be worked through **with
Claude**, one item at a time — ask Claude to walk you through this list, Claude**, one item at a time — ask Claude to walk you through this list,
and it will run each step with you, wait for your output, confirm it and it will run each step with you, wait for your output, confirm it
worked, and only then move to the next one. See `CLAUDE.md` at the repo worked, and only then move to the next one. See `CLAUDE.md` at the repo
root for how Claude is primed to do this. root for how Claude is primed to do this.
Each item names the doc it's drawn from, in case you want the full Each item names the doc it's drawn from, in case you want the full
explanation before or after running it. Replace `<NODE_IP>`, explanation before or after running it. Replace `<NODE1_IP>`/`<NODE2_IP>`/`<NODE3_IP>`,
`YOUR_GITHUB_USERNAME`, and the MetalLB IP addresses with your real values `<VIP_IP>`, `YOUR_GITHUB_USERNAME`, and the MetalLB IP addresses with your
throughout — Claude will remind you if you paste a command still real values throughout — Claude will remind you if you paste a command
containing a placeholder, and will ask you for the IP addresses directly still containing a placeholder, and will ask you for all of these directly
in Phase 5 rather than guessing them. in Phases 2 and 5 rather than guessing them.
## Phase 1 — Prerequisites ([`01`](01-prerequisites.md)) ## Phase 1 — Prerequisites ([`01`](01-prerequisites.md))
- [ ] Target machine chosen, reachable on the network, with a stable - [ ] Three target machines chosen, each network-reachable, each with a
(ideally DHCP-reserved) IP address stable (ideally DHCP-reserved) IP address, each with a second disk
- [ ] Talos amd64 ISO downloaded, machine booted from it, sitting at the ≥900GB set aside for Longhorn
boot screen - [ ] Custom Talos schematic built at factory.talos.dev with `iscsi-tools`
and `util-linux-tools`, ISO downloaded
- [ ] All three machines booted from that ISO, sitting at the boot screen
- [ ] `talosctl version --client` runs successfully - [ ] `talosctl version --client` runs successfully
- [ ] `kubectl version --client` runs successfully - [ ] `kubectl version --client` runs successfully
- [ ] `argocd version --client` runs successfully - [ ] `argocd version --client` runs successfully
@@ -27,12 +29,16 @@ in Phase 5 rather than guessing them.
## Phase 2 — Talos bootstrap ([`02`](02-talos-bootstrap.md), [`talos/README.md`](../talos/README.md)) ## Phase 2 — Talos bootstrap ([`02`](02-talos-bootstrap.md), [`talos/README.md`](../talos/README.md))
- [ ] `talosctl gen config gitops-starter https://<NODE_IP>:6443 --output-dir _out --config-patch-control-plane @talos/patch-controlplane.yaml` run — `_out/controlplane.yaml`, `worker.yaml`, `talosconfig` exist - [ ] **Ask Claude** to help pick: your three nodes' IP addresses (or
confirm DHCP reservations) and a separate VIP address — all based on
your actual LAN/DHCP setup
- [ ] `talosctl gen config gitops-starter https://<VIP_IP>:6443 --output-dir _out --config-patch-control-plane @talos/patch-controlplane.yaml --config-patch-control-plane @talos/patch-longhorn.yaml` run — `_out/controlplane.yaml`, `worker.yaml`, `talosconfig` exist
- [ ] `TALOSCONFIG` exported (or `talosctl config endpoint`/`node` set) to `_out/talosconfig` - [ ] `TALOSCONFIG` exported (or `talosctl config endpoint`/`node` set) to `_out/talosconfig`
- [ ] `talosctl apply-config --insecure --nodes <NODE_IP> --file _out/controlplane.yaml` run, node rebooted - [ ] For **each** of the three nodes: `talosctl get disks --insecure --nodes <NODE_IP>` confirms two disks, then `talosctl apply-config --insecure --nodes <NODE_IP> --file _out/controlplane.yaml` run, node rebooted
- [ ] `talosctl bootstrap --nodes <NODE_IP>` run — **exactly once** - [ ] `talosctl bootstrap --nodes <NODE1_IP>` run — **exactly once**, against **exactly one** node
- [ ] `talosctl kubeconfig --nodes <NODE_IP> .` run, `KUBECONFIG` pointed at the result - [ ] `talosctl kubeconfig --nodes <VIP_IP> .` run, `KUBECONFIG` pointed at the result
- [ ] `kubectl get nodes` shows one node, `Ready` - [ ] `kubectl get nodes` shows **all three** nodes, `Ready`
- [ ] `docs/network.md` updated with all three node addresses and the VIP
## Phase 3 — Argo CD install ([`03`](03-installing-argocd.md), [`argocd/README.md`](../argocd/README.md)) ## Phase 3 — Argo CD install ([`03`](03-installing-argocd.md), [`argocd/README.md`](../argocd/README.md))
@@ -45,13 +51,15 @@ in Phase 5 rather than guessing them.
## Phase 4 — Hand the cluster to Git ([`04`](04-app-of-apps.md)) ## Phase 4 — Hand the cluster to Git ([`04`](04-app-of-apps.md))
- [ ] `spec.source.repoURL` fixed in all three placeholder files: run - [ ] `spec.source.repoURL` fixed in all four placeholder files: run
`grep -rl YOUR_GITHUB_USERNAME .` to find them `grep -rl YOUR_GITHUB_USERNAME .` to find them
(`argocd/app-of-apps/app-of-apps.yaml`, `apps/demo.yaml`, `apps/metallb.yaml`) (`argocd/app-of-apps/app-of-apps.yaml`, `apps/demo.yaml`,
`apps/metallb.yaml`, `apps/longhorn.yaml`)
- [ ] Edit committed and pushed to your fork's `main` branch - [ ] Edit committed and pushed to your fork's `main` branch
- [ ] `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml` - [ ] `kubectl apply -f argocd/app-of-apps/app-of-apps.yaml`
- [ ] `argocd app list` shows `app-of-apps`, `Synced`/`Healthy` - [ ] `argocd app list` shows `app-of-apps`, `Synced`/`Healthy`
- [ ] `argocd app list` shows `demo` and `metallb`, appeared automatically, both `Synced`/`Healthy` - [ ] `argocd app list` shows `demo`, `metallb`, and `longhorn`, appeared
automatically, all `Synced`/`Healthy`
- [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running - [ ] `kubectl -n demo get pods,svc` shows the demo app's Pod running
(Service `EXTERNAL-IP` still `<pending>` — expected for now) (Service `EXTERNAL-IP` still `<pending>` — expected for now)
@@ -68,7 +76,15 @@ in Phase 5 rather than guessing them.
- [ ] `argocd-server` Service patched to `LoadBalancer` with your chosen Argo CD address (`kubectl -n argocd patch svc argocd-server ...`) - [ ] `argocd-server` Service patched to `LoadBalancer` with your chosen Argo CD address (`kubectl -n argocd patch svc argocd-server ...`)
- [ ] Argo CD UI reachable at `https://<that-address>` with no port-forward running - [ ] Argo CD UI reachable at `https://<that-address>` with no port-forward running
## Phase 6 — Prove the loop ([`06`](06-your-first-change.md)) ## Phase 6 — Longhorn: real storage ([`06`](06-longhorn.md))
- [ ] `kubectl -n longhorn-system get pods` shows manager/CSI Pods
`Running` on all three nodes
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
with a healthy, replicated Longhorn volume behind it
## Phase 7 — Prove the loop ([`07`](07-your-first-change.md))
- [ ] Edited the `<h1>` text in `apps/demo/configmap.yaml` - [ ] Edited the `<h1>` text in `apps/demo/configmap.yaml`
- [ ] Committed and pushed - [ ] Committed and pushed
@@ -85,6 +101,6 @@ push" and the page updating. If you can't yet, that's the sign to ask
Claude for a walkthrough of whichever phase felt shaky before calling this Claude for a walkthrough of whichever phase felt shaky before calling this
finished. finished.
From here: [`07-adding-new-apps.md`](07-adding-new-apps.md) to add From here: [`08-adding-new-apps.md`](08-adding-new-apps.md) to add
something of your own, or [`08-growing-up.md`](08-growing-up.md) for the something of your own, or [`09-growing-up.md`](09-growing-up.md) for the
bigger next steps (more nodes, ingress, secrets management). bigger next steps (more nodes, ingress, secrets management).
+61 -7
View File
@@ -29,7 +29,7 @@ steps — installing Argo CD, applying the app-of-apps — that get the
GitOps loop started before Git takes over. GitOps loop started before Git takes over.
**cluster** — A group of one or more machines (nodes) working together as **cluster** — A group of one or more machines (nodes) working together as
one Kubernetes system. This repo's cluster starts as a single node. one Kubernetes system. This repo's cluster starts as three nodes.
**CNI (Container Network Interface)** — The plugin responsible for giving **CNI (Container Network Interface)** — The plugin responsible for giving
Pods network connectivity and IP addresses. Talos ships Flannel, a CNI, by Pods network connectivity and IP addresses. Talos ships Flannel, a CNI, by
@@ -42,9 +42,9 @@ demo app's HTML page.
**control plane** — The set of components that run the cluster itself **control plane** — The set of components that run the cluster itself
(API server, scheduler, etcd) as opposed to your applications. Normally (API server, scheduler, etcd) as opposed to your applications. Normally
kept on separate machines from your workloads; this repo's single-node kept on separate machines from your workloads; this repo's three nodes
setup deliberately blurs that line for simplicity — see deliberately blur that line, each acting as control plane and worker at
[`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml). once — see [`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml).
**CRD (Custom Resource Definition)** — A way to teach Kubernetes about a **CRD (Custom Resource Definition)** — A way to teach Kubernetes about a
new *kind* of object beyond its built-ins (Pod, Service, etc.). Argo CD's new *kind* of object beyond its built-ins (Pod, Service, etc.). Argo CD's
@@ -74,13 +74,19 @@ the real system to match it. The core idea this whole repo demonstrates.
**Ingress** — A Kubernetes object that routes external HTTP(S) traffic to **Ingress** — A Kubernetes object that routes external HTTP(S) traffic to
services inside the cluster by hostname/path, via an Ingress controller. services inside the cluster by hostname/path, via an Ingress controller.
Not used in this repo's v1 — see Not used in this repo's v1 — see
[`docs/08-growing-up.md`](08-growing-up.md). [`docs/09-growing-up.md`](09-growing-up.md).
**IPAddressPool** — A MetalLB custom resource (see CRD) declaring which IP **IPAddressPool** — A MetalLB custom resource (see CRD) declaring which IP
addresses it's allowed to hand out to `LoadBalancer` Services. Configured addresses it's allowed to hand out to `LoadBalancer` Services. Configured
in [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) in [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml)
— see [`docs/05-metallb.md`](05-metallb.md). — see [`docs/05-metallb.md`](05-metallb.md).
**iSCSI** — A protocol for accessing block storage over a network as if it
were a local disk. Longhorn's underlying engine uses it, which is why
nodes need the `iscsi-tools` Talos system extension (see below) baked
into their image before Longhorn will work — see
[`docs/01-prerequisites.md`](01-prerequisites.md).
**kubeconfig** — A file holding the credentials and connection info **kubeconfig** — A file holding the credentials and connection info
`kubectl` (and other Kubernetes tools) use to talk to a specific cluster. `kubectl` (and other Kubernetes tools) use to talk to a specific cluster.
Retrieved with `talosctl kubeconfig` in this setup. Retrieved with `talosctl kubeconfig` in this setup.
@@ -96,6 +102,12 @@ cloud provider this provisions an actual cloud load balancer; on bare
metal (this repo) MetalLB fills that role instead — see metal (this repo) MetalLB fills that role instead — see
[`docs/05-metallb.md`](05-metallb.md). [`docs/05-metallb.md`](05-metallb.md).
**Longhorn** — Software installed in this repo (`apps/longhorn.yaml`) that
turns a plain disk on each node into one pool of storage, replicated
across nodes, so apps can request real persistent storage on bare metal —
a job a cloud provider's disk service would normally do. See
[`docs/06-longhorn.md`](06-longhorn.md).
**machine config** — Talos's equivalent of a Kubernetes manifest, but for **machine config** — Talos's equivalent of a Kubernetes manifest, but for
the operating system itself: a declarative file describing how a node the operating system itself: a declarative file describing how a node
should be configured, generated with `talosctl gen config` and sent to the should be configured, generated with `talosctl gen config` and sent to the
@@ -116,19 +128,37 @@ independently.
**node** — A single machine (physical or virtual) that's part of a **node** — A single machine (physical or virtual) that's part of a
Kubernetes cluster. Kubernetes cluster.
**PersistentVolumeClaim (PVC)** — How an app asks Kubernetes for storage:
"I need this much space," without caring which disk or which node
provides it. Longhorn is what actually fulfills that request in this
repo — see [`docs/06-longhorn.md`](06-longhorn.md).
**Pod** — The smallest deployable unit in Kubernetes: one or more **Pod** — The smallest deployable unit in Kubernetes: one or more
containers that run together, sharing network and storage. You'll rarely containers that run together, sharing network and storage. You'll rarely
create Pods directly — a Deployment creates and manages them for you. create Pods directly — a Deployment creates and manages them for you.
**quorum** — The minimum number of etcd members (a majority) that have to
agree before the cluster's control plane can do anything. This is why
control-plane node counts matter, and why 3 is a meaningfully different
number than 2 or 4 — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker)
and [`docs/09-growing-up.md`](09-growing-up.md#why-not-a-fourth-control-plane-node).
**reconciliation** — The repeating process of comparing desired state **reconciliation** — The repeating process of comparing desired state
(Git, for Argo CD) against actual state (the live cluster) and correcting (Git, for Argo CD) against actual state (the live cluster) and correcting
any difference. See the diagram in any difference. See the diagram in
[`docs/00-what-is-all-this.md`](00-what-is-all-this.md#the-reconciliation-loop). [`docs/00-what-is-all-this.md`](00-what-is-all-this.md#the-reconciliation-loop).
**schematic** — A custom Talos boot image, built at
[factory.talos.dev](https://factory.talos.dev) by selecting which system
extensions (see below) it should include. This repo's nodes need one with
`iscsi-tools` and `util-linux-tools` for Longhorn to work — see
[`docs/01-prerequisites.md`](01-prerequisites.md).
**Secret** — A Kubernetes object for holding sensitive data. Its values are **Secret** — A Kubernetes object for holding sensitive data. Its values are
only base64-*encoded*, not encrypted — never commit a raw Secret manifest only base64-*encoded*, not encrypted — never commit a raw Secret manifest
containing real credentials to Git. See containing real credentials to Git. See
[`docs/08-growing-up.md`](08-growing-up.md) for how to do this safely. [`docs/09-growing-up.md`](09-growing-up.md) for how to do this safely.
**selfHeal** — An Argo CD sync policy setting: if the live cluster drifts **selfHeal** — An Argo CD sync policy setting: if the live cluster drifts
from what's in Git (e.g. someone runs `kubectl edit`), Argo CD reverts it from what's in Git (e.g. someone runs `kubectl edit`), Argo CD reverts it
@@ -141,6 +171,12 @@ uses a `LoadBalancer` Service for the demo app, backed by MetalLB — see
[`apps/demo/service.yaml`](../apps/demo/service.yaml) and [`apps/demo/service.yaml`](../apps/demo/service.yaml) and
[`docs/05-metallb.md`](05-metallb.md). [`docs/05-metallb.md`](05-metallb.md).
**StorageClass** — A Kubernetes object describing a "kind" of storage
that's available (who provides it, how). Longhorn's chart registers itself
as the cluster's default StorageClass, so a plain
`PersistentVolumeClaim` with no class named gets Longhorn storage
automatically — see [`docs/06-longhorn.md`](06-longhorn.md).
**sync** — Argo CD's term for actually applying a change (a reconciliation **sync** — Argo CD's term for actually applying a change (a reconciliation
pass that found and fixed a difference). "Synced" means the cluster pass that found and fixed a difference). "Synced" means the cluster
currently matches Git. currently matches Git.
@@ -153,9 +189,27 @@ waits for a wave to be healthy before starting the next. Used in
apply before the `IPAddressPool`/`L2Advertisement` that depend on them — apply before the `IPAddressPool`/`L2Advertisement` that depend on them —
see [`docs/05-metallb.md`](05-metallb.md). see [`docs/05-metallb.md`](05-metallb.md).
**Talos Linux** — The operating system this repo's node(s) run: built only **system extension** — An optional add-on baked into a Talos boot image at
build time (see "schematic," above), since Talos has no package manager to
install one onto an already-running node. This repo's nodes need
`iscsi-tools` and `util-linux-tools` for Longhorn.
**Talos Linux** — The operating system this repo's nodes run: built only
to run Kubernetes, configured entirely declaratively over the network, with to run Kubernetes, configured entirely declaratively over the network, with
no SSH or shell. See [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md). no SSH or shell. See [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
**talosconfig** — The credentials file `talosctl` uses to talk to a Talos **talosconfig** — The credentials file `talosctl` uses to talk to a Talos
node's API — the Talos-level equivalent of a kubeconfig. node's API — the Talos-level equivalent of a kubeconfig.
**UserVolumeConfig** — A standalone Talos configuration resource (not part
of the normal `machine:`/`cluster:` structure) that tells Talos to find,
format, and mount a disk on its own, based on rules you give it (which
disk to pick, minimum size). Used in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml) to prepare
each node's storage for Longhorn.
**VIP (Virtual IP)** — One IP address that Talos keeps pointed at whichever
of several control-plane nodes is currently healthy, re-electing within
seconds if that node goes down. Everything that talks to this cluster
should use the VIP, not any one node's own address — see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#the-vip-one-address-for-a-cluster-that-moves).
+17 -3
View File
@@ -13,6 +13,21 @@ assigned or changed, so you (or Claude) never have to reverse-engineer
> instruction that tells Claude to do this on its own when it makes such a > instruction that tells Claude to do this on its own when it makes such a
> change itself. > change itself.
## The three nodes and the VIP
None of these are from the MetalLB pool below — they're the machines
themselves and the shared address Talos elects between them. Set in
[`talos/README.md`](../talos/README.md); see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md) for what the VIP is
and why it's separate from any one node's address.
| Address | What | Port | URL |
|---|---|---|---|
| `EDIT ME` | Node 1 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Node 2 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Node 3 | 6443 | `https://EDIT ME:6443` |
| `EDIT ME` | Cluster VIP (use this one, not a node's own address) | 6443 | `https://EDIT ME:6443` |
## MetalLB pool ## MetalLB pool
| | | | | |
@@ -20,16 +35,15 @@ assigned or changed, so you (or Claude) never have to reverse-engineer
| Range | `EDIT ME — e.g. 192.168.1.240-192.168.1.250` | | Range | `EDIT ME — e.g. 192.168.1.240-192.168.1.250` |
| Configured in | [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) | | Configured in | [`apps/metallb/templates/ipaddresspool.yaml`](../apps/metallb/templates/ipaddresspool.yaml) |
## Allocated addresses ## Allocated addresses (from the MetalLB pool)
| Address | What | Namespace | Port(s) | URL | Source | | Address | What | Namespace | Port(s) | URL | Source |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| `<NODE_IP>` | Talos node — Kubernetes API server | — | 6443 | `https://<NODE_IP>:6443` | Not from the MetalLB pool — the node's own address, set in [`talos/README.md`](../talos/README.md) |
| `EDIT ME` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://EDIT ME` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) | | `EDIT ME` | Argo CD UI/API (`argocd-server`) | `argocd` | 443 | `https://EDIT ME` | Patched to `LoadBalancer` by hand — see [`docs/05-metallb.md`](05-metallb.md) |
| `EDIT ME` | Demo app | `demo` | 80 | `http://EDIT ME` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) | | `EDIT ME` | Demo app | `demo` | 80 | `http://EDIT ME` | [`apps/demo/service.yaml`](../apps/demo/service.yaml) |
Add a row here every time you add a new app with a `LoadBalancer` Service Add a row here every time you add a new app with a `LoadBalancer` Service
(see [`docs/07-adding-new-apps.md`](07-adding-new-apps.md)). Leave the URL (see [`docs/08-adding-new-apps.md`](08-adding-new-apps.md)). Leave the URL
column's port number off only when it's the protocol default (80 for column's port number off only when it's the protocol default (80 for
`http://`, 443 for `https://`) — include it otherwise. `http://`, 443 for `https://`) — include it otherwise.
+68 -44
View File
@@ -5,64 +5,81 @@ This is the detailed, copy-pasteable companion to
first for the explanations — this one is here so you have every exact first for the explanations — this one is here so you have every exact
command in one place once you understand what they do. command in one place once you understand what they do.
Throughout, replace `<NODE_IP>` with your machine's real IP address. If it's Throughout, replace `<NODE1_IP>`, `<NODE2_IP>`, `<NODE3_IP>` with your three
a VM, give it a DHCP reservation (a fixed IP tied to its MAC address) in your machines' real IP addresses, and `<VIP_IP>` with the shared address you
router or hypervisor before you start, so the address never changes under picked (with Claude's help) for the cluster's VIP — see
the cluster. Talos bakes the IP into certificates during setup, and changing [`docs/02-talos-bootstrap.md`](../docs/02-talos-bootstrap.md#the-vip-one-address-for-a-cluster-that-moves).
it later means starting over. Give each machine a DHCP reservation (a fixed IP tied to its MAC address)
in your router or hypervisor before you start, so none of these addresses
change under the cluster. Talos bakes them into certificates during setup,
and changing one later means starting over for that node.
Steps 1–2 below run **once, total**. Steps 3 runs **once per node** (three
times). Steps 4–6 run **once, total**, against whichever node you like
(or, once it exists, the VIP).
All commands assume you're running them from the root of this repo. All commands assume you're running them from the root of this repo.
## 1. Generate machine configuration ## 1. Generate machine configuration
```sh ```sh
talosctl gen config gitops-starter https://<NODE_IP>:6443 \ talosctl gen config gitops-starter https://<VIP_IP>:6443 \
--output-dir _out \ --output-dir _out \
--config-patch-control-plane @talos/patch-controlplane.yaml --config-patch-control-plane @talos/patch-controlplane.yaml \
--config-patch-control-plane @talos/patch-longhorn.yaml
``` ```
Note the cluster endpoint here is the **VIP**, not any one node's address
— every node needs to agree in advance on where the cluster "lives," and
that address has to be the one that keeps working no matter which node is
currently answering.
This writes three files into `_out/` (already covered by this repo's This writes three files into `_out/` (already covered by this repo's
`.gitignore` — they contain private keys and must never be committed): `.gitignore` — they contain private keys and must never be committed):
- `controlplane.yaml` — the full machine configuration for a control-plane - `controlplane.yaml` — the machine configuration for a control-plane
node, with our single-node patch baked in. node, with both patches baked in. Applied identically to all three
- `worker.yaml` — a worker machine configuration. Unused for now (single nodes — there's no separate worker config in this setup, since every
node), kept for when you follow docs/08-growing-up.md. node is both.
- `worker.yaml` — a plain worker config, generated automatically but
unused here since every node in this repo's setup is both control plane
and worker. Kept around for [`docs/09-growing-up.md`](../docs/09-growing-up.md),
where a fourth-and-beyond node uses exactly this file.
- `talosconfig` — *your* client credentials for talking to the Talos API on - `talosconfig` — *your* client credentials for talking to the Talos API on
this cluster (equivalent in spirit to a kubeconfig, but for Talos itself this cluster (equivalent in spirit to a kubeconfig, but for Talos itself
rather than Kubernetes). rather than Kubernetes).
`gitops-starter` is the cluster name — an arbitrary label baked into the `gitops-starter` is the cluster name — an arbitrary label baked into the
generated certificates and config, used to tell clusters apart if you ever generated certificates and config, used to tell clusters apart if you ever
manage more than one. `https://<NODE_IP>:6443` is the cluster's Kubernetes manage more than one. Port 6443 is the standard Kubernetes API server port.
API endpoint: port 6443 is the standard Kubernetes API server port, and this
same address gets written into the certificates and your future kubeconfig.
## 2. Point talosctl at this cluster ## 2. Point talosctl at this cluster
```sh ```sh
export TALOSCONFIG=_out/talosconfig export TALOSCONFIG=_out/talosconfig
talosctl config endpoint <NODE_IP> talosctl config endpoint <NODE1_IP> <NODE2_IP> <NODE3_IP>
talosctl config node <NODE_IP>
``` ```
`endpoint` is which machine talosctl *talks to*. `node` is which machine a `endpoint` is which machine(s) talosctl is willing to *talk to* — giving
command *applies to*. On a single-node cluster they're the same address, but it all three means it can still reach the cluster if any one of them is
Talos always keeps the two concepts separate, because on a multi-node down. You'll pass `--nodes` per-command below for which machine a given
cluster you often want to ask one node about another. command actually *applies to* — Talos always keeps those two concepts
separate.
## 3. Apply the configuration ## 3. Apply the configuration — once per node
Boot the target machine from the Talos ISO (see Boot each machine from your custom Talos ISO (see
[docs/01-prerequisites.md](../docs/01-prerequisites.md) for where to get it). [docs/01-prerequisites.md](../docs/01-prerequisites.md)). Each boots into
It boots into "maintenance mode" — alive enough to accept a configuration, "maintenance mode" — alive enough to accept a configuration, with nothing
with nothing installed yet. Confirm it's reachable, then apply: installed yet. For **each** node in turn:
```sh ```sh
talosctl get disks --insecure --nodes <NODE_IP> talosctl get disks --insecure --nodes <NODE_IP>
``` ```
Note the disk name you want Talos installed to (e.g. `/dev/sda`). Then: Confirm you see two disks: the one Talos will install itself to, and the
≥900GB one `talos/patch-longhorn.yaml`'s `UserVolumeConfig` will claim for
Longhorn. Then:
```sh ```sh
talosctl apply-config --insecure \ talosctl apply-config --insecure \
@@ -73,32 +90,37 @@ talosctl apply-config --insecure \
`--insecure` is required for this one command only: the node has no `--insecure` is required for this one command only: the node has no
certificates yet, so talosctl can't authenticate to it normally. This is certificates yet, so talosctl can't authenticate to it normally. This is
the one moment during setup where you're trusting the network — do this on the one moment during setup where you're trusting the network — do this on
a network you control. The node will reboot, install Talos to disk, and a network you control. The node will reboot, install Talos to its system
come back up running the configuration you gave it — now *with* disk, format and mount its second disk for Longhorn, and come back up
certificates, so every command after this uses the secure path. running the configuration you gave it — now *with* certificates, so every
command after this uses the secure path.
## 4. Bootstrap etcd Repeat this for `<NODE1_IP>`, `<NODE2_IP>`, and `<NODE3_IP>` before moving on.
Talos is installed and Kubernetes components are starting, but the cluster ## 4. Bootstrap etcd — once, total
has no data store yet. This one-time command initializes it:
Talos is installed on all three nodes and Kubernetes components are
starting, but the cluster has no data store yet. This one-time command
initializes it:
```sh ```sh
talosctl bootstrap --nodes <NODE_IP> talosctl bootstrap --nodes <NODE1_IP>
``` ```
Run this exactly once per cluster, against exactly one control-plane node, Run this **exactly once**, against **exactly one** of your three nodes —
even if you later add more. Running it again on an existing cluster can even though all three will end up running etcd as peers, only one needs
corrupt it. telling to *start* the cluster; the other two discover and join the
existing one. Running it again against an existing cluster can corrupt it.
## 5. Get your kubeconfig ## 5. Get your kubeconfig
```sh ```sh
talosctl kubeconfig --nodes <NODE_IP> . talosctl kubeconfig --nodes <VIP_IP> .
``` ```
Downloads a kubeconfig into the current directory (also gitignored — Downloads a kubeconfig into the current directory (also gitignored —
it grants full cluster access). Merge it with `~/.kube/config` or point it grants full cluster access), pointed at the VIP rather than any one
`KUBECONFIG` at it directly: node. Merge it with `~/.kube/config` or point `KUBECONFIG` at it directly:
```sh ```sh
export KUBECONFIG=$PWD/kubeconfig export KUBECONFIG=$PWD/kubeconfig
@@ -110,14 +132,16 @@ export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes kubectl get nodes
``` ```
Expect to see one node, `Ready`, within a couple of minutes (it takes a Expect to see **three** nodes, `Ready`, within a couple of minutes (it
moment for the default CNI — Talos ships Flannel out of the box — to come takes a moment for the default CNI — Talos ships Flannel out of the box —
up and clear the node's networking checks). to come up and clear each node's networking checks).
``` ```
NAME STATUS ROLES AGE VERSION NAME STATUS ROLES AGE VERSION
talos-... Ready control-plane 90s v1.3x.x talos-... Ready control-plane 90s v1.3x.x
talos-... Ready control-plane 85s v1.3x.x
talos-... Ready control-plane 80s v1.3x.x
``` ```
If it's stuck on `NotReady`, give it another minute before troubleshooting — If any node is stuck on `NotReady`, give it another minute before
first boot does a lot of one-time work. troubleshooting — first boot does a lot of one-time work.
+36 -8
View File
@@ -1,15 +1,43 @@
# A patch applied on top of the controlplane.yaml that `talosctl gen config` # A patch applied on top of the controlplane.yaml that `talosctl gen config`
# produces (see talos/README.md and docs/02-talos-bootstrap.md for the full # produces (see talos/README.md and docs/02-talos-bootstrap.md for the full
# command). Talos's generated defaults assume a multi-node cluster where # command). Applied identically to all three control-plane nodes.
# control-plane nodes only run the Kubernetes control plane itself (API
# server, scheduler, etcd, etc.) and never your actual workloads. For a
# single-node learning cluster, that would mean nothing can ever be
# scheduled — there's no other node to run it on.
# #
# This one field undoes that restriction, letting the single node be both # --- allowSchedulingOnControlPlanes ---
# the control plane AND a worker. # Talos's generated defaults assume control-plane nodes only run the
# Kubernetes control plane itself (API server, scheduler, etcd, etc.) and
# never your actual workloads — normal practice, so a workload crashing
# never has a chance to take the control plane down with it. This repo's
# three nodes are deliberately both control plane AND worker at once (no
# separate worker nodes), to get real multi-node resilience without needing
# more than three machines. This one field undoes the default restriction.
# #
# >> Ask Claude: "why does Kubernetes normally keep workloads off control # >> Ask Claude: "why does Kubernetes normally keep workloads off control
# plane nodes?" if you want the reasoning before you turn it off. # plane nodes, and what am I giving up by turning that off here?" if you
# want the reasoning before you turn it off.
#
# --- vip ---
# With three control-plane nodes, "the cluster's API server" isn't one
# fixed machine anymore — it's whichever of the three happen to be up.
# Talos's VIP (Virtual IP) feature solves this: the three nodes
# continuously elect one of themselves to answer on this one shared
# address, and re-elect within seconds if that node goes down. Point
# everything that talks to the cluster (talosctl, kubectl, this repo's
# Application manifests) at the VIP, never at one node's own address —
# that's what makes losing any single node a non-event instead of an
# outage.
#
# Replace both the interface name and the IP below with your real values
# before generating config — see docs/02-talos-bootstrap.md.
#
# >> Ask Claude to help you pick the VIP address — like the MetalLB pool in
# docs/05-metallb.md, it has to be a real, unused address on your LAN,
# outside your DHCP range.
cluster: cluster:
allowSchedulingOnControlPlanes: true allowSchedulingOnControlPlanes: true
machine:
network:
interfaces:
- interface: eth0 # EDIT ME — match your node's real NIC name (`talosctl get links --insecure --nodes <NODE_IP>` from maintenance mode will show it)
dhcp: true
vip:
ip: 192.168.1.239 # EDIT ME — replace with your chosen VIP address
+60
View File
@@ -0,0 +1,60 @@
# Prepares each node for Longhorn (apps/longhorn.yaml) — the storage layer
# this repo ships pre-installed. Applied identically to all three
# control-plane nodes, alongside patch-controlplane.yaml.
#
# This file has TWO documents (separated by `---`), because it's making two
# different kinds of change: a patch to the normal machine config, and a
# standalone Talos resource that doesn't live inside `machine:`/`cluster:`
# at all.
#
# --- Document 1: mount the volume into the kubelet's view ---
# Talos runs the kubelet in its own isolated mount namespace. Without this,
# a volume mounted at /var/mnt/longhorn on the node itself would be
# invisible to any Pod using a hostPath-style mount there — which is
# exactly how Longhorn's own manager and engine Pods access node storage.
# This bind-mounts that same path into the kubelet's namespace so Pods can
# actually see it. `rshared` matters specifically: it lets mounts Longhorn
# creates later *underneath* this path (one per volume) propagate back out
# to the host, instead of being invisible outside the container that made
# them.
#
# --- Document 2: UserVolumeConfig ---
# A Talos UserVolumeConfig tells Talos itself to find, format, and mount a
# disk — no manual partitioning on your part. `diskSelector.match:
# '!system_disk'` means "any disk that isn't the one Talos is installed
# on" (never touches your OS disk), and `minSize: 900GiB` means "only
# consider disks at least that large" — so if a node has more than one
# extra disk, Talos won't accidentally pick a small one. Adjust `minSize`
# if your nodes' data disks are smaller than 900GiB; just make sure it's
# still large enough to exclude anything that *isn't* meant for Longhorn.
# `grow: false` keeps the volume at its initial size even if the
# underlying disk is later resized — deliberate here, so growing storage
# is always a conscious decision (see docs/09-growing-up.md).
#
# The `name: longhorn` below is what determines the mount path Talos uses
# — user volumes are always mounted at /var/mnt/<name>, which is why
# Document 1 above and apps/longhorn/values.yaml both hardcode
# /var/mnt/longhorn. Renaming this would mean updating both.
#
# >> Ask Claude: "how do I find out which of my disks Talos will actually
# pick for this?" before you boot for real — worth confirming on your
# specific hardware rather than assuming.
machine:
kubelet:
extraMounts:
- destination: /var/mnt/longhorn
type: bind
source: /var/mnt/longhorn
options:
- bind
- rshared
- rw
---
apiVersion: v1alpha1
kind: UserVolumeConfig
name: longhorn
provisioning:
diskSelector:
match: "!system_disk"
minSize: 900GiB
grow: false