06 — Longhorn: real, replicated storage

The app-of-apps you applied in docs/04-app-of-apps.md already brought up longhorn alongside demo and metallb. This doc explains what it does and how to actually use it — there's no address to pick this time, unlike MetalLB; the interesting decisions already happened back in docs/02-talos-bootstrap.md.

Why this is needed at all

A Kubernetes Pod's own filesystem disappears the moment that Pod does — fine for a stateless web server, useless for anything that needs to keep data around (a database, uploaded files, anything you'd be upset to lose). Kubernetes' answer is a PersistentVolumeClaim (PVC): an app asks for some amount of storage, and something has to actually provide it. On a cloud provider, that something is the cloud's own disk service. On bare metal — your three Talos nodes — nothing provides that on its own, the same gap MetalLB fills for IP addresses rather than storage.

Longhorn is that provider: it turns a plain disk on each node into one pool of storage, and replicates each volume across more than one node. Lose a node, and the data backing any volume that had a replica elsewhere is still there. This is why Longhorn needs more than one storage-providing node at all: with only one, "replicated storage" has nothing to replicate to.

This cluster deliberately keeps storage off the three Raspberry Pi control planes — they have no data disk and are not meant to run application workloads (see docs/02-talos-bootstrap.md). Longhorn instead runs only on the two dedicated Proxmox workers, each with its own local NVMe-backed disk on a separate physical host (Gimli and Titus). defaultReplicaCount: 2 in apps/longhorn/values.yaml matches that: with exactly two storage nodes, two replicas is the most redundancy possible — losing either worker still leaves one complete copy.

What's already running

apps/longhorn.yaml → one Application, pointing at apps/longhorn, a Helm chart that pulls in the official upstream Longhorn chart (see apps/longhorn/Chart.yaml) — the same umbrella-chart pattern as MetalLB, just without any extra templates of its own this time.

The interesting configuration already happened at the Talos layer, back in talos/patch-longhorn.yaml: each worker found, formatted, and mounted its own ≥300GiB disk at /var/mnt/longhorn. apps/longhorn/values.yaml points Longhorn's defaultDataPath at that exact path — this is the thing that makes Longhorn use that disk instead of carving space out of each node's root filesystem (small, and on Talos, not meant for this kind of data anyway). talos/patch-worker.yaml also labels each worker node.longhorn.io/create-default-disk=true, which is what actually limits default-disk creation to these two nodes instead of every node in the cluster — without it, Longhorn would try (and fail) to create a default disk on the Raspberry Pi control planes too.

Verify it's actually working

kubectl -n longhorn-system get pods -o wide

Expect a longhorn-manager and longhorn-csi-plugin Pod on each of the two workers (not the control planes), plus one instance-manager per worker once a volume exists, all Running. Confirm Longhorn agrees which nodes are eligible storage nodes:

kubectl -n longhorn-system get nodes.longhorn.io -o wide

Only the two workers should show READY=true, ALLOWSCHEDULING=true, SCHEDULABLE=true.

kubectl get storageclass

Longhorn's chart creates a default StorageClass called longhorn automatically — this is what makes it the thing Kubernetes reaches for whenever an app asks for storage without naming one specifically.

Try it: give the demo app real storage

The demo app's HTML currently lives in a ConfigMap (see docs/07-your-first-change.md) — fine for a few lines of text, but not how a real app would store real data. As a hands-on check that Longhorn actually works, ask Claude to help you:

Add a small PersistentVolumeClaim to apps/demo/, requesting a couple of gigabytes with no storageClassName set (so it uses Longhorn's default).
Mount it into the demo Deployment somewhere harmless, like /usr/share/nginx/html/data.
Push it, let Argo CD sync, and confirm:
```
kubectl -n demo get pvc
```
should show your claim Bound, and
```
kubectl -n longhorn-system get volumes
```
should show a matching volume, healthy, with more than one replica.

This isn't part of the checklist below — treat it as an optional exercise once the pre-installed pieces are confirmed working, not a requirement to move on.

Ask Claude: "what would happen to that PVC's data if I lost one of my two storage workers right now?" — worth being able to answer before you trust this with anything real.

Checklist

kubectl -n longhorn-system get pods -o wide shows manager/CSI Pods Running on both workers, and none on the control planes
kubectl -n longhorn-system get nodes.longhorn.io shows exactly the two workers as schedulable storage nodes
kubectl get storageclass shows a default longhorn StorageClass
(Optional) Gave the demo app a real PVC and confirmed it's Bound with a healthy, replicated Longhorn volume behind it

Continue to docs/07-your-first-change.md.

5.5 KiB Raw Blame History