k8s-starter-pack/docs/06-longhorn.md

# 06 — Longhorn: real, replicated storage

The app-of-apps you applied in [`docs/04-app-of-apps.md`](04-app-of-apps.md)
already brought up `longhorn` alongside `demo` and `metallb`. This doc
explains what it does and how to actually use it — there's no address to
pick this time, unlike MetalLB; the interesting decisions already happened
back in [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).

## Why this is needed at all

A Kubernetes Pod's own filesystem disappears the moment that Pod does —
fine for a stateless web server, useless for anything that needs to keep
data around (a database, uploaded files, anything you'd be upset to lose).
Kubernetes' answer is a **PersistentVolumeClaim (PVC)**: an app asks for
some amount of storage, and something has to actually provide it. On a
cloud provider, that something is the cloud's own disk service. On bare
metal — your three Talos nodes — nothing provides that on its own, the
same gap MetalLB fills for IP addresses rather than storage.

**Longhorn** is that provider: it turns a plain disk on each node into one
pool of storage, and replicates each volume across more than one node. Lose
a node, and the data backing any volume that had a replica elsewhere is
still there. This is why Longhorn needs more than one storage-providing
node at all: with only one, "replicated storage" has nothing to replicate
*to*.

This cluster deliberately keeps storage off the three Raspberry Pi control
planes — they have no data disk and are not meant to run application
workloads (see [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md)).
Longhorn instead runs only on the two dedicated Proxmox workers, each with
its own local NVMe-backed disk on a separate physical host (Gimli and
Titus). `defaultReplicaCount: 2` in
[`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml) matches that:
with exactly two storage nodes, two replicas is the most redundancy
possible — losing either worker still leaves one complete copy.

## What's already running

**`apps/longhorn.yaml`** → one Application, pointing at `apps/longhorn`, a
Helm chart that pulls in the official upstream Longhorn chart (see
[`apps/longhorn/Chart.yaml`](../apps/longhorn/Chart.yaml)) — the same
umbrella-chart pattern as MetalLB, just without any extra templates of its
own this time.

The interesting configuration already happened at the Talos layer, back in
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml): each worker
found, formatted, and mounted its own ≥300GiB disk at `/var/mnt/longhorn`.
[`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml) points Longhorn's
`defaultDataPath` at that exact path — this is the thing that makes Longhorn
use *that* disk instead of carving space out of each node's root filesystem
(small, and on Talos, not meant for this kind of data anyway).
[`talos/patch-worker.yaml`](../talos/patch-worker.yaml) also labels each
worker `node.longhorn.io/create-default-disk=true`, which is what actually
limits default-disk creation to these two nodes instead of every node in
the cluster — without it, Longhorn would try (and fail) to create a
default disk on the Raspberry Pi control planes too.

## Verify it's actually working

```sh
kubectl -n longhorn-system get pods -o wide
```

Expect a `longhorn-manager` and `longhorn-csi-plugin` Pod on each of the
two workers (not the control planes), plus one `instance-manager` per
worker once a volume exists, all `Running`. Confirm Longhorn agrees which
nodes are eligible storage nodes:

```sh
kubectl -n longhorn-system get nodes.longhorn.io -o wide
```

Only the two workers should show `READY=true`, `ALLOWSCHEDULING=true`,
`SCHEDULABLE=true`.

```sh
kubectl get storageclass
```

Longhorn's chart creates a default `StorageClass` called `longhorn`
automatically — this is what makes it the thing Kubernetes reaches for
whenever an app asks for storage without naming one specifically.

## Try it: give the demo app real storage

The demo app's HTML currently lives in a ConfigMap (see
[`docs/07-your-first-change.md`](07-your-first-change.md)) — fine for a
few lines of text, but not how a real app would store real data. As a
hands-on check that Longhorn actually works, ask Claude to help you:

1. Add a small `PersistentVolumeClaim` to `apps/demo/`, requesting a
   couple of gigabytes with no `storageClassName` set (so it uses
   Longhorn's default).
2. Mount it into the demo Deployment somewhere harmless, like
   `/usr/share/nginx/html/data`.
3. Push it, let Argo CD sync, and confirm:
   ```sh
   kubectl -n demo get pvc
   ```
   should show your claim `Bound`, and
   ```sh
   kubectl -n longhorn-system get volumes
   ```
   should show a matching volume, `healthy`, with more than one replica.

This isn't part of the checklist below — treat it as an optional exercise
once the pre-installed pieces are confirmed working, not a requirement to
move on.

> **Ask Claude:** "what would happen to that PVC's data if I lost one of
> my two storage workers right now?" — worth being able to answer before you
> trust this with anything real.

## Checklist

- [ ] `kubectl -n longhorn-system get pods -o wide` shows manager/CSI Pods
      `Running` on both workers, and none on the control planes
- [ ] `kubectl -n longhorn-system get nodes.longhorn.io` shows exactly the
      two workers as schedulable storage nodes
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
      with a healthy, replicated Longhorn volume behind it

Continue to [`docs/07-your-first-change.md`](07-your-first-change.md).