125 lines
5.5 KiB
Markdown
125 lines
5.5 KiB
Markdown
# 06 — Longhorn: real, replicated storage
|
|
|
|
The app-of-apps you applied in [`docs/04-app-of-apps.md`](04-app-of-apps.md)
|
|
already brought up `longhorn` alongside `demo` and `metallb`. This doc
|
|
explains what it does and how to actually use it — there's no address to
|
|
pick this time, unlike MetalLB; the interesting decisions already happened
|
|
back in [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md).
|
|
|
|
## Why this is needed at all
|
|
|
|
A Kubernetes Pod's own filesystem disappears the moment that Pod does —
|
|
fine for a stateless web server, useless for anything that needs to keep
|
|
data around (a database, uploaded files, anything you'd be upset to lose).
|
|
Kubernetes' answer is a **PersistentVolumeClaim (PVC)**: an app asks for
|
|
some amount of storage, and something has to actually provide it. On a
|
|
cloud provider, that something is the cloud's own disk service. On bare
|
|
metal — your three Talos nodes — nothing provides that on its own, the
|
|
same gap MetalLB fills for IP addresses rather than storage.
|
|
|
|
**Longhorn** is that provider: it turns a plain disk on each node into one
|
|
pool of storage, and replicates each volume across more than one node. Lose
|
|
a node, and the data backing any volume that had a replica elsewhere is
|
|
still there. This is why Longhorn needs more than one storage-providing
|
|
node at all: with only one, "replicated storage" has nothing to replicate
|
|
*to*.
|
|
|
|
This cluster deliberately keeps storage off the three Raspberry Pi control
|
|
planes — they have no data disk and are not meant to run application
|
|
workloads (see [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md)).
|
|
Longhorn instead runs only on the two dedicated Proxmox workers, each with
|
|
its own local NVMe-backed disk on a separate physical host (Gimli and
|
|
Titus). `defaultReplicaCount: 2` in
|
|
[`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml) matches that:
|
|
with exactly two storage nodes, two replicas is the most redundancy
|
|
possible — losing either worker still leaves one complete copy.
|
|
|
|
## What's already running
|
|
|
|
**`apps/longhorn.yaml`** → one Application, pointing at `apps/longhorn`, a
|
|
Helm chart that pulls in the official upstream Longhorn chart (see
|
|
[`apps/longhorn/Chart.yaml`](../apps/longhorn/Chart.yaml)) — the same
|
|
umbrella-chart pattern as MetalLB, just without any extra templates of its
|
|
own this time.
|
|
|
|
The interesting configuration already happened at the Talos layer, back in
|
|
[`talos/patch-longhorn.yaml`](../talos/patch-longhorn.yaml): each worker
|
|
found, formatted, and mounted its own ≥300GiB disk at `/var/mnt/longhorn`.
|
|
[`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml) points Longhorn's
|
|
`defaultDataPath` at that exact path — this is the thing that makes Longhorn
|
|
use *that* disk instead of carving space out of each node's root filesystem
|
|
(small, and on Talos, not meant for this kind of data anyway).
|
|
[`talos/patch-worker.yaml`](../talos/patch-worker.yaml) also labels each
|
|
worker `node.longhorn.io/create-default-disk=true`, which is what actually
|
|
limits default-disk creation to these two nodes instead of every node in
|
|
the cluster — without it, Longhorn would try (and fail) to create a
|
|
default disk on the Raspberry Pi control planes too.
|
|
|
|
## Verify it's actually working
|
|
|
|
```sh
|
|
kubectl -n longhorn-system get pods -o wide
|
|
```
|
|
|
|
Expect a `longhorn-manager` and `longhorn-csi-plugin` Pod on each of the
|
|
two workers (not the control planes), plus one `instance-manager` per
|
|
worker once a volume exists, all `Running`. Confirm Longhorn agrees which
|
|
nodes are eligible storage nodes:
|
|
|
|
```sh
|
|
kubectl -n longhorn-system get nodes.longhorn.io -o wide
|
|
```
|
|
|
|
Only the two workers should show `READY=true`, `ALLOWSCHEDULING=true`,
|
|
`SCHEDULABLE=true`.
|
|
|
|
```sh
|
|
kubectl get storageclass
|
|
```
|
|
|
|
Longhorn's chart creates a default `StorageClass` called `longhorn`
|
|
automatically — this is what makes it the thing Kubernetes reaches for
|
|
whenever an app asks for storage without naming one specifically.
|
|
|
|
## Try it: give the demo app real storage
|
|
|
|
The demo app's HTML currently lives in a ConfigMap (see
|
|
[`docs/07-your-first-change.md`](07-your-first-change.md)) — fine for a
|
|
few lines of text, but not how a real app would store real data. As a
|
|
hands-on check that Longhorn actually works, ask Claude to help you:
|
|
|
|
1. Add a small `PersistentVolumeClaim` to `apps/demo/`, requesting a
|
|
couple of gigabytes with no `storageClassName` set (so it uses
|
|
Longhorn's default).
|
|
2. Mount it into the demo Deployment somewhere harmless, like
|
|
`/usr/share/nginx/html/data`.
|
|
3. Push it, let Argo CD sync, and confirm:
|
|
```sh
|
|
kubectl -n demo get pvc
|
|
```
|
|
should show your claim `Bound`, and
|
|
```sh
|
|
kubectl -n longhorn-system get volumes
|
|
```
|
|
should show a matching volume, `healthy`, with more than one replica.
|
|
|
|
This isn't part of the checklist below — treat it as an optional exercise
|
|
once the pre-installed pieces are confirmed working, not a requirement to
|
|
move on.
|
|
|
|
> **Ask Claude:** "what would happen to that PVC's data if I lost one of
|
|
> my two storage workers right now?" — worth being able to answer before you
|
|
> trust this with anything real.
|
|
|
|
## Checklist
|
|
|
|
- [ ] `kubectl -n longhorn-system get pods -o wide` shows manager/CSI Pods
|
|
`Running` on both workers, and none on the control planes
|
|
- [ ] `kubectl -n longhorn-system get nodes.longhorn.io` shows exactly the
|
|
two workers as schedulable storage nodes
|
|
- [ ] `kubectl get storageclass` shows a default `longhorn` StorageClass
|
|
- [ ] (Optional) Gave the demo app a real PVC and confirmed it's `Bound`
|
|
with a healthy, replicated Longhorn volume behind it
|
|
|
|
Continue to [`docs/07-your-first-change.md`](07-your-first-change.md).
|