128 lines
6.4 KiB
Markdown
128 lines
6.4 KiB
Markdown
# 09 — Growing up: what to add next
|
|
|
|
Everything up to here is deliberately minimal — enough to understand the
|
|
whole loop, nothing more. This doc is a map of the natural next steps, each
|
|
with why you'd want it and roughly what it involves. None of it is built
|
|
for you; each is its own project once you're ready.
|
|
|
|
## More nodes
|
|
|
|
This cluster already separates roles — three Raspberry Pi control planes
|
|
(`talos-455-i3b`, `talos-9xy-iuk`, `talos-k1b-7j5`) and two Proxmox
|
|
workers (`talos-rss-v5l` on Gimli, `talos-vba-ood` on Titus; see
|
|
[`docs/network.md`](network.md) for current addresses). Losing any one
|
|
control plane still leaves a majority of etcd voters, and losing either
|
|
worker still leaves one healthy Longhorn replica of any volume that had
|
|
data on it. Growing further is about *capacity*, not fault tolerance you
|
|
don't already have — so beyond this point, add another plain **worker**,
|
|
not a fourth control plane.
|
|
|
|
1. Boot the new machine from a Talos image matching its own architecture
|
|
— `rpi_generic` (arm64) for another Raspberry Pi, or the amd64 metal
|
|
image for another x86 machine/VM. Both custom schematics already exist
|
|
for this cluster — see [`talos/patch-controlplane.yaml`](../talos/patch-controlplane.yaml)
|
|
and [`talos/patch-worker.yaml`](../talos/patch-worker.yaml) for the
|
|
exact installer image references, and
|
|
[`docs/01-prerequisites.md`](01-prerequisites.md) for building a new one
|
|
if this is a genuinely new architecture/extension set.
|
|
2. Generate (or reuse, if unchanged) a worker config with both
|
|
`talos/patch-worker.yaml` and `talos/patch-longhorn.yaml` applied, then
|
|
apply it to the new node:
|
|
```sh
|
|
talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
|
|
```
|
|
3. If this node will also provide Longhorn storage, give it its own
|
|
dedicated data disk sized to fit `talos/patch-longhorn.yaml`'s
|
|
`minSize`, and label it the same way
|
|
[`talos/patch-worker.yaml`](../talos/patch-worker.yaml) labels the
|
|
existing two workers — `node.longhorn.io/create-default-disk=true` —
|
|
or Longhorn will never offer it a disk.
|
|
4. `kubectl get nodes` should show the new node shortly after, without the
|
|
`control-plane` role. With a third storage-providing worker, consider
|
|
raising Longhorn's `defaultReplicaCount` and
|
|
`persistence.defaultClassReplicaCount` in
|
|
[`apps/longhorn/values.yaml`](../apps/longhorn/values.yaml) from `2` to
|
|
`3` — see [`docs/06-longhorn.md`](06-longhorn.md) for why both settings
|
|
have to move together.
|
|
|
|
### Why not a fourth control-plane node
|
|
|
|
It's tempting to reach for a fourth control-plane node when you're growing
|
|
a cluster you already made resilient with three — but it actually makes
|
|
things worse, not better. etcd (the control-plane database) needs a
|
|
**majority** of its members to agree before it can do anything, and
|
|
majority only means something clean with an odd number: with 3 members, 1
|
|
can fail and 2 (a majority) still agree — you can lose one. With 4, you
|
|
still need 3 to agree, so you can *still* only lose one — you paid for an
|
|
extra machine and gained nothing. Worse, 4 is *more* likely to have
|
|
exactly a 2-2 split during a network hiccup, which stalls the cluster
|
|
until it's resolved. If you ever do want more control-plane fault
|
|
tolerance than three nodes gives you, the next meaningful number is 5 —
|
|
not 4.
|
|
|
|
> **Ask Claude:** "walk me through the quorum math for 3 vs. 4 vs. 5
|
|
> control-plane nodes" if you want to see this worked out concretely
|
|
> rather than taking it on faith.
|
|
|
|
## Ingress
|
|
|
|
Right now, thanks to MetalLB ([`docs/05-metallb.md`](05-metallb.md)), the
|
|
demo app gets its own dedicated IP address
|
|
(see [`apps/demo/service.yaml`](../apps/demo/service.yaml)). That's a real
|
|
improvement over NodePort, but it doesn't scale past a handful of apps
|
|
either: every app still needs its own IP address out of a pool that's only
|
|
so big, and there's no way to route by hostname (`app1.example.com` vs
|
|
`app2.example.com` sharing one IP on port 443).
|
|
|
|
An **Ingress controller** is itself just another app you'd deploy the same
|
|
way as anything else in this repo (see
|
|
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md)) — commonly as a
|
|
Helm-based app, since most ingress controllers are distributed that way,
|
|
same as MetalLB. It typically sits behind one single MetalLB
|
|
`LoadBalancer` Service (one IP, total, for every app), and you add an
|
|
`Ingress` resource per app instead of giving each one its own
|
|
`LoadBalancer` Service — trading "one IP per app" for "one IP, many
|
|
hostnames."
|
|
|
|
## cert-manager
|
|
|
|
Once you have Ingress and real hostnames, you'll want HTTPS. `cert-manager`
|
|
is the standard tool for this — it watches for certificate requests and
|
|
automatically obtains and renews them (commonly via Let's Encrypt). Also
|
|
just another app in this repo's structure once you get there.
|
|
|
|
## Secrets management
|
|
|
|
**This repo is public. Never commit a plaintext secret, credential, token,
|
|
or private key to it — anyone can read your Git history, forever, even if
|
|
you delete the file in a later commit.**
|
|
|
|
Kubernetes' own `Secret` resource does not encrypt anything — it just
|
|
base64-encodes data, which is not encryption, only encoding (trivially
|
|
reversible by anyone). None of the manifests in this repo need secrets yet,
|
|
which is exactly why this hasn't come up so far. When you add something
|
|
that does (a database password, an API key), you need one of:
|
|
|
|
- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** —
|
|
encrypts a secret using a key that only your cluster holds, producing a
|
|
`SealedSecret` file that's safe to commit; the cluster decrypts it into a
|
|
normal Secret on the fly.
|
|
- **[SOPS](https://github.com/getsops/sops)** (often paired with Argo CD via
|
|
a plugin) — encrypts values inside a normal-looking YAML file using a key
|
|
from a system like age or a cloud KMS, decrypted at sync time.
|
|
|
|
Either is a reasonable choice; don't improvise your own approach. Ask
|
|
Claude to walk you through setting one up when you have a real secret to
|
|
manage — it's a short, well-trodden path once you're at that point.
|
|
|
|
> **Ask Claude:** if you're about to commit anything you're even slightly
|
|
> unsure about, paste it (or describe it) and ask first. Cheaper than a
|
|
> `git filter-branch` later.
|
|
|
|
## Where to go from here
|
|
|
|
You've now seen the whole shape: Talos underneath, Argo CD watching Git,
|
|
app-of-apps organizing what it manages. Everything in this doc is more of
|
|
the same pattern — a new app, a new file, a push. Pick whichever expansion
|
|
is actually useful to you and start there.
|