Move to a three-node cluster with a Talos VIP; add Longhorn
Every node is now both control plane and worker, giving real tolerance for losing any single machine (etcd quorum needs 3, not 1). A Talos VIP gives the cluster one stable address regardless of which node answers. Also pre-installs Longhorn for replicated storage, backed by a Talos UserVolume on each node's second disk (>=900GB, non-system) rather than the default path — mirroring the pattern from the homescale repo. Docs renumbered to make room for 06-longhorn.md; checklist, glossary, network map, and CLAUDE.md's guided-bootstrap instructions updated to match, including asking the user for node IPs, VIP, and disk info before generating any Talos config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
92c3ad3191
commit
110bd97cca
@@ -0,0 +1,108 @@
|
||||
# 09 — Growing up: what to add next
|
||||
|
||||
Everything up to here is deliberately minimal — enough to understand the
|
||||
whole loop, nothing more. This doc is a map of the natural next steps, each
|
||||
with why you'd want it and roughly what it involves. None of it is built
|
||||
for you; each is its own project once you're ready.
|
||||
|
||||
## More nodes
|
||||
|
||||
Your three nodes already give you real resilience — losing any one still
|
||||
leaves a majority of control-plane voters (see
|
||||
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker))
|
||||
and, once Longhorn has real volumes, a replica of your data likely
|
||||
survives too. Growing further is about *capacity*, not fault tolerance you
|
||||
don't already have — so beyond this point, add plain **worker** nodes, not
|
||||
more control-plane ones:
|
||||
|
||||
1. Boot a new machine from the same custom Talos ISO as the first three
|
||||
(see [`docs/01-prerequisites.md`](01-prerequisites.md)).
|
||||
2. Apply the **worker** config instead of the control-plane one — you
|
||||
already have `_out/worker.yaml` from
|
||||
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now:
|
||||
```sh
|
||||
talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
|
||||
```
|
||||
3. `kubectl get nodes` should show a fourth node shortly after, without
|
||||
the `control-plane` role.
|
||||
|
||||
### Why not a fourth control-plane node
|
||||
|
||||
It's tempting to reach for a fourth control-plane node when you're growing
|
||||
a cluster you already made resilient with three — but it actually makes
|
||||
things worse, not better. etcd (the control-plane database) needs a
|
||||
**majority** of its members to agree before it can do anything, and
|
||||
majority only means something clean with an odd number: with 3 members, 1
|
||||
can fail and 2 (a majority) still agree — you can lose one. With 4, you
|
||||
still need 3 to agree, so you can *still* only lose one — you paid for an
|
||||
extra machine and gained nothing. Worse, 4 is *more* likely to have
|
||||
exactly a 2-2 split during a network hiccup, which stalls the cluster
|
||||
until it's resolved. If you ever do want more control-plane fault
|
||||
tolerance than three nodes gives you, the next meaningful number is 5 —
|
||||
not 4.
|
||||
|
||||
> **Ask Claude:** "walk me through the quorum math for 3 vs. 4 vs. 5
|
||||
> control-plane nodes" if you want to see this worked out concretely
|
||||
> rather than taking it on faith.
|
||||
|
||||
## Ingress
|
||||
|
||||
Right now, thanks to MetalLB ([`docs/05-metallb.md`](05-metallb.md)), the
|
||||
demo app gets its own dedicated IP address
|
||||
(see [`apps/demo/service.yaml`](../apps/demo/service.yaml)). That's a real
|
||||
improvement over NodePort, but it doesn't scale past a handful of apps
|
||||
either: every app still needs its own IP address out of a pool that's only
|
||||
so big, and there's no way to route by hostname (`app1.example.com` vs
|
||||
`app2.example.com` sharing one IP on port 443).
|
||||
|
||||
An **Ingress controller** is itself just another app you'd deploy the same
|
||||
way as anything else in this repo (see
|
||||
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md)) — commonly as a
|
||||
Helm-based app, since most ingress controllers are distributed that way,
|
||||
same as MetalLB. It typically sits behind one single MetalLB
|
||||
`LoadBalancer` Service (one IP, total, for every app), and you add an
|
||||
`Ingress` resource per app instead of giving each one its own
|
||||
`LoadBalancer` Service — trading "one IP per app" for "one IP, many
|
||||
hostnames."
|
||||
|
||||
## cert-manager
|
||||
|
||||
Once you have Ingress and real hostnames, you'll want HTTPS. `cert-manager`
|
||||
is the standard tool for this — it watches for certificate requests and
|
||||
automatically obtains and renews them (commonly via Let's Encrypt). Also
|
||||
just another app in this repo's structure once you get there.
|
||||
|
||||
## Secrets management
|
||||
|
||||
**This repo is public. Never commit a plaintext secret, credential, token,
|
||||
or private key to it — anyone can read your Git history, forever, even if
|
||||
you delete the file in a later commit.**
|
||||
|
||||
Kubernetes' own `Secret` resource does not encrypt anything — it just
|
||||
base64-encodes data, which is not encryption, only encoding (trivially
|
||||
reversible by anyone). None of the manifests in this repo need secrets yet,
|
||||
which is exactly why this hasn't come up so far. When you add something
|
||||
that does (a database password, an API key), you need one of:
|
||||
|
||||
- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** —
|
||||
encrypts a secret using a key that only your cluster holds, producing a
|
||||
`SealedSecret` file that's safe to commit; the cluster decrypts it into a
|
||||
normal Secret on the fly.
|
||||
- **[SOPS](https://github.com/getsops/sops)** (often paired with Argo CD via
|
||||
a plugin) — encrypts values inside a normal-looking YAML file using a key
|
||||
from a system like age or a cloud KMS, decrypted at sync time.
|
||||
|
||||
Either is a reasonable choice; don't improvise your own approach. Ask
|
||||
Claude to walk you through setting one up when you have a real secret to
|
||||
manage — it's a short, well-trodden path once you're at that point.
|
||||
|
||||
> **Ask Claude:** if you're about to commit anything you're even slightly
|
||||
> unsure about, paste it (or describe it) and ask first. Cheaper than a
|
||||
> `git filter-branch` later.
|
||||
|
||||
## Where to go from here
|
||||
|
||||
You've now seen the whole shape: Talos underneath, Argo CD watching Git,
|
||||
app-of-apps organizing what it manages. Everything in this doc is more of
|
||||
the same pattern — a new app, a new file, a push. Pick whichever expansion
|
||||
is actually useful to you and start there.
|
||||
Reference in New Issue
Block a user