k8s-starter-pack/docs/09-growing-up.md

09 — Growing up: what to add next

Everything up to here is deliberately minimal — enough to understand the whole loop, nothing more. This doc is a map of the natural next steps, each with why you'd want it and roughly what it involves. None of it is built for you; each is its own project once you're ready.

More nodes

Your three nodes already give you real resilience — losing any one still leaves a majority of control-plane voters (see docs/02-talos-bootstrap.md) and, once Longhorn has real volumes, a replica of your data likely survives too. Growing further is about capacity, not fault tolerance you don't already have — so beyond this point, add plain worker nodes, not more control-plane ones:

1. Boot a new machine from the same custom Talos ISO as the first three (see docs/01-prerequisites.md).
2. Apply the worker config instead of the control-plane one — you already have _out/worker.yaml from docs/02-talos-bootstrap.md, unused until now:

   talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
   

3. kubectl get nodes should show a fourth node shortly after, without the control-plane role.

Why not a fourth control-plane node

It's tempting to reach for a fourth control-plane node when you're growing a cluster you already made resilient with three — but it actually makes things worse, not better. etcd (the control-plane database) needs a majority of its members to agree before it can do anything, and majority only means something clean with an odd number: with 3 members, 1 can fail and 2 (a majority) still agree — you can lose one. With 4, you still need 3 to agree, so you can still only lose one — you paid for an extra machine and gained nothing. Worse, 4 is more likely to have exactly a 2-2 split during a network hiccup, which stalls the cluster until it's resolved. If you ever do want more control-plane fault tolerance than three nodes gives you, the next meaningful number is 5 — not 4.

Ask Claude: "walk me through the quorum math for 3 vs. 4 vs. 5 control-plane nodes" if you want to see this worked out concretely rather than taking it on faith.

Ingress

Right now, thanks to MetalLB (docs/05-metallb.md), the demo app gets its own dedicated IP address (see apps/demo/service.yaml). That's a real improvement over NodePort, but it doesn't scale past a handful of apps either: every app still needs its own IP address out of a pool that's only so big, and there's no way to route by hostname (app1.example.com vs app2.example.com sharing one IP on port 443).

An Ingress controller is itself just another app you'd deploy the same way as anything else in this repo (see docs/08-adding-new-apps.md) — commonly as a Helm-based app, since most ingress controllers are distributed that way, same as MetalLB. It typically sits behind one single MetalLB LoadBalancer Service (one IP, total, for every app), and you add an Ingress resource per app instead of giving each one its own LoadBalancer Service — trading "one IP per app" for "one IP, many hostnames."

cert-manager

Once you have Ingress and real hostnames, you'll want HTTPS. cert-manager is the standard tool for this — it watches for certificate requests and automatically obtains and renews them (commonly via Let's Encrypt). Also just another app in this repo's structure once you get there.

Secrets management

This repo is public. Never commit a plaintext secret, credential, token, or private key to it — anyone can read your Git history, forever, even if you delete the file in a later commit.

Kubernetes' own Secret resource does not encrypt anything — it just base64-encodes data, which is not encryption, only encoding (trivially reversible by anyone). None of the manifests in this repo need secrets yet, which is exactly why this hasn't come up so far. When you add something that does (a database password, an API key), you need one of:

• Sealed Secrets — encrypts a secret using a key that only your cluster holds, producing a SealedSecret file that's safe to commit; the cluster decrypts it into a normal Secret on the fly.
• SOPS (often paired with Argo CD via a plugin) — encrypts values inside a normal-looking YAML file using a key from a system like age or a cloud KMS, decrypted at sync time.

Either is a reasonable choice; don't improvise your own approach. Ask Claude to walk you through setting one up when you have a real secret to manage — it's a short, well-trodden path once you're at that point.

Ask Claude: if you're about to commit anything you're even slightly unsure about, paste it (or describe it) and ask first. Cheaper than a git filter-branch later.

Where to go from here

You've now seen the whole shape: Talos underneath, Argo CD watching Git, app-of-apps organizing what it manages. Everything in this doc is more of the same pattern — a new app, a new file, a push. Pick whichever expansion is actually useful to you and start there.