k8s-starter-pack/docs/09-growing-up.md

# 09 — Growing up: what to add next

Everything up to here is deliberately minimal — enough to understand the
whole loop, nothing more. This doc is a map of the natural next steps, each
with why you'd want it and roughly what it involves. None of it is built
for you; each is its own project once you're ready.

## More nodes

Your three nodes already give you real resilience — losing any one still
leaves a majority of control-plane voters (see
[`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md#three-nodes-each-both-control-plane-and-worker))
and, once Longhorn has real volumes, a replica of your data likely
survives too. Growing further is about *capacity*, not fault tolerance you
don't already have — so beyond this point, add plain **worker** nodes, not
more control-plane ones:

1. Boot a new machine from the same custom Talos ISO as the first three
   (see [`docs/01-prerequisites.md`](01-prerequisites.md)).
2. Apply the **worker** config instead of the control-plane one — you
   already have `_out/worker.yaml` from
   [`docs/02-talos-bootstrap.md`](02-talos-bootstrap.md), unused until now:
   ```sh
   talosctl apply-config --insecure --nodes <NEW_NODE_IP> --file _out/worker.yaml
   ```
3. `kubectl get nodes` should show a fourth node shortly after, without
   the `control-plane` role.

### Why not a fourth control-plane node

It's tempting to reach for a fourth control-plane node when you're growing
a cluster you already made resilient with three — but it actually makes
things worse, not better. etcd (the control-plane database) needs a
**majority** of its members to agree before it can do anything, and
majority only means something clean with an odd number: with 3 members, 1
can fail and 2 (a majority) still agree — you can lose one. With 4, you
still need 3 to agree, so you can *still* only lose one — you paid for an
extra machine and gained nothing. Worse, 4 is *more* likely to have
exactly a 2-2 split during a network hiccup, which stalls the cluster
until it's resolved. If you ever do want more control-plane fault
tolerance than three nodes gives you, the next meaningful number is 5 —
not 4.

> **Ask Claude:** "walk me through the quorum math for 3 vs. 4 vs. 5
> control-plane nodes" if you want to see this worked out concretely
> rather than taking it on faith.

## Ingress

Right now, thanks to MetalLB ([`docs/05-metallb.md`](05-metallb.md)), the
demo app gets its own dedicated IP address
(see [`apps/demo/service.yaml`](../apps/demo/service.yaml)). That's a real
improvement over NodePort, but it doesn't scale past a handful of apps
either: every app still needs its own IP address out of a pool that's only
so big, and there's no way to route by hostname (`app1.example.com` vs
`app2.example.com` sharing one IP on port 443).

An **Ingress controller** is itself just another app you'd deploy the same
way as anything else in this repo (see
[`docs/08-adding-new-apps.md`](08-adding-new-apps.md)) — commonly as a
Helm-based app, since most ingress controllers are distributed that way,
same as MetalLB. It typically sits behind one single MetalLB
`LoadBalancer` Service (one IP, total, for every app), and you add an
`Ingress` resource per app instead of giving each one its own
`LoadBalancer` Service — trading "one IP per app" for "one IP, many
hostnames."

## cert-manager

Once you have Ingress and real hostnames, you'll want HTTPS. `cert-manager`
is the standard tool for this — it watches for certificate requests and
automatically obtains and renews them (commonly via Let's Encrypt). Also
just another app in this repo's structure once you get there.

## Secrets management

**This repo is public. Never commit a plaintext secret, credential, token,
or private key to it — anyone can read your Git history, forever, even if
you delete the file in a later commit.**

Kubernetes' own `Secret` resource does not encrypt anything — it just
base64-encodes data, which is not encryption, only encoding (trivially
reversible by anyone). None of the manifests in this repo need secrets yet,
which is exactly why this hasn't come up so far. When you add something
that does (a database password, an API key), you need one of:

- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** —
  encrypts a secret using a key that only your cluster holds, producing a
  `SealedSecret` file that's safe to commit; the cluster decrypts it into a
  normal Secret on the fly.
- **[SOPS](https://github.com/getsops/sops)** (often paired with Argo CD via
  a plugin) — encrypts values inside a normal-looking YAML file using a key
  from a system like age or a cloud KMS, decrypted at sync time.

Either is a reasonable choice; don't improvise your own approach. Ask
Claude to walk you through setting one up when you have a real secret to
manage — it's a short, well-trodden path once you're at that point.

> **Ask Claude:** if you're about to commit anything you're even slightly
> unsure about, paste it (or describe it) and ask first. Cheaper than a
> `git filter-branch` later.

## Where to go from here

You've now seen the whole shape: Talos underneath, Argo CD watching Git,
app-of-apps organizing what it manages. Everything in this doc is more of
the same pattern — a new app, a new file, a push. Pick whichever expansion
is actually useful to you and start there.