k8s-starter-pack/docs/12-cert-manager.md

# 12 — cert-manager: real, auto-renewing HTTPS

`demo.ahmza.com` works over plain HTTP (`docs/11-ingress.md`). This doc adds
a real, browser-trusted HTTPS certificate for it — issued automatically,
renewed automatically, never touched by hand again once it's working.

## The pieces

- **cert-manager** — watches for `Certificate` resources (or, more simply,
  one annotation on an `Ingress`), requests a certificate from an ACME
  provider, and keeps it renewed. Installed the same umbrella-chart way as
  MetalLB and Longhorn — see
  [`apps/cert-manager/Chart.yaml`](../apps/cert-manager/Chart.yaml).
- **Let's Encrypt** — the free, automated certificate authority
  cert-manager talks to.
- **DNS-01 challenge** — proves you control `ahmza.com` by creating a
  temporary DNS TXT record, via Cloudflare's API, rather than requiring
  Let's Encrypt to reach your server over the public internet. Chosen
  specifically because this cluster is currently LAN-only (see
  `docs/09-growing-up.md`) but will eventually be public — DNS-01 works
  identically either way, and also supports wildcard certificates
  (`*.ahmza.com`), unlike HTTP-01.
- **A Cloudflare API token** — the one real secret this needs, stored
  through the SOPS+age pipeline from
  [`docs/10-secrets.md`](10-secrets.md), never in plaintext.

## What's already running

**`apps/cert-manager.yaml`** installs the chart plus two
[`ClusterIssuer`](../apps/cert-manager/templates/)s — a cluster-wide
resource telling cert-manager *how* to prove ownership and *which* ACME
server to use:

- `letsencrypt-staging` — Let's Encrypt's staging environment. Generous
  rate limits, but its certificates aren't trusted by browsers. Use this
  first, always, for a new hostname.
- `letsencrypt-prod` — the real thing. Much stricter rate limits; a
  repeated mistake here can temporarily lock you out of requesting real
  certificates for that hostname.

Both reference the same Secret, `cloudflare-api-token`, which doesn't
exist as plaintext anywhere in this repo — it's a `SopsSecret`
(`apps/cert-manager/templates/cloudflare-api-token.sops.yaml`), decrypted
into a real Secret by `sops-secrets-operator`.

## One-time setup: the Cloudflare API token

1. In Cloudflare: **My Profile → API Tokens → Create Token → "Edit zone
   DNS"** template, scoped only to the zone(s) you actually need (e.g.
   `ahmza.com`) — not a broader token than necessary, and never the
   account's legacy Global API Key.
2. On your machine — never in this chat, never in a file you might
   `git add` before encrypting:
   ```sh
   cat > apps/cert-manager/templates/cloudflare-api-token.sops.yaml <<'EOF'
   apiVersion: isindir.github.com/v1alpha3
   kind: SopsSecret
   metadata:
     name: cloudflare-api-token
     namespace: cert-manager
   spec:
     secretTemplates:
       - name: cloudflare-api-token
         stringData:
           api-token: PASTE_THE_REAL_TOKEN_HERE
   EOF
   ```
3. Edit that file, replacing `PASTE_THE_REAL_TOKEN_HERE` with the real
   token — still plaintext on disk at this point, still not staged.
4. Encrypt it in place, immediately:
   ```sh
   sops --encrypt --in-place apps/cert-manager/templates/cloudflare-api-token.sops.yaml
   ```
5. Confirm `apiVersion`, `kind`, and `metadata` are still readable and only
   `stringData.api-token`'s value is ciphertext (see
   [`docs/10-secrets.md`](10-secrets.md) for what this should look like),
   *then* `git add`, commit, and push.

Note this file must live under `apps/cert-manager/templates/`, not the
chart root — `apps/cert-manager/` has a `Chart.yaml`, so Argo CD treats it
as a Helm source and only applies files inside `templates/` (the exact
mistake `docs/10-secrets.md`'s own pipeline test caught).

## Requesting the certificate

`apps/demo/ingress.yaml` already carries the
`cert-manager.io/cluster-issuer: letsencrypt-staging` annotation and a
`tls:` block. Once the token above is in place and synced, cert-manager
notices, creates a `Certificate`, and works through a DNS-01 challenge
against Cloudflare automatically.

```sh
kubectl -n demo get certificate
kubectl -n demo describe certificate demo-tls
```

`READY` should become `True` within a minute or two. If it doesn't:

```sh
kubectl -n cert-manager get challenges -A
kubectl -n cert-manager logs deploy/cert-manager -f
```

Once staging genuinely works — confirm with `curl -k` (accepting the
untrusted staging cert just to prove the handshake completes) — switch
`apps/demo/ingress.yaml`'s annotation to `letsencrypt-prod`, push, and
resync. This second request against the *real* Let's Encrypt server is
what your browser will actually trust.

## Checklist

- [ ] Cloudflare API token created, scoped to only the zone(s) needed
- [ ] `cloudflare-api-token.sops.yaml` encrypted correctly (metadata
      readable, `stringData` values ciphertext) and committed
- [ ] `cert-manager` Application `Synced`/`Healthy`
- [ ] `kubectl -n cert-manager get clusterissuer` shows both issuers `READY`
- [ ] Certificate issued successfully against `letsencrypt-staging` first
- [ ] Switched to `letsencrypt-prod`, certificate reissued and trusted
- [ ] `https://demo.ahmza.com` loads with a real, browser-trusted padlock,
      no `-k`/`--insecure` needed

You've now built every piece `docs/09-growing-up.md` described as "more
of the same pattern" — a new app, a new file, a push. This is the whole
shape of the repo, end to end.