Add cert-manager with Cloudflare DNS-01 ClusterIssuers

apps/cert-manager.yaml

@@ -0,0 +1,38 @@
+# cert-manager, installed like MetalLB and Longhorn: an Application pointing
+# at apps/cert-manager, a Helm chart that pulls in the upstream cert-manager
+# chart plus this cluster's ClusterIssuers. Once this syncs, an Ingress
+# anywhere in this repo can request a real, auto-renewing HTTPS certificate
+# just by adding one annotation. See docs/12-cert-manager.md.
+#
+# Deliberately NO resources-finalizer.argocd.argoproj.io here, same
+# reasoning as apps/longhorn.yaml. cert-manager's CRDs (Certificate,
+# ClusterIssuer, Order, Challenge) are cluster-wide — deleting them cascades
+# through Kubernetes' own garbage collector to delete every Certificate
+# custom resource in the cluster, and each Certificate owns its resulting
+# TLS Secret by default. Cascading this Application's deletion could
+# silently delete every HTTPS certificate this cluster has, cluster-wide,
+# not just cert-manager's own Pods. Removing this file from Git should be a
+# rare, deliberate act, confirmed by hand.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cert-manager
+  namespace: argocd
+spec:
+  project: default
+
+  source:
+    repoURL: https://tea.ahmza.com/ahmza/k8s-starter-pack.git
+    targetRevision: main
+    path: apps/cert-manager
+
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cert-manager
+
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

apps/cert-manager/Chart.yaml

@@ -0,0 +1,19 @@
+# An "umbrella" chart, same pattern as apps/metallb/ and apps/longhorn/:
+# declares a dependency on the upstream cert-manager chart. Argo CD detects
+# this as a Helm source because a Chart.yaml exists at this path.
+#
+# It also carries two ClusterIssuer templates and one SopsSecret (see
+# apps/cert-manager/templates/) — cert-manager's own custom resources that
+# tell it how to actually prove domain ownership and request certificates.
+# See docs/12-cert-manager.md.
+apiVersion: v2
+name: cert-manager-app
+description: Installs cert-manager plus this cluster's Let's Encrypt ClusterIssuers using Cloudflare DNS-01
+version: 0.1.0
+dependencies:
+  - name: cert-manager
+    # A version range rather than an exact pin, so this doesn't silently go
+    # stale. Check https://cert-manager.io/docs/releases/ for the current
+    # release and pin an exact version for a fully reproducible install.
+    version: ">=1.16.0"
+    repository: https://charts.jetstack.io

apps/cert-manager/templates/clusterissuer-prod.yaml

@@ -0,0 +1,23 @@
+# Let's Encrypt's real, publicly-trusted production environment. Switch an
+# Ingress to this one only after confirming the same hostname worked
+# against letsencrypt-staging (see clusterissuer-staging.yaml) — production
+# has much stricter rate limits, and a repeated mistake here can lock you
+# out of requesting real certificates for that hostname for a while.
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: certs@ahmza.com
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token

apps/cert-manager/templates/clusterissuer-staging.yaml

@@ -0,0 +1,26 @@
+# Let's Encrypt's staging environment: unlimited (well, generous) request
+# limits, but its certificates are not trusted by browsers. Use this one
+# first to prove the whole DNS-01/Cloudflare flow actually works, without
+# risking Let's Encrypt's much stricter production rate limits on a typo.
+#
+# sync-wave "1" defers this until cert-manager's own chart resources
+# (wave "0" by default, including its CRDs) are synced and healthy — same
+# reasoning as apps/metallb/templates/*.
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  acme:
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    email: certs@ahmza.com
+    privateKeySecretRef:
+      name: letsencrypt-staging-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token

apps/cert-manager/values.yaml

@@ -0,0 +1,9 @@
+# Values for the upstream cert-manager chart, nested under `cert-manager:`
+# (the dependency's name in Chart.yaml).
+cert-manager:
+  crds:
+    # cert-manager's CRDs (Certificate, ClusterIssuer, Order, Challenge...)
+    # installed as part of this same Helm release, rather than as a
+    # separate manual step. `installCRDs` is the older, deprecated name for
+    # this same setting.
+    enabled: true

apps/demo/ingress.yaml

@@ -7,14 +7,28 @@
 # yourself" convenience Longhorn gives PVCs that don't name a
 # storageClassName.
 #
-# No TLS yet — see docs/12-cert-manager.md for adding a real HTTPS
-# certificate for this exact hostname.
+# cert-manager.io/cluster-issuer triggers cert-manager's "ingress-shim": it
+# notices this annotation, creates a Certificate resource for the
+# hostname(s) in tls.hosts below, and keeps it renewed automatically
+# forever. Deliberately pointed at letsencrypt-staging first — see
+# docs/12-cert-manager.md for why, and switch to letsencrypt-prod only
+# after confirming staging actually works for this hostname.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: demo
   namespace: demo
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-staging
 spec:
+  tls:
+    - hosts:
+        - demo.ahmza.com
+      # cert-manager creates and keeps this Secret updated automatically —
+      # nothing to create by hand, and nothing to encrypt with sops; a TLS
+      # certificate isn't a credential you chose, it's issued fresh by
+      # Let's Encrypt and safe to regenerate at any time.
+      secretName: demo-tls
   rules:
     - host: demo.ahmza.com
       http:

docs/12-cert-manager.md

@@ -0,0 +1,124 @@
+# 12 — cert-manager: real, auto-renewing HTTPS
+
+`demo.ahmza.com` works over plain HTTP (`docs/11-ingress.md`). This doc adds
+a real, browser-trusted HTTPS certificate for it — issued automatically,
+renewed automatically, never touched by hand again once it's working.
+
+## The pieces
+
+- **cert-manager** — watches for `Certificate` resources (or, more simply,
+  one annotation on an `Ingress`), requests a certificate from an ACME
+  provider, and keeps it renewed. Installed the same umbrella-chart way as
+  MetalLB and Longhorn — see
+  [`apps/cert-manager/Chart.yaml`](../apps/cert-manager/Chart.yaml).
+- **Let's Encrypt** — the free, automated certificate authority
+  cert-manager talks to.
+- **DNS-01 challenge** — proves you control `ahmza.com` by creating a
+  temporary DNS TXT record, via Cloudflare's API, rather than requiring
+  Let's Encrypt to reach your server over the public internet. Chosen
+  specifically because this cluster is currently LAN-only (see
+  `docs/09-growing-up.md`) but will eventually be public — DNS-01 works
+  identically either way, and also supports wildcard certificates
+  (`*.ahmza.com`), unlike HTTP-01.
+- **A Cloudflare API token** — the one real secret this needs, stored
+  through the SOPS+age pipeline from
+  [`docs/10-secrets.md`](10-secrets.md), never in plaintext.
+
+## What's already running
+
+**`apps/cert-manager.yaml`** installs the chart plus two
+[`ClusterIssuer`](../apps/cert-manager/templates/)s — a cluster-wide
+resource telling cert-manager *how* to prove ownership and *which* ACME
+server to use:
+
+- `letsencrypt-staging` — Let's Encrypt's staging environment. Generous
+  rate limits, but its certificates aren't trusted by browsers. Use this
+  first, always, for a new hostname.
+- `letsencrypt-prod` — the real thing. Much stricter rate limits; a
+  repeated mistake here can temporarily lock you out of requesting real
+  certificates for that hostname.
+
+Both reference the same Secret, `cloudflare-api-token`, which doesn't
+exist as plaintext anywhere in this repo — it's a `SopsSecret`
+(`apps/cert-manager/templates/cloudflare-api-token.sops.yaml`), decrypted
+into a real Secret by `sops-secrets-operator`.
+
+## One-time setup: the Cloudflare API token
+
+1. In Cloudflare: **My Profile → API Tokens → Create Token → "Edit zone
+   DNS"** template, scoped only to the zone(s) you actually need (e.g.
+   `ahmza.com`) — not a broader token than necessary, and never the
+   account's legacy Global API Key.
+2. On your machine — never in this chat, never in a file you might
+   `git add` before encrypting:
+   ```sh
+   cat > apps/cert-manager/templates/cloudflare-api-token.sops.yaml <<'EOF'
+   apiVersion: isindir.github.com/v1alpha3
+   kind: SopsSecret
+   metadata:
+     name: cloudflare-api-token
+     namespace: cert-manager
+   spec:
+     secretTemplates:
+       - name: cloudflare-api-token
+         stringData:
+           api-token: PASTE_THE_REAL_TOKEN_HERE
+   EOF
+   ```
+3. Edit that file, replacing `PASTE_THE_REAL_TOKEN_HERE` with the real
+   token — still plaintext on disk at this point, still not staged.
+4. Encrypt it in place, immediately:
+   ```sh
+   sops --encrypt --in-place apps/cert-manager/templates/cloudflare-api-token.sops.yaml
+   ```
+5. Confirm `apiVersion`, `kind`, and `metadata` are still readable and only
+   `stringData.api-token`'s value is ciphertext (see
+   [`docs/10-secrets.md`](10-secrets.md) for what this should look like),
+   *then* `git add`, commit, and push.
+
+Note this file must live under `apps/cert-manager/templates/`, not the
+chart root — `apps/cert-manager/` has a `Chart.yaml`, so Argo CD treats it
+as a Helm source and only applies files inside `templates/` (the exact
+mistake `docs/10-secrets.md`'s own pipeline test caught).
+
+## Requesting the certificate
+
+`apps/demo/ingress.yaml` already carries the
+`cert-manager.io/cluster-issuer: letsencrypt-staging` annotation and a
+`tls:` block. Once the token above is in place and synced, cert-manager
+notices, creates a `Certificate`, and works through a DNS-01 challenge
+against Cloudflare automatically.
+
+```sh
+kubectl -n demo get certificate
+kubectl -n demo describe certificate demo-tls
+```
+
+`READY` should become `True` within a minute or two. If it doesn't:
+
+```sh
+kubectl -n cert-manager get challenges -A
+kubectl -n cert-manager logs deploy/cert-manager -f
+```
+
+Once staging genuinely works — confirm with `curl -k` (accepting the
+untrusted staging cert just to prove the handshake completes) — switch
+`apps/demo/ingress.yaml`'s annotation to `letsencrypt-prod`, push, and
+resync. This second request against the *real* Let's Encrypt server is
+what your browser will actually trust.
+
+## Checklist
+
+- [ ] Cloudflare API token created, scoped to only the zone(s) needed
+- [ ] `cloudflare-api-token.sops.yaml` encrypted correctly (metadata
+      readable, `stringData` values ciphertext) and committed
+- [ ] `cert-manager` Application `Synced`/`Healthy`
+- [ ] `kubectl -n cert-manager get clusterissuer` shows both issuers `READY`
+- [ ] Certificate issued successfully against `letsencrypt-staging` first
+- [ ] Switched to `letsencrypt-prod`, certificate reissued and trusted
+- [ ] `https://demo.ahmza.com` loads with a real, browser-trusted padlock,
+      no `-k`/`--insecure` needed
+
+You've now built every piece `docs/09-growing-up.md` described as "more
+of the same pattern" — a new app, a new file, a push. This is the whole
+shape of the repo, end to end.