k8s-starter-pack/docs/13-blog-gitops.md

13 — Deploy the Hugo blog through Argo CD

The blog source lives in ahmza/blog-ahmza-com. This GitOps repository owns desired state: the Kubernetes resources and exact image that should run. Argo CD compares those files with the cluster and applies changes. CI no longer needs a kubeconfig, Kubernetes credentials, or a per-job NetBird connection. The runner still needs working HTTPS access to Gitea and its container registry.

What owns what

blog source push to main
  -> Gitea Actions builds amd64 + arm64 images
  -> registry returns the multi-architecture image digest
  -> CI commits that digest to this repository
  -> app-of-apps discovers apps/blog-ahmza-com.yaml
  -> Argo CD applies apps/blog-ahmza-com/ to the cluster

A digest (sha256:...) identifies exact image contents, unlike the mutable latest tag. Both architecture variants are referenced by the same index digest, so the Pi and x86 workers select their matching image automatically.

The new files are:

• apps/blog-ahmza-com.yaml: the Argo CD Application, meaning which Git path to watch and where to apply it.
• apps/blog-ahmza-com/namespace.yaml: the namespace, an isolation boundary for this application's resources.
• apps/blog-ahmza-com/deployment.yaml: two Pods (running container instances), with zero-unavailable rolling updates and the SWS /health probes.
• apps/blog-ahmza-com/service.yaml: a ClusterIP Service, a stable internal destination for those replaceable Pods.
• apps/blog-ahmza-com/ingress.yaml: routes HTTPS blog.ahmza.com to that Service through the already-running ingress-nginx at 10.10.60.12.

There is no PVC or extra Caddy Deployment. The image includes the full generated site, and TLS is handled by the existing ingress/cert-manager infrastructure. The Application starts with automatic pruning disabled and no cascading-delete finalizer. Review deletion behavior before changing either: removing the Application initially leaves its workloads running rather than deleting them.

Prepare credentials before enabling the new workflow

In the blog source repository, configure these Gitea Actions secrets:

Secret	Purpose
GITEATOKEN	Existing credential that can push ahmza/blog-ahmza-com registry packages
GITOPS_TOKEN	Separate bot PAT permitted to read/write ahmza.com/k8s-starter-pack

The bot must be allowed to push to the GitOps repository's main branch. Scope its token and membership as narrowly as Gitea supports; it does not need Kubernetes credentials or administrator access. Do not commit either token.

At preparation time, the published blog image was readable with an anonymous registry bearer-token exchange. Therefore no image pull secret is shipped in these manifests. If package visibility is changed to private, manage a read-only registry credential in this namespace through the existing SOPS mechanism (docs/10-secrets.md) and reference it using imagePullSecrets. The source repository and registry package have separate visibility settings.

First deployment: GitOps first, workflow second

1. Review and commit the Application, manifests, and documentation in this repository, then push to main. They must exist remotely before the new CI workflow can update the Deployment file. Nothing local is visible to Argo CD.

2. The existing app-of-apps watches apps/ non-recursively. It should create the child Application without another manual kubectl apply. Initial deployment uses the already-published image index pinned in the manifest. Check:

   kubectl -n argocd get application blog-ahmza-com
   kubectl -n blog-ahmza-com get deployment,pods,service,ingress,certificate
   kubectl -n blog-ahmza-com rollout status deployment/blog --timeout=5m
   

   These commands are diagnostics, not the deployment mechanism. If the child Application does not appear, inspect app-of-apps sync status and conditions. Do not solve this by letting CI apply resources directly again.

3. Verify the hostname without changing public routing first:

   curl --fail --show-error --resolve blog.ahmza.com:443:10.10.60.12 \
     https://blog.ahmza.com/
   

   TLS verification remains enabled. If certificate issuance is still pending, inspect certificate/blog-tls and cert-manager events; do not bypass TLS. Reuse of the ready production DNS-01 issuer follows the other cluster sites.

4. Update external routing only after the new site is healthy. For the existing NetBird -> ingress1 Caddy -> Kubernetes topology, add blog.ahmza.com to the Caddy matcher forwarding to 10.10.60.12:443, preserving the hostname for TLS SNI and HTTP Host. Check that the public NetBird service sends that hostname to Caddy. Set the LAN's Technitium override to 10.10.60.12 if LAN clients should use the internal ingress directly. These settings are outside this repository and have not been changed by adding an Ingress manifest.

   Keep public TLS termination, compression and security headers owned by the intended proxy layer; SWS deliberately has compression/security headers off. Cloudflare Full/Full (strict), if used, requires HTTPS to Cloudflare's actual origin. An encrypted NetBird hop later in the path is not a substitute for that TLS connection.

5. With GITOPS_TOKEN configured, commit/push the updated workflow and helper files in the blog source repository. That push builds the site and updates only apps/blog-ahmza-com/deployment.yaml here to the new digest.

6. Check both the Gitea job and Argo CD. A successful CI job means the image and desired-state commit were published, NOT that Kubernetes is already healthy. Argo CD applies the commit asynchronously; inspect its sync/health status and the Deployment's rollout before declaring the release successful.

7. After a successful cutover, remove obsolete blog repository secrets KUBE_CONFIG, NETBIRD_SETUP_KEY, and NETBIRD_URL if no other workflow uses them. If the kubeconfig had broad cluster access, revoke/rotate its credential separately; deleting a CI secret does not revoke the underlying credential. Remove no account-wide NetBird policies/keys without checking other users.

Concurrent publishing and rollback

The blog workflow serializes its publishes. Other applications can still push to this GitOps repository, so it retries non-fast-forward pushes with a rebase. Conflicting changes fail visibly instead of force-pushing over someone else's work. The digest updater refuses missing/duplicate image lines or a different image repository.

To roll back, revert the GitOps commit changing the blog's image digest, then push that revert. Keep the old digest in the registry so it remains pullable. Do not use kubectl set image: Argo CD's self-healing would put the Git version back. A future source build will deliberately publish another desired-state commit.

Checkpoint: if an image build succeeds but the GitOps push fails, should Argo deploy that image? No: the desired-state commit did not land, so it should keep the previous image. This separation is why CI no longer needs cluster access.