Devin Flake


Site Reliability Engineer @Adobe
GNU/Linux + Open Source
home
Share: 

Dramble: GitOps goes live

My last post ended with a promise: bootstrap Sealed Secrets, install Argo CD, point it at master, and stop running kubectl apply by hand. All of that happened — plus one thing I said I’d do that I ended up deleting instead.

Argo logo

Argo CD, phase 1

Argo CD 3.6.1 (chart 10.18.0) now runs in its own namespace, installed from gitops/bootstrap/ — the one step that has to stay imperative, because Argo CD cannot install itself.

Phase 2 followed the same day: mysite-nginx, jekyll-website, dashboards and dns are now Applications too, all five with automated sync, self-heal and prune.

One password, five surfaces

I got tired of five different credentials, so everything now shares one:

Surface Gate
Grafana its own login
Prometheus shared basic-auth
Longhorn shared basic-auth
Traefik dashboard shared basic-auth
Argo CD bcrypt in the chart values

Verified end to end rather than assumed: all five answer 200 with the one credential, and a wrong password still gets 401.

The Grafana part was the annoying one — it’s a distroless image with no shell, and its admin password lives in SQLite, so changing the environment variable does nothing. It needed grafana cli admin reset-admin-password. Then its ReadWriteOnce PVC deadlocked the rollout: the surge pod could not attach a volume the old pod still held, and the old pod would not die until the new one was ready. Same bug Traefik’s acme.json had. Fixed with Recreate.

Sealed Secrets

Three credentials — authsecret, grafana, dyn-dns-credentials — are now committed to a public repository as SealedSecrets. The controller runs in kube-system from a vendored v0.41.2 release manifest (the Helm chart’s repository URL returns 404 now).

Two things worth recording, because both would have bitten me later:

  1. A rebuild mints a new key, which cannot unseal anything already in git. The install script now restores the key backup before starting the controller, while there is still no competing key.
  2. The controller refuses to take over a Secret it does not own. It fails, retries for about fifteen seconds, then gives up — and deleting the Secret afterwards does not wake it up. The working sequence is: delete the SealedSecret, delete the Secret, re-apply the SealedSecret.

And the gap during migration matters more than it looks. Deleting authsecret did not merely disable basic-auth — Traefik could not resolve the middleware’s Secret, so it dropped the router entirely. Longhorn, Prometheus and the Traefik dashboard returned 404, not 401, while Grafana and the blog kept working, which made it look like an isolated problem when it wasn’t.

CI: a bad manifest cannot merge

GitHub Actions logo

Four independent jobs on every pull request:

Chart versions are read out of the files that install them, so CI cannot drift from the install path.

It earned its keep immediately. The first run found three leaks — MySQL passwords from Jeff Geerling’s 2018 Drupal demo in the inherited upstream history, in a file deleted years ago. Allowlisted by commit rather than by path, so a future file with that name carrying a real credential is still caught.

And it failed on my own PR: actionlint reported a shellcheck warning I could not reproduce locally, because actionlint only runs shellcheck when shellcheck is installed — my machine did not have it, so my “clean” result had been skipping an entire class of findings.

Fewer moving parts

Three things got removed rather than added:

Verification as a habit

The docs kept claiming things I had stopped checking, so I wrote scripts/verify-docs.sh: seventeen assertions against the live cluster — object counts, hostnames versus routes, a certificate probe for every hostname, chart pins versus live releases, sealed-secret sync state, image tags, and that Argo CD tracks the default branch.

It found real drift on its first run: adding Sealed Secrets had pushed pods 65 → 66 and the documented totals still said 65.

Kubernetes logo

The static half — links and anchors — runs in CI. The live half needs a kubeconfig, so it stays local on purpose: CI should not hold cluster credentials.

How this was written

A large part of this work was produced with AI assistance. The agent used was MiMo-V2.6-Flash (model ID mimo-v2.6-flash), running in OpenCode, and it did the bulk of the implementation: writing the manifests and workflows, running the validation suites, diagnosing failures, and drafting this post.

I read and verified all of it myself before publishing. Every claim here — the versions, the counts, the failure modes, the exit codes — was checked against the cluster and the repository rather than taken on trust. Where the agent was wrong, I made it fix the work: the shellcheck gap above, a slugifier that disagreed with GitHub, and a cut that silently produced an empty tag are all examples of errors that were caught by running the checks rather than by reading the code.

That last point is the real lesson. The tooling in this post exists because “it looks right” is not a verification strategy — for either of us.

Next

Sealed Secrets closed the phase-2 blocker, so the remaining work is phase 3 (addons as Argo Applications, Argo CD managing itself) and the resilience gaps: Longhorn has no off-cluster backup target, Grafana still has zero dashboards, and Alertmanager has nowhere to send alerts.

The full state lives in the repo’s cluster status.


Logos in this post are from Simple Icons, released under CC0 1.0; they remain trademarks of their respective owners and are used here to refer to the projects discussed. See assets/images/CREDITS.md.

, , , ,