🤖 Generated by the Daily AI Engineer
Problem
scripts/refresh-flux-ghcr-auth.sh reboots every node it selects for a GHCR credential sync (#2635 — the reboot is the only way a running containerd adopts new registry auth; Talos refuses service cri restart).
A node that was just created — an autoscaler scale-out, a DR create — already booted with the current credential, so its reboot is wasted. It gets one avoidable drain+reboot on the next deploy, which churns exactly the capacity the autoscaler had just added.
Why the obvious fix is wrong
The tempting discriminator is the platform.devantler.tech/ghcr-pull-desired-revision marker: reboot only nodes whose marker is stale. That is unsafe, and it fails in the direction that hurts.
The marker tracks machine-config state — and machine-config state being decoupled from what containerd actually loaded is the bug the reboot exists to fix. Two live paths stamp the current marker on nodes whose containerd still holds the old credential:
- the deploy runs the bridge twice, before and after
ksail cluster update (.github/actions/deploy-prod), and cluster update stamps the marker on every node;
- the manual runbook path can run
cluster update ahead of the bridge.
Skipping a reboot on that evidence would silently re-create the 2026-07-14 outage (four releases behind, prod green, no alert). So #2635 keeps the reboot unconditional: an over-eager reboot costs ~90s on a drained node; a skipped one cost a day of ImagePullBackOff.
Proposed direction
Skip the reboot only on evidence about containerd itself, never the machine config. The invariant to prove is: containerd started AFTER the current credential landed in its config.
Candidate signal: compare containerd's process start (talosctl -n <ip> service cri → Process(...) started ... ago) against the mtime of the rendered /etc/cri/conf.d/01-registries.part, both read before the auth patch. If containerd started after the file was last written, it loaded the current contents → skip the reboot.
Needs care: the parse is fragile (human-formatted "(258h12m ago)"), and Talos may rewrite the file on a no-op apply, which would make a fresh node look stale (safe direction — a wasted reboot, not a missed one). Fail toward rebooting.
Acceptance criteria
Rough size
S–M. Small diff, but the correctness argument is the whole job.
Deferred from the Codex review on #2635.
Problem
scripts/refresh-flux-ghcr-auth.shreboots every node it selects for a GHCR credential sync (#2635 — the reboot is the only way a running containerd adopts new registry auth; Talos refusesservice cri restart).A node that was just created — an autoscaler scale-out, a DR create — already booted with the current credential, so its reboot is wasted. It gets one avoidable drain+reboot on the next deploy, which churns exactly the capacity the autoscaler had just added.
Why the obvious fix is wrong
The tempting discriminator is the
platform.devantler.tech/ghcr-pull-desired-revisionmarker: reboot only nodes whose marker is stale. That is unsafe, and it fails in the direction that hurts.The marker tracks machine-config state — and machine-config state being decoupled from what containerd actually loaded is the bug the reboot exists to fix. Two live paths stamp the current marker on nodes whose containerd still holds the old credential:
ksail cluster update(.github/actions/deploy-prod), andcluster updatestamps the marker on every node;cluster updateahead of the bridge.Skipping a reboot on that evidence would silently re-create the 2026-07-14 outage (four releases behind, prod green, no alert). So #2635 keeps the reboot unconditional: an over-eager reboot costs ~90s on a drained node; a skipped one cost a day of ImagePullBackOff.
Proposed direction
Skip the reboot only on evidence about containerd itself, never the machine config. The invariant to prove is: containerd started AFTER the current credential landed in its config.
Candidate signal: compare containerd's process start (
talosctl -n <ip> service cri→Process(...) started ... ago) against the mtime of the rendered/etc/cri/conf.d/01-registries.part, both read before the auth patch. If containerd started after the file was last written, it loaded the current contents → skip the reboot.Needs care: the parse is fragile (human-formatted "(258h12m ago)"), and Talos may rewrite the file on a no-op apply, which would make a fresh node look stale (safe direction — a wasted reboot, not a missed one). Fail toward rebooting.
Acceptance criteria
cluster updatehas already stamped the current desired marker on it.scripts/tests/test_refresh_flux_ghcr_auth.py, including the post-cluster updatecase.Rough size
S–M. Small diff, but the correctness argument is the whole job.
Deferred from the Codex review on #2635.