fix(deploy): hcloud server-limit wedges the ENTIRE merge queue (autoscaler snapshot build needs a free server slot)

[devantler]

🤖 Generated by the Daily AI Assistant

Problem

Every merge_group 🚀 Deploy to Prod run fails while ensuring the autoscaler Talos snapshot:

✗ failed to apply updates: failed to ensure autoscaler config secret: looking up snapshot image for
autoscaler secret: failed to ensure Talos snapshot: talos snapshot build failed: creating the temporary
server failed: server limit reached (resource_limit_exceeded)

The snapshot build spins up a temporary hcloud server, but the project is at its server limit (10 servers: 3 control-plane + 3 static workers + 4 autoscale-cx33). With no free slot the temp server can't be created, so the snapshot (Talos v1.13.5, schematic df884f4f…) never builds.

This wedges the ENTIRE merge queue, not just Talos-config PRs. deploy-prod ensures the autoscaler config secret on every deploy, and that ensure step always looks up (and, when missing, builds) the snapshot image — independent of what the PR changed. Live proof today: it evicted #2416 and #2417 (Talos-config), and also #2418 — a workflows-only pin bump with a zero manifest diff — all with the identical resource_limit_exceeded on schematic df884f4f. So the earlier "non-Talos/workflows-only PRs still merge" assumption was wrong.

Acute impact: main is currently red (the TODOs workflow, healed by #2418→actions v9.0.0) and there is no path to green until a server slot frees, because #2418 itself cannot clear the queue. The autoscale pool has decayed 7→4 nodes (VPA recovery is underway) but is still at the project limit.

Proposed directions (pick one/combine)

1. Raise the hcloud project server limit by a couple of slots (maintainer-only account action) — immediate unblock; the v1.13.5 snapshot builds once, then is reused and pressure eases.
2. Check for / clean orphaned temp snapshot-build servers from prior failed builds accumulating against the limit (needs hcloud access).
3. Pre-build/cache the Talos snapshot out-of-band (scheduled job, or make deploy reuse an existing image and only build when genuinely absent) so the prod hot path never has to create a temp server against a full project — the durable fix (ksail deploy-mechanism hardening).
4. Shed over-provisioning — the node inflation is the underlying cause; continue letting VPA/autoscaler decay so a slot frees naturally.

Rough size

S–M. Direction 1/2 is a one-off maintainer/account action (immediate); 3 is deploy-mechanism hardening in ksail; 4 is operational and already trending down.

Impact

Blocks all merges to main (queue fully wedged) and keeps main red until a server slot frees. Highest-priority acute blocker.

[devantler]

🤖 Generated by the Daily AI Assistant

Live update + remediation linkage. The chosen fix is structural slot reduction, not raising the limit: #2420 (drop cx33 as an overflow node type — cx33/8G nodes can never scale down, ~3Gi of DaemonSets alone exceed the 50% scale-down threshold) plus ksail#5756 (autoscaler scale-down-eligibility knobs). That is already taking effect — the autoscale pool dropped from 4 → 2 cx33 during the last hour, so prod is now 8 servers (3 control-plane + 3 static workers + 2 cx33), freeing hcloud slots.

Deadlock to note: #2420 is a ksail.prod.yaml change that only lands via a deploy-prod, which is itself queue-wedged — so the live cluster-autoscaler patch (or an out-of-band apply) carries it until #2420 can merge.

Unblock order once a slot is free: re-enqueue #2418 (the promoted TODOs-red heal, workflows-only) first — a successful deploy also builds the missing v1.13.5 Talos snapshot, unblocking the rest — then #2416/#2417. Re-enqueue of #2418 from the unattended run was classifier-gated (prod-deploy path); leaving it promoted + CLEAN for the maintainer or the next run to enqueue now that slots are freeing.

[devantler]

🤖 Generated by the Daily AI Assistant

Update — the hcloud server-limit wedge has relieved; a different blocker is now on top. The 17:26Z merge_group deploy found the Talos snapshot (ID 404452940), so the temp-server slot is available (prod is down to 8 nodes after the cx33 drop). The queue is now blocked by a server-side-apply field-manager conflict on the live cluster-autoscaler Deployment's .command — left by the out-of-band kubectl-patch that dropped cx33 from --nodes ahead of #2420. Deploy-prod's ksail Apply can't take .command, so every merge_group deploy (and the heal job) fails.

Fix + one-click posted on #2420 (strip the stale kubectl-patch managed-fields entry, then re-enqueue #2420). The hcloud-limit hardening this issue tracks (pre-build snapshot / raise limit / clean orphaned temp servers) is still worth doing for resilience, but is no longer what's holding the queue right now.

[devantler]

🤖 Generated by the Daily AI Engineer

Recurred today (2026-07-16) — identically, for a new Talos version

This is not a one-off. It is back with v1.13.6 (schematic df884f4f…), same error, same mechanism:

✗ failed to apply updates: failed to ensure autoscaler config secret: looking up snapshot image for
autoscaler secret: failed to ensure Talos snapshot: talos snapshot build failed: creating the temporary
server failed: server limit reached (resource_limit_exceeded)

Timeline (from merge_group run history, not impression): every merge_group CI run succeeded through 10:21Z (#2662). #2662 — the Talos v1.13.5 → v1.13.6 bump — merged at 10:38Z. Every merge_group run since has failed: #2652 (12:33Z), #2635 (12:45Z), #2665 (12:49Z), and again #2665 (14:12Z), which was evicted from the queue a second time.

Why the bump is the trigger: while the pinned Talos version's snapshot already existed, findExistingSnapshot hit and no temporary server was ever needed. A version bump invalidates that lookup, so every deploy must now build the snapshot — and the build needs a temporary server the project has no room for. This makes it a standing trap: every future Talos bump re-wedges the queue, and Renovate proposes those regularly.

Escalation vs. 07-03: 🩹 Heal Prod is failing too. The heal path re-deploys main through the same ensure step, so it hits the identical failure (run 29505546754, both 🚀 Deploy to Prod and 🩹 Heal Prod = failure). There is currently no working CD path to prod, not merely a blocked queue.

Root cause, precisely

The invariant is in the committed config, which is self-documenting about it:

• provider.hetzner.serverLimit: 10 — the account's hard cap.
• autoscaler.node.maxNodesTotal: 10 — "the Hetzner account cap: 6 static (3 control-planes + 3 workers) + up to 4 autoscaler nodes".

maxNodesTotal is set equal to the account server limit, so a fully scaled-out cluster is entitled to consume every slot the account has — leaving zero headroom for the temporary snapshot-build server. The deadlock isn't a flake or a leak; the temp server fails at creation, so nothing is orphaned. It is a budgeting invariant: the cluster's ceiling must sit strictly below the account cap, because the deploy hot path needs one slot the cluster doesn't own.

This also explains why direction 4 (“let the pool decay”) appeared to fix it on 07-03 — decay frees a slot, so the symptom clears while the invariant stays broken and waits for the next bump.

Recommendation — one slot of headroom (direction 3/4, made durable)

Reserve one slot: hold maxNodesTotal one below serverLimit (9 <= 10). One line, declarative, and it makes the deadlock unreachable rather than waiting for decay. Cost: peak autoscale capacity 4 → 3 nodes. If that peak is genuinely needed, the equivalent is raise the hcloud account limit to 11+ and keep maxNodesTotal one below it (direction 1, and it preserves capacity).

I could not ship this myself: ksail.prod.yaml is protected — never modify per this repo's AGENTS.md, so the change is the maintainer's. I prepared and reverted it rather than cross that guardrail. I also can't free a slot directly (the local hcloud token is no longer valid).

Unblocking action (any one is sufficient, and immediate): free a single server slot — raise the account limit, or remove one autoscaler node. The v1.13.6 snapshot then builds once and is reused, and the queue drains on its own.

Suggested follow-up once unblocked: add a CI guard asserting maxNodesTotal < serverLimit, so the invariant can't silently regress.

[devantler]

🤖 Generated by the Daily AI Engineer

Adding the prevention side — filed devantler-tech/ksail#6171.

The immediate unwedge here is a capacity decision (raise the cap, or free a slot). But the reason this configuration was reachable at all is a KSail validation gap: pkg/apis/cluster/v1alpha1/validation.go:466 only rejects reachableTotal > serverLimit, so a total equal to the cap validates clean — leaving KSail no slot for the temporary server it needs itself to build the Talos snapshot. autoscaler_test.go:830 currently pins that as valid ("maxNodesTotal equal to serverLimit is valid despite large pools").

So KSail happily blesses a config that deadlocks its own deploy path on the next Talos bump. #6171 proposes reserving that slot at validation time, so the failure surfaces as a config error instead of a mid-deploy resource_limit_exceeded.

Live state as of ~14:35Z: the merge queue is now empty — all three enqueued PRs (#2665, #2652, #2635) were evicted, each by this same failure, and 🩹 Heal Prod fails identically. So there is currently no working CD path to prod, not just a blocked queue.

I did not re-queue anything: with the cluster confirmed at exactly the cap, re-queuing re-hits the same failure. I also did not touch ksail.prod.yaml — platform's AGENTS.md ## Maintenance lists it as protected, so the maxNodesTotal change is yours to make.