Skip to content

fix(deploy): keep the scaffold's pod spread in lockstep with the platform#88

Merged
devantler merged 3 commits into
mainfrom
claude/tenant-spread-matchlabelkeys-87
Jul 19, 2026
Merged

fix(deploy): keep the scaffold's pod spread in lockstep with the platform#88
devantler merged 3 commits into
mainfrom
claude/tenant-spread-matchlabelkeys-87

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Daily AI Engineer

Why

Tenants built from this template can hit a stalled rolling update the moment they scale past one replica — which the scaffold itself invites ("a scale-up to 2-3 replicas is actually drain-safe").

The platform already solved this. Its spread-pods policy adds a guard that scopes pod spreading per deployment revision, so a rolling update's new pods are not counted against the old ones. But that policy only applies when a workload has no spread rule of its own — and this scaffold ships one. So the scaffold quietly opts every tenant out of the platform's fix while keeping the strict scheduling rule the fix exists to make safe.

What

Brings the scaffold's spread rule in line with the platform's, and adds a CI check that compares the two against the platform's live policy — so if the platform changes its remedy, this repo goes red instead of silently drifting again.

Fixes #87 · Part of #6

…form

The platform's spread-pods ClusterPolicy injects its topology spread
constraint through Kyverno's add-if-not-present anchor
'+(topologySpreadConstraints)'. The scaffold authors its own constraint,
so that mutation never fires and the scaffold's block is what runs --
including the DoNotSchedule hazard, but WITHOUT the
matchLabelKeys: [pod-template-hash] guard the platform deliberately
added to stop rolling updates deadlocking (surge pods of the new
revision otherwise count against the old revision's skew).

Only bites at 2+ replicas, which the scaffold's own comment invites.

Adds the key, and a CI step that reads the expectation from the
platform's LIVE policy so the two cannot drift apart again.

Fixes #87
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 200d6040cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/validate-scaffold.yaml Outdated
The four-field projection silently discarded any field it did not name,
so the day the platform adds minDomains / nodeAffinityPolicy /
nodeTaintsPolicy the gate would stay green while the scaffold diverged
-- defeating the drift check in exactly the scenario it exists for.

Compare the complete constraint instead, normalising away only
labelSelector (genuinely different spelling, identical selection) and
sorting keys so authoring order on either side does not matter.

Addresses the P1/P2 review finding on #88.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

P2 confirmed and fixed in bcb6a17 — the finding is correct, and it is the same defect class the fix itself was written against: my guard measured a four-field proxy for "the constraint agrees" rather than the property. A platform-side minDomains / nodeAffinityPolicy / nodeTaintsPolicy would have been projected away on both sides, leaving the gate green through precisely the drift it exists to catch.

Now compares the complete constraint, normalising away only labelSelector (genuinely different spelling, identical selection) and sort_keys(..) so authoring order on neither side matters.

RED/GREEN-proved by executing the embedded run: block verbatim, including your scenario:

  • GREEN — current head passes.
  • RED AmatchLabelKeys dropped (the original defect): fails with both sides shown.
  • RED B — an out-of-allowlist field (nodeTaintsPolicy: Honor) introduced: now fails; under the old projection this was invisible. Injected on the scaffold side since the platform is not mine to edit — the comparison is symmetric, so it proves the same property.
  • RED C — block deleted entirely: fails closed on the empty read.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bcb6a1735b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/validate-scaffold.yaml Outdated
Comment thread .github/workflows/validate-scaffold.yaml Outdated
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Both round-2 P2s are CORRECT and remain OPEN — this PR stays a draft

Recording the state honestly rather than promoting on a partial fix.

Both findings are valid, and both are the same defect shape as round 1 — my check keeps measuring a projection instead of the property:

  • [0] indexing — only the first constraint is compared, while the anchor injects the whole list. A future zone-spread constraint could be omitted silently.
  • del(.labelSelector) — deleting it blinds the gate to a change in which pods the platform counts, and the selector is exactly what determines the skew.

Why it is not fixed in this push. I attempted the full fix (compare whole arrays; canonicalise matchLabelsmatchExpressions(In) and substitute each side's "this app" token with a sentinel, so equivalent selectors compare equal while genuine semantic change still fails). Two defects remain in that attempt, both verified locally:

  1. the matchLabels-fold emits a spurious empty {operator: In, values: []} entry on the platform side, which has no matchLabels at all; and
  2. the sentinel substitution leaves the two sides differently quoted ('<SELF>' vs <SELF>), so they compare unequal while being identical.

Shipping that would be the third iteration of the very defect being reported — a comparison routine that is itself subtly wrong. So I reverted to the pushed state and left it here.

Current head bcb6a17 is strictly better than main (whole-constraint compare instead of a four-field allowlist, and the scaffold now carries matchLabelKeys), and its CI is green — but it is not what this gate should finally be, so it is not promotion-ready. Carried forward.

…shapes

The lockstep gate read only `[0]` and deleted `labelSelector` outright, so it
was blind to two real drifts: the platform adding a second constraint the
scaffold never inherits, and the platform changing which pods its selector
counts. Compare the full arrays, and assert each side's selector against the
shape hand-verified equivalent rather than canonicalising two shapes into one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: d00b8d2cd2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

User-evaluation at head d00b8d2. The "user" of this gate is the CI job, so
I extracted the step's script straight out of the workflow YAML — verifying the
heredocs survive the block-scalar indent strip — and ran that exact script
against real inputs (kubectl kustomize of both the live platform policies and
the scaffold).

case expectation result
real platform + real scaffold green exit 0
platform gains a 2nd constraint red exit 1
platform changes its selector key red exit 1
platform adds minDomains red exit 1
scaffold selector stops selecting its own app red exit 1

Each fixture was derived from the live source plus exactly one delta, with the
delta asserted as a precondition before use — the first attempt produced a
fixture with 25 constraints instead of 2, and that assertion is what caught it.

Non-vacuity: the previous version of this step stays green on drifts 1,
2 and 4. The green run also prints the compared constraint and the selector
line, so a pass is visibly not an empty read.

@devantler
devantler marked this pull request as ready for review July 19, 2026 16:23
@devantler
devantler requested a review from a team as a code owner July 19, 2026 16:23
@devantler
devantler merged commit 89e806e into main Jul 19, 2026
13 checks passed
@devantler
devantler deleted the claude/tenant-spread-matchlabelkeys-87 branch July 19, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The scaffold opts itself out of the platform's rolling-update deadlock guard

1 participant