fix(cdn): serialize builds per project to stop concurrent-build corruption#154
Merged
Conversation
added 2 commits
July 16, 2026 23:11
…bundle A webhook build for a push that changes no content models (a pure code push) still ran the full pipeline: it re-uploaded `_manifest.json` with the new commit SHA but skipped the locale-bundle block (which only runs when a model is affected). That left `_manifest.json.commitSha` ahead of every `_bundle/*.json`, and CDN consumers that key content freshness off the manifest rendered stale/empty content until a manual full rebuild re-aligned the two — reproduced on staging (Lanista/collabers), where every 22-file code-only webhook build stranded content and forced an 88-file manual rebuild, while 28-file content builds were always fine. The manifest tracks the CONTENT version, so a content-less push must not bump it. Early-return a 0-file no-op from executeCDNBuild when a selective build resolves zero affected models. Manual rebuilds (fullRebuild) and config/model-def changes never reach this branch, so they are unaffected; the same push also stops wasting a build cycle.
…ption Two CDN build triggers (the GitHub push webhook and the manual "Rebuild now" endpoint) ran fire-and-forget with no coordination, so two builds for one project could run at once and race on the shared R2 namespace — a manual full rebuild's "delete every object not in this build" cleanup can wipe a concurrent webhook build's fresh uploads, stranding content. Gate both triggers through a single atomic claim: `createCDNBuild` now calls the `claim_cdn_build` SQL function (per-project transaction-scoped advisory lock + stale-slot reclaim + conditional insert) and returns the new row, or null when a build is already in flight. The webhook skips on null; the manual trigger returns 409. Cross-instance safe (single Postgres), crash-safe (a dead build's slot is reclaimed after 15 min), and provider-agnostic (both DatabaseProvider impls call the same function, mirroring increment_mcp_cloud_usage_if_allowed). A shared runner (server/utils/cdn-build-runner.ts) persists each build's result and performs a bounded catch-up: if the branch head advanced while the build ran (a mid-build push whose own webhook was turned away by the claim), it claims and runs one more build at the new head, so no push is stranded. Serialization makes catch-up's full rebuilds safe. Validated against a real postgres:16: the claim returns an id, blocks the second concurrent claim (null), reclaims a >15-min-stale slot, and keeps exactly one in-flight row. db:verify:pg + full suite green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Both CDN build triggers — the GitHub push webhook (
server/api/webhooks/github.post.ts) and the manual "Rebuild now" endpoint (.../cdn/builds/trigger.post.ts) — ranexecuteCDNBuildfire-and-forget with no coordination. Two builds for one project could run concurrently and race on the shared R2 namespace. In particular the manual full rebuild's "delete every object not in this build" cleanup can wipe a concurrent webhook build's fresh uploads, stranding content until the next rebuild. Observed on staging: a manual build (19:36–19:37) overlapping a webhook build (19:37–19:38).Fix — one atomic claim, cross-instance + crash safe
Production runs on managed+postgres and can scale to multiple Nitro instances, so an in-process lock is insufficient. Both triggers now go through a single atomic gate:
supabase/migrations/019_cdn_build_single_inflight.sql—claim_cdn_build(...): aSECURITY DEFINERfunction that takes a per-project transaction-scoped advisory lock (released when the claim commits — never held for the build), reclaims slots whose build died mid-flight (>15 min), and inserts a newbuildingrow only if none is in flight. Returns the id, orNULLwhen blocked. Mirrorsincrement_mcp_cloud_usage_if_allowed; shared migration lineage (both provider pairs).DatabaseProvider.createCDNBuild→Promise<DatabaseRow | null>— both impls (postgres-db/cdn.tsviasql,supabase-db/cdn.tsviarpc) call the function.null= a build is already running.null(the running build's catch-up chases this commit). Manual returns 409 (cdn.build_in_progress).server/utils/cdn-build-runner.ts— shared completion path: persists the result, then does a bounded catch-up — if the branch head advanced while the build ran (a mid-build push turned away by the claim), it claims and runs one more full rebuild at the new head. Serialization makes those full rebuilds safe.Validation
postgres:16(full lineage applied): claim returns an id → second concurrent claim returnsnull→ a >15-min-stale slot is reclaimed (old rowfailed) and re-claimed → exactly one in-flight row remains.pnpm db:verify:pggreen.cdn-build-runner.test.ts): catch-up chases an advanced head, stops when stable, stops when the follow-up claim is blocked, and never runs after a failed build.Staging verification (post-merge)
cdn_buildsshows no overlappingbuildingrows.