feat: support partial (multi-request) scene uploads#1949
Open
LautaroPetaccio wants to merge 17 commits into
Open
feat: support partial (multi-request) scene uploads#1949LautaroPetaccio wants to merge 17 commits into
LautaroPetaccio wants to merge 17 commits into
Conversation
allow a scene's content to be uploaded across several POST /entities requests via an optional `partial=true` form field. content is staged until every referenced file is present, at which point the request that completes the set runs the full validation + deploy pipeline and the entity goes live (auto-finalize); earlier requests return 202 with the still-missing hashes. - new `pending_deployments` table + pending-deployments-repository, and a partial-deployments logic component - staging validation reuses the content-validator's individual validate fns (structure, signature, scene, land access) minus the size and content-completeness checks; size is checked cumulatively instead - at most one pending deployment per parcel set; a newer partial deploy on overlapping pointers replaces it - deployment-ttl check anchored on the pending row's created_at so an upload can span longer than the ttl; the deploy transaction removes the pending row - garbage collection treats non-expired pending hashes as referenced; a dedicated cleanup job expires stale rows (PENDING_DEPLOYMENT_TTL, default 24h) - requests without the flag behave exactly as before
add integration tests for concurrent staging of the same entity: two distinct content batches uploaded in parallel, and two requests completing the content set at the same time (concurrent finalize). both assert the entity is deployed exactly once with no leftover pending row and no server errors.
- keep the deployment-TTL pending lookup off the hot path: it now runs only when the entity is already too old by wall clock and only for scenes, so the high-volume profile/other deploys no longer pay a pending_deployments query on every deploy - fail a staging request fast when a referenced, already-stored content file has no determinable size, instead of counting it as 0 and only failing at finalize after the whole scene was uploaded - drop the correlated pending_deployments anti-join from the GC candidate query (a per-row subplan over the whole content_files table); the per-batch findReferencedHashes re-check already protects non-expired pending content - detect the concurrent-finalize pointer-lock conflict via an exported constant instead of matching the human-readable message text - move the staging advisory-lock SQL into the pending-deployments repository (acquireStagingLock), keeping raw SQL out of the logic component - store a staging batch's files in bounded-parallel batches and drop a redundant existence sweep; reuse the shared sleep helper
a rate-limited staging request previously threw a generic 400, which a client treats as terminal. rate limiting is transient (staged content is preserved for 24h), so return 429 instead, letting the client tell a resumable rejection from a real validation error. adds a regression test.
each batch of a multi-request upload was re-running the slow on-chain/subgraph land access check. resume batches now skip it, but only when the signer is the deployer who created the pending record — creating it required passing the check, uploaded bytes are hash-verified against the staged manifest, and finalize re-runs the full validation before anything goes live. any other signer takes the full staging validation, so a third party can't ride an existing upload's fast path. the pending row is still upserted on every batch (always before storing files): re-asserting it resurrects the row if a competing overlapping upload replaced it between requests, keeping the batch's staged content protected from garbage collection. expired rows are purged inside the same transaction so a resurrected row never carries a stale created_at. also: - deleteUnreferencedFiles (maintenance) now re-verifies each delete batch via findReferencedHashes right before deletion, so uploads that start after the bloom filter was built are no longer swept mid-upload - exhausting the finalize pointer-conflict retries returns 429 (transient, resumable) instead of 400 adds a test asserting the access check runs only on the record-creating request.
the protocol's access validation is historical by design: it proves ownership at the block of entity.timestamp, which is safe for vanilla deploys because REQUEST_TTL_BACKWARDS bounds the entity's age to minutes. a partial upload relaxes that bound to PENDING_DEPLOYMENT_TTL (24h), opening a gap: start an upload while owning the land, sell it, and complete the upload — the finalize access re-check would pass against the old block and deploy the seller's scene onto land they no longer own. the deploy pipeline now additionally requires access against the CURRENT chain state when a local scene deploy is older than the vanilla freshness bound (the exact case where only a pending-upload anchor let it through the TTL check). fresh deploys never hit the extra check, and it covers both completion paths — auto-finalize and a vanilla POST of a pending entity — since both go through the pipeline. respects the configured access strategy, including the IGNORE_BLOCKCHAIN_ACCESS_CHECKS no-op. adds regression tests: the completing request of a "sold mid-upload" scene is rejected, and finalizes once access is restored.
- restore bounded parallelism in deleteUnreferencedFiles: batches now run through a small concurrent window (4) so folder-based storage — whose delete() unlinks serially — doesn't degrade the maintenance sweep to fully sequential unlinks interleaved with blocking reference re-checks - drop the global expired-row sweep from the per-request staging transaction; the upsert now resets an expired row's created_at itself (fresh TTL window on conflict), leaving global expiry to the cleanup job where it belongs - reuse the size-check fileInfoMultiple lookup to decide which staged files to store, removing a redundant storage existence round trip per batch - restructure the partial-deployment integration spec per the dcl-testing standard: all setup and actions in beforeEach, it() blocks assert-only, flows expressed as nested describes (13 scenarios -> 26 assert-only cases)
storeMissing decided which files to skip from a fileInfoMultiple snapshot taken before the pending row (and its GC protection) was committed, so a file present at snapshot time but swept before the store ran would be skipped even though the client uploaded its bytes in that batch — the completeness read then reported it missing and the client burned a resume cycle. Store every uploaded file unconditionally: storage is content-addressed so a re-store is an idempotent no-op, and the client already omits files reported by /available-content, so this rarely re-sends present content. Also corrects a stale comment that claimed expired rows are purged in the staging transaction (that was replaced by the upsert's created_at reset; global expiry is the cleanup job's job).
…nups - the single per-pointer-set pending slot now goes to the NEWEST scene (deployment ordering: entity.timestamp, tie-break entity id). A strictly-newer overlapping upload is rejected instead of replaced, so a stale/older upload can't evict a newer competitor's staged content and two clients can't ping-pong. Adds entity_timestamp to the (unshipped) pending_deployments migration + getOverlappingPointers to the repository. - cap concurrent non-expired pending uploads per deployer (MAX_PENDING_DEPLOYMENTS_PER_DEPLOYER, default 10) so one account can't pin storage across many pointer-sets for the TTL. Only new uploads count. - type the pointer-lock conflict: InvalidResult gains kind: 'pointer-conflict' and the finalize retry branches on it instead of substring-matching the error message. - extract the shared bounded-parallel content-store loop (store-content.ts), used by both the vanilla deploy path and partial staging. - consolidate the duplicated REQUEST_TTL_BACKWARDS wall-clock condition behind a single predicate shared by the TTL-anchor logic and the current-access gate.
The count was read before the staging transaction, so concurrent new uploads by one deployer could each observe a count below the cap and then all insert, bypassing the limit. Move the count-and-check inside the staging transaction, under the advisory lock that already serializes staging, so the cap is enforced atomically against the insert.
- Decide the per-deployer concurrent-pending cap on the NET row-count change, under the advisory lock and after the overlap-replace, instead of a stale "is this new?" flag: countActiveByDeployer now excludes the current entity id, so `count + 1` is the deployer's post-upsert total. A resume, or a newer scene that replaced one of the deployer's own overlapping rows, no longer counts as an increase (and concurrent same-entity first batches don't wrongly reject). - Add a functional index on (LOWER(deployer_address), created_at) so the cap query doesn't scan all pending rows.
…load Mirrors the worlds change. A completed partial upload can be finalized by several requests at once (the client's parallel worker pool and retries), and each would independently run the full deploy pipeline (validation + on-chain access + deploy); today only the deployments unique-id insert stops the duplicate, after the expensive work is done. Add a lease to pending_deployments (status UPLOADING/FINALIZING + finalizing_at): a completing request atomically claims the lease and only then runs finalize; concurrent completers that miss the lease either see the entity already deployed (idempotent 200) or report in-progress (202) so the client's resume loop converges. A lease older than a 2-minute TTL is reclaimable, so a crashed finalizer doesn't wedge the upload; the lease is released on a finalize that fails without deploying. The existing idempotency remains the correctness backstop if two requests ever race.
Bumps @dcl/content-validator to ^7.4.0 and replaces the hand-composed staging validator with the library's public createStagingValidator. Drops the eight deep-imports into @dcl/content-validator/dist/validations/* (validateAll, entity-structure, ipfs-hashing, metadata-schema, ADR45, signature, scene, and the reject-extras content fn) and the local createStagingSceneValidateFn — the staging subset now lives behind the package's public api, so it no longer breaks on the library's internal reorganization. Behavior is unchanged: same validation set and order, includeAccessCheck toggles the resume variant, and the library additionally rejects non-scene entities up front (catalyst already does).
- revert the finalization lease: correctness already rests on the
deployments unique entity-id constraint plus finalize's pointer-conflict
retry, and the lease's 2-min ttl vastly exceeds the client's resume
budget, so a crashed/slow holder turned a succeeding upload into a
client failure; its unfenced release could also clobber a live
takeover's lease
- re-verify content presence immediately before finalize and return a
resumable 202 { missing } if a gc sweep reclaimed a reused file in the
validation window, instead of a terminal 400 the client can't resume
- filter getOverlappingPointers by ttl so an expired (dead) row no longer
triggers a newer-conflict rejection
- break the overlap ordering tie by lowercased entity id, matching the
deployments happenedBefore comparator
- enforce the per-deployer pending cap only when the upsert creates a new
row, so lowering the cap never wedges in-flight resumes
- cap the resume entity-file read-back at 10mb while streaming (parity
with worlds)
- fix a stale comment referencing validateAll
…scope staging locks
correctness:
- finalize now distinguishes the transient/racy causes that deployEntity
returns as kind-less invalid results (previously all terminal 400s the
client cannot resume): already-deployed -> idempotent success (a concurrent
finalize, incl. on another process whose duplicate insert hit the unique
entity-id constraint), content reclaimed by GC during validation -> 202
{ missing } so the client re-uploads, rate limited -> 429
- delete the pending row on every successful finalize: deployEntity's
already-deployed fast path returns without running the in-tx delete, so a
row re-created by a straggling resume batch would otherwise linger for the
full TTL and pin one of the deployer's cap slots
- reject a >10mb entity manifest on the first request too (the resume
read-back already caps at 10mb), so such a scene can't stage then wedge on
every resume
performance:
- replace the single global staging advisory lock with a per-deployer lock
plus one lock per pointer (sorted, deadlock-free), so uploads on disjoint
pointers by different deployers no longer serialize on one key
security / cleanup:
- restrict the resume fast path's overlap replace to the deployer's own rows,
so a deployer who lost access mid-upload can't evict another deployer's
staged upload (the fast path skips the access check)
- stop disclosing another upload's entity id in the newer-conflict error
- use the canonical happenedBefore comparator for pending-slot arbitration so
it can't diverge from how deployments are ordered at finalize
Expose the per-entity-type rate-limit window from the deploy rate limiter and attach it as a Retry-After header on the 429 responses the partial-deployment staging and finalize paths return for rate limiting. The batched client honors it as its resume backoff floor, so a redeploy inside the (20s scene) window waits the window out instead of exhausting its short exponential backoff and failing terminally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what
adds support for uploading a scene's content across several
POST /entitiesrequests instead of a single one, so scenes too large for one request can be deployed. an entity is not counted as live until all of its content is present.how
partial=trueform field onPOST /entitiesmarks a staging request. requests without the flag behave exactly as today.200 { creationTimestamp }(auto-finalize); otherwise it returns202 { missing: [hashes] }.created_at) so an upload can span longer than the ttl.PENDING_DEPLOYMENT_TTL(default 24h).key changes
pending_deploymentstable (migration) +src/adapters/pending-deployments-repository/.src/logic/partial-deployments/orchestration component.src/adapters/content-validator/component.ts).isRateLimitedexposed.content-files-repository+garbage-collectioncomponent; new cleanup job.testing
test/integration/controller/partial-deployments.spec.ts(multi-batch upload, resume/idempotency, single-request auto-finalize, missing entity file, hash mismatch, legacy behavior unchanged).yarn buildandyarn lintclean.follow-ups (out of scope)
PostEntity202type to@dcl/catalyst-api-specs.dcl-catalyst-clientto a batched deploy method and bump the dependency here.