Open
Conversation
New deploy.Sync() replaces the filesystem-based SyncToR2 approach: - Queries packages where content_hash != deployed_hash - Serializes Composer p2 JSON from DB via composer.SerializePackage() - Uploads tagged + dev files to R2 in parallel (50 goroutines) - Deletes p2 files for deactivated packages (fixes existing gap where closures detected by check-status left orphaned files on R2) - Conditionally uploads packages.json (skips if ETag matches) - Stamps deployed_hash after successful upload Also adds deleteObjectWithRetry and headObject helpers to r2.go. Integration test (db_sync_test.go) validates full cycle against gofakes3: first sync uploads all, second sync is idempotent, deactivation triggers deletion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deduplicates the retry loop from putObjectWithRetry and deleteObjectWithRetry into a single withRetry function with exponential backoff and context cancellation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Concurrency field to R2Config (default 50) instead of hardcoding the errgroup limit in SyncToR2 and Sync. Follows the same pattern as DiscoveryConfig.Concurrency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Package.ComposerMeta() to centralize the nullable-field-to-PackageMeta mapping that was duplicated across sync.go, composer.go, and builder.go. Move dirty package and deactivated package queries into the packages package as GetDirtyPackages() and GetDeactivatedDeployedPackages(), keeping DB access out of the deploy layer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PackageFiles encapsulates the tagged/dev file split — callers get back
a slice of {Key, Data} without needing to know about ~dev naming, the
themes-have-no-dev rule, or trunk-only plugin handling.
R2KeysForPackage returns all possible R2 keys for deletion without
hardcoding suffix lists at call sites.
Simplifies deploy.Sync to a loop over PackageFiles for uploads and
R2KeysForPackage for deletions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
swalkinshaw
force-pushed
the
incremental-sync/db-driven-r2-sync
branch
from
aa52a97 to
20b5884
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #52
Summary
Add
deploy.Sync()— a new DB-driven R2 sync function that replaces the filesystem-based approach. This is additive (no existing code removed) and prepares for the Phase 3 cutover.deploy.Sync(): Queries packages wherecontent_hash != deployed_hash, serializes Composer p2 JSON viacomposer.PackageFiles(), uploads to R2 in parallel, deletes p2 files for deactivated packages, conditionally uploadspackages.json(ETag check), and stampsdeployed_hashcomposer.PackageFiles(): New abstraction that encapsulates the tagged/dev file split — callers get[]PackageFile{Key, Data}without knowing about~devnaming conventionscomposer.ObjectKeysForPackage(): Returns all possible Object keys for a package (used for deletion)Package.ComposerMeta(): Centralizes the nullable DB field →PackageMetamapping that was duplicated across sync, HTTP handler, and builderGetDirtyPackages()/GetDeactivatedDeployedPackages(): DB queries extracted from deploy into packages layerwithRetry(): Shared retry helper with exponential backoff, replacing duplicated loops inputObjectWithRetryanddeleteObjectWithRetryR2Config.Concurrency: Upload concurrency is now configurable (default 50) instead of hardcodedTest plan
go test ./internal/packages/— unit tests passgo test ./internal/deploy/— unit tests passgo test -tags=integration -run TestDBDrivenSync ./internal/integration/— full sync cycle passesTestR2Syncintegration test still passes (old path untouched)