fix: S3 blobstore multi-tenant, GCS ACL, blob CID, video embeds#166
Open
rabble wants to merge 1 commit intoblacksky-algorithms:mainfrom
Open
fix: S3 blobstore multi-tenant, GCS ACL, blob CID, video embeds#166rabble wants to merge 1 commit intoblacksky-algorithms:mainfrom
rabble wants to merge 1 commit intoblacksky-algorithms:mainfrom
Conversation
…, and video embed support - Separate S3 bucket name (PDS_BLOBSTORE_S3_BUCKET) from DID path prefix, fixing multi-tenant deployments where all actors share one bucket - Skip object ACLs when endpoint is storage.googleapis.com (GCS uses uniform bucket-level access, not per-object ACLs) - Fix copy_source format to use bucket/key instead of requiring AWS_ENDPOINT_BUCKET env var - Use correct raw codec 0x55 for blob CIDs (was incorrectly using 0x77) - Always emit firehose commit ops for all writes, not just those with swap_cid (fixes missing events on putRecord) - Add video/mp4 and text/vtt blob constraints for app.bsky.embed.video - Walk Ipld::List and Ipld::Map variants in find_blob_refs so video embed blobs are discovered when records are deserialized from JSON - Stop double-encoding DID URLs in PLC client requests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PDS_BLOBSTORE_S3_BUCKETenv var) from the DID-based path prefix. Falls back to using the DID as the bucket name for backwards compatibility with single-tenant setups.storage.googleapis.cominAWS_ENDPOINTand skip per-object ACLs, since Google Cloud Storage uses uniform bucket-level access control.bucket/keyformat directly instead of requiring a separateAWS_ENDPOINT_BUCKETenv var.0x55for blob CIDs. The previous value0x77is an obsolete/incorrect codec that causes CID mismatches with the ATProto reference implementation.swap_cidset. This fixes missing firehose events when usingputRecordwithout swap.Ipld::ListandIpld::Mapvariants infind_blob_refsso video embed blobs are discovered when records come from JSON deserialization. Addvideo/mp4andtext/vttblob constraints forapp.bsky.embed.video.Test plan
PDS_BLOBSTORE_S3_BUCKETset to a shared bucket namePDS_BLOBSTORE_S3_BUCKET(falls back to DID)cargo test -p rsky-pdsfor new unit tests on blob ref discovery and constraintscargo test -p rsky-commonfor CID codec tests🤖 Generated with Claude Code