feat(sync): support direct cloud upload#87
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
Please ensure your PR meets the following requirements:
Summary
This PR adds direct cloud upload support for Keystone Edge, including DP CLI configuration loading, robot asset ID resolution, stable multipart upload behavior, and an explicit resync path for already-synced episodes.
Motivation
Changes
Modified Files
cmd/keystone-edge/main.go- Wires cloud sync configuration into the edge server startup path.internal/config/config.go- Adds cloud sync configuration options and validation coverage.internal/services/sync_worker.go- Adds direct cloud upload execution, retry handling, credential refresh behavior, and episode resync queueing.internal/cloud/uploader.go- Implements direct multipart upload handling with stable upload part size across credential refreshes.internal/api/handlers/sync.go- Adds the episode resync endpoint.internal/api/handlers/robot.go- Supports robot asset ID management used by direct sync.internal/api/handlers/transfer.go- Persists asset ID snapshot metadata for transfer-created records.internal/cloud/cloudpb/proto/data_gateway.proto- Adds gateway API support needed by direct uploads.Added Files
internal/services/dp_config_loader.go- Loads DP device upload configuration.internal/services/dp_asset_resolver.go- Resolves asset IDs for episode upload context.internal/services/dp_raw_tags.go- Builds raw tags for direct DP uploads.internal/services/sync_errors.go- Defines sync retry and non-retryable error helpers.internal/db/migrations/000004_robot_asset_id.up.sql- Adds robot asset ID persistence.internal/db/migrations/000004_robot_asset_id.down.sql- Rolls back robot asset ID persistence.docs/designs/cli-cloud-sync-sidepath.md- Documents the CLI cloud sync side path design.docs/designs/cli-cloud-sync-sidepath.zh.html- Adds rendered design documentation.docs/designs/cloud-sync-go-direct-upload.zh.html- Adds rendered direct upload design documentation.Deleted Files
Type of Change
Impact Analysis
Breaking Changes
None.
Backward Compatibility
Fully backward compatible. Direct cloud sync is additive and controlled by configuration. Existing sync history remains intact when an episode is resynced.
Testing
Test Environment
GOCACHE=/tmp/go-cache.gofmt -landgit diff --check.Test Cases
Manual Testing Steps
Not performed.
Test Coverage
Validated with:
Screenshots / Recordings
Not applicable.
Performance Impact
Documentation
Related Issues
Additional Notes
pendingsync log instead of settingepisodes.cloud_syncedback to false.Reviewers
No specific reviewers requested.
Notes for Reviewers
internal/services/sync_worker.gofor retry, resync, and worker restart behavior.internal/cloud/uploader.gofor credential refresh and multipart completion behavior.Checklist for Reviewers