Skip to content

feat(sync): support direct cloud upload#87

Merged
shark0F0497 merged 7 commits into
mainfrom
feat/direct-cloud-upload
Jun 4, 2026
Merged

feat(sync): support direct cloud upload#87
shark0F0497 merged 7 commits into
mainfrom
feat/direct-cloud-upload

Conversation

@shark0F0497
Copy link
Copy Markdown
Collaborator

Pull Request Checklist

Please ensure your PR meets the following requirements:

  • Code follows the style guidelines
  • Tests pass locally
  • Code is formatted
  • Documentation updated if needed
  • Commit messages follow conventional commits
  • PR description is complete and clear

Summary

This PR adds direct cloud upload support for Keystone Edge, including DP CLI configuration loading, robot asset ID resolution, stable multipart upload behavior, and an explicit resync path for already-synced episodes.


Motivation

  • Edge sync currently needs a direct path to upload MCAP data to the cloud data platform without relying on the previous transfer side path.
  • Long-running uploads need robust STS credential refresh while keeping multipart session parameters stable.
  • Operators sometimes need to re-upload data that has already completed cloud sync without rewriting historical sync records.

Changes

Modified Files

  • cmd/keystone-edge/main.go - Wires cloud sync configuration into the edge server startup path.
  • internal/config/config.go - Adds cloud sync configuration options and validation coverage.
  • internal/services/sync_worker.go - Adds direct cloud upload execution, retry handling, credential refresh behavior, and episode resync queueing.
  • internal/cloud/uploader.go - Implements direct multipart upload handling with stable upload part size across credential refreshes.
  • internal/api/handlers/sync.go - Adds the episode resync endpoint.
  • internal/api/handlers/robot.go - Supports robot asset ID management used by direct sync.
  • internal/api/handlers/transfer.go - Persists asset ID snapshot metadata for transfer-created records.
  • internal/cloud/cloudpb/proto/data_gateway.proto - Adds gateway API support needed by direct uploads.

Added Files

  • internal/services/dp_config_loader.go - Loads DP device upload configuration.
  • internal/services/dp_asset_resolver.go - Resolves asset IDs for episode upload context.
  • internal/services/dp_raw_tags.go - Builds raw tags for direct DP uploads.
  • internal/services/sync_errors.go - Defines sync retry and non-retryable error helpers.
  • internal/db/migrations/000004_robot_asset_id.up.sql - Adds robot asset ID persistence.
  • internal/db/migrations/000004_robot_asset_id.down.sql - Rolls back robot asset ID persistence.
  • docs/designs/cli-cloud-sync-sidepath.md - Documents the CLI cloud sync side path design.
  • docs/designs/cli-cloud-sync-sidepath.zh.html - Adds rendered design documentation.
  • docs/designs/cloud-sync-go-direct-upload.zh.html - Adds rendered direct upload design documentation.

Deleted Files

  • None.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update (documentation changes only)
  • Refactoring (code improvement without functional changes)
  • Performance improvement (code changes that improve performance)
  • Test changes (adding, modifying, or removing tests)

Impact Analysis

Breaking Changes

None.

Backward Compatibility

Fully backward compatible. Direct cloud sync is additive and controlled by configuration. Existing sync history remains intact when an episode is resynced.


Testing

Test Environment

  • Go tests run locally with GOCACHE=/tmp/go-cache.
  • Formatting checked with gofmt -l and git diff --check.

Test Cases

  • Unit tests pass locally
  • Integration tests pass locally
  • E2E tests pass (if applicable)
  • Manual testing completed

Manual Testing Steps

Not performed.

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained or improved

Validated with:

GOCACHE=/tmp/go-cache go test ./internal/services -run 'TestEnqueueEpisodeResync_AllowsAlreadySyncedEpisode|TestDispatchPendingSyncLogs_TreatsSyncedPendingRowsAsResync|TestProcessEnqueuedEpisode_HoldsMarkerUntilProcessingReturns'
GOCACHE=/tmp/go-cache go test ./internal/api/handlers -run 'TestSync|TestRobotHandlerAssetID|TestAssetIDSnapshotMetadata'

Screenshots / Recordings

Not applicable.


Performance Impact

  • Memory usage: No expected material change
  • CPU usage: No expected material change
  • Throughput: Improved for direct cloud upload workflows
  • Lock contention: No expected material change

Documentation


Related Issues

  • None.

Additional Notes

  • Resync intentionally creates a new pending sync log instead of setting episodes.cloud_synced back to false.
  • The original multipart upload part size is kept stable for the full upload lifecycle, even when STS credentials are refreshed.

Reviewers

No specific reviewers requested.


Notes for Reviewers

  • Please review internal/services/sync_worker.go for retry, resync, and worker restart behavior.
  • Please review internal/cloud/uploader.go for credential refresh and multipart completion behavior.
  • Please verify the robot asset ID migration and API behavior are backward compatible.

Checklist for Reviewers

  • Code changes are correct and well-implemented
  • Tests are adequate and pass
  • Documentation is updated and accurate
  • No unintended side effects
  • Performance impact is acceptable
  • Backward compatibility maintained (if applicable)

@shark0F0497 shark0F0497 merged commit ec14027 into main Jun 4, 2026
5 checks passed
@shark0F0497 shark0F0497 deleted the feat/direct-cloud-upload branch June 4, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant