Conversation
Sync agent.proto with pennsieve-agent and regenerate agent_pb2.py and agent_pb2_grpc.py. Picks up Push (for map-based uploads) and Deregister (account removal), both previously missing from the Python stubs. Generated with grpcio-tools ~=1.51 / protobuf ~=4.21 to match the project's pinned versions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New Map class mirrors the manifest and timeSeries wrappers: constructed
against the agent stub, exposes the four map RPCs, and adds wait helpers
for the two fire-and-forget flows:
- wait_for_pull polls .pennsieve/state.json for isLocal=True entries,
since the agent doesn't emit pull-progress events.
- wait_for_push subscribes and counts upload_status COMPLETE messages,
which the existing UploadManifest flow already emits.
Wired onto Pennsieve.connect() as client.map. Tests cover RPC dispatch,
both pull wait happy path / timeout / late-arrival, invalid state.json,
and push completion counting / timeout via a fake streaming stub.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces blanket timeouts on wait_for_pull / wait_for_push with an idle-timeout that resets on each progress event, so transfers of any total duration succeed as long as the agent keeps making progress. Fails fast only when the stream actually goes silent. Defaults chosen for Pennsieve's dataset scale (~300-400 GB, ~700 MB max file): - wait_for_pull: 1800s (30 min) — must cover worst-case single-file download since state.json only updates per file completion. - wait_for_push: 300s (5 min) — the agent emits IN_PROGRESS on every S3 chunk read, so any real silence means a stall. Adds a Mapped Datasets section to the README documenting the wrappers, the two progress signals, and how to override idle_timeout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catches up the python client to add push
Also wrapped commands for pennsieve map / pull / push / diff
Some readme updates, and some clean up to prevent review dog from touching generated gRPC files