feat(cli): self-syncing cache — p cache sync, query auto-sync, manifest artifact index#131
Open
benbaarber wants to merge 20 commits into
Open
feat(cli): self-syncing cache — p cache sync, query auto-sync, manifest artifact index#131benbaarber wants to merge 20 commits into
p cache sync, query auto-sync, manifest artifact index#131benbaarber wants to merge 20 commits into
Conversation
`path p cache sync [types…]` enumerates sessions across the installed
agent harnesses (claude/gemini/codex/opencode/cursor/pi, or just the
named ones) and derives into the cache only what is new or changed.
The manifest at ~/.toolpath/sync.json maps artifact type → session id →
{project?, cache_id, last_activity?, message_count, synced_at}; a
session re-derives only when its fingerprint (last_activity +
message_count) differs. Manifest writes are atomic (temp + rename,
0600) and checkpointed per type; sync overwrites what it re-derives
(refresh semantics) and never deletes — the cache is an archive, not a
mirror.
Enumeration reuses cmd_share::gather_sessions; per-session derivation
goes through new *_with variants of the cmd_import session helpers so
listing and derivation share provider managers (and tests can inject
fixture resolvers). Real-data numbers: 34 sessions / 129 MB of Claude
JSONL → 5.7 s cold, 0.9 s incremental no-op.
path-cli 0.15.0 → 0.16.0 (+ toolpath-cli shim bump).
Sessions are just the first artifact kind sync and share operate on; the row type shouldn't bake that in. Fields keep their session-specific names until a second artifact kind actually lands.
ArtifactType (in sync.rs, pub, clap::ValueEnum) replaces the
Harness/HarnessArg pair and is now the single enum naming artifact
sources everywhere: p cache sync types, share/resume --harness,
resume's source inference, and cmd_import's cache-id prefixes (the
five wrap_paths_* clones collapse into one wrap_paths keyed on it).
It lives ungated so emscripten builds of cmd_import still see it; the
sync engine sits behind the cfg in a private submodule.
ArtifactRow sheds its session-specific shape: harness → artifact_type,
project → path, and message_count is gone — last_activity alone is the
sync fingerprint, and the manifest record slims to {path?, cache_id,
last_activity?, synced_at}. Pi's listing reports session start rather
than last activity, so collect_pi now stats the session file's mtime;
without that (or the old message_count), a growing pi session would
never re-sync.
ArtifactType will grow non-session kinds (git, github), and those must be unrepresentable where the CLI means "an agent runtime": you can't resume into a git repo. Harness (cmd_share.rs, ValueEnum) names the six runtimes and is what share/resume --harness and resume's picker, availability checks, invocation, and projection take; it maps into the general enum via artifact_type(), with ArtifactType::harness() as the partial inverse. Sync, the manifest, import prefixes, and ArtifactRow stay on ArtifactType.
SyncRecord regains message_count as an Option, giving session fingerprints a second signal alongside last_activity. It stays a harness-only notion: sync_rows strips it via artifact_type.harness() so future non-session artifact kinds record None, and ArtifactRow carries it as Option<usize> populated by the six harness collectors. The share picker shows the msgs column again as a side effect.
Change detection previously compared last_activity + message_count, both of which come from listings that parse every session body — the fingerprint cost as much as the thing it was avoiding. Sync now enumerates ArtifactStubs whose fingerprint is the source file's mtime + size (claude: head segment via the bounded first-lines peek of list_conversations; codex: rollout stat, id from the stem's trailing uuid; pi: file stat, id from a one-line header peek) or the DB row's updated-at (opencode: header-only SELECT; cursor: composer headers, bubble-less drafts skipped, workspace-less composers now included). Real data: a 34-session no-op sync drops 0.9s → 11ms. Gemini still enumerates via its metadata listing (main files carry the session id inside the JSON); its fingerprint is a stat of the listed file, and a peek-level listing in toolpath-gemini is the follow-up. message_count stays in the manifest but is informational only, recorded at derive time from the source the derive already read (DerivedDoc.message_count), still restricted to harness types. The stamp is taken before the derive reads the source, so a write racing the derive re-syncs next run instead of going unnoticed.
…fest toolpath-gemini 0.7.0 adds PathResolver::list_session_entries — each session's listing id, inner sessionId, and backing file/dir path — and bounds peek_session_id to the first 4 KiB of a main chat file (identity fields lead the JSON; full parse only as fallback, guarded so a sessionId appearing after the messages key is never trusted from the prefix). list_sessions delegates unchanged. Sync's gemini stubs now use it, so no provider reads chat bodies to decide nothing changed. message_count leaves SyncRecord and DerivedDoc entirely: nothing ever read it from the manifest. ArtifactRow keeps its count for the share picker column, which is its only consumer.
list_session_entries is purely additive; for 0.y.z crates cargo treats z as the compatible slot and a y bump signals potential breakage, so ^0.6 consumers should get this for free.
Every session derive now carries a provenance ArtifactStub on DerivedDoc — identity plus a source stamp captured before the read, using the same per-provider lookups sync's enumeration uses (head-file stat for claude, entries listing for gemini with the canonical inner uuid, resolved rollout stat + uuid for codex, header SELECTs for opencode/cursor, peeked session file for pi). The cache-write sites in p import and share upsert the manifest via sync::record_stub, so the next sync sees those artifacts as unchanged instead of re-deriving them. The multi-select loops route through the derive_*_session_with helpers so picker imports are recorded too; bulk --all derives for claude/gemini/pi/codex carry no provenance and are picked up by the next sync instead. --no-cache paths record nothing. ArtifactType gains Git (7 variants): git imports are recorded — repo path plus the <repo-tag>-<graph-id> id — but never re-derived by sync, since repos aren't discoverable. Github and pathbase stay out of the manifest: remote services, not local artifact sources.
The bulk --all arms for claude/gemini/pi/codex and every most-recent fallback now enumerate ids and loop the derive_*_session_with helpers instead of calling the providers' bulk derive_project, so every cache write carries provenance and lands in the sync manifest. There was nothing to the bulk path performance-wise — read_all_* reads the same files one at a time — and wrap_paths plus the dead per-fn configs go with it. Breaking: p import pi --all now emits one Path document per session, consistent with every other provider (previously a single combined Graph via derive_graph).
A query now freshens exactly the slice of the cache it will read: --source claude syncs only claude, --id prefixes narrow likewise, a bare cache-wide query syncs every artifact type, and --input-only queries never touch the cache. Reporting is quiet unless something was actually ingested (synced claude: 3 new, 1 updated); a sync failure warns and degrades to querying the cache as-is; --no-sync opts out. Real data: a first-ever query over an empty cache ingests 32 sessions and answers in one command; the steady-state sync adds nothing measurable. Existing query integration tests pin $HOME to a shared empty sandbox so the implicit sync runs against nothing instead of the developer's real sessions.
SyncRecord.cache_id becomes optional: a record without one is known
but not materialized. p cache rm downgrades records instead of
orphaning them — fixing rm'd docs previously staying gone until their
source changed — and sync verifies the doc file exists before
skipping, so out-of-band deletions self-heal the same way.
--parent-dir <dir> / -d on p cache sync and path query restricts
ingestion (and the query's reads) to artifacts under a directory.
The stat gate always runs first: unchanged+cached artifacts skip
before any scope check; only artifacts already facing a derive get
the constraint. Path-keyed providers prune whole projects before
enumerating; opencode/cursor check the directory their cheap headers
carry; codex — whose cwd lives inside the rollout — gets a one-line
peek only when new/changed, memoized into the record's path so it
happens at most once per artifact. Out-of-scope work tallies
separately and never touches a materialized record's stamp, so
staleness stays visible to the next in-scope sync.
Claude's dir slugs are lossy ('/', '_', '.' all became '-'), so its
scope matching happens in slug space, and its session derives stop
passing the slug-derived project string into DeriveConfig — path.base
now comes from the session's own recorded cwd.
Rebased ben/cache-sync onto the rewritten ben/query (now on main with the Copilot provider). Copilot becomes a full participant: an ArtifactType::Copilot + Harness::Copilot pair, stat-only enumeration over session-state/<id>/events.jsonl (dir name is the id), a bounded first-line peek for the session.start context cwd — memoized like codex's — per-session import loops with provenance, the share aggregator and resume arms restored onto the Harness-typed surfaces, and query auto-sync via --source copilot.
sync::fresh_cache_id answers "is this artifact's cached doc current?" — a fresh stat matches the manifest stamp and the doc file exists — using the same stub enumeration sync itself trusts. share_explicit checks it before deriving: on a hit it uploads the cached JSON as-is (a derive would reproduce the same bytes) and says so on stderr; a grown session or an evicted doc steps around the fast path and derives as before. --no-cache skips the check entirely.
The cache is the archive and now always holds the fullest derivation: claude and gemini ingest thinking unconditionally (breaking: p import gemini loses --include-thinking), while the privacy decision moves to the egress surfaces — share and p export pathbase strip thinking from uploads by default with --include-thinking to opt in there. Local projection keeps everything, so resume fidelity improves. This also dissolves the review's top finding: sync can no longer clobber a flag-derived doc because per-doc derive variance no longer exists. The other nine review findings: - p import --all warns-and-skips unreadable sessions again (all five per-session providers) instead of aborting the batch - importing an artifact the implicit query-sync already cached is a no-op with a friendly message, gated on real (non-None) stamps so git re-imports still honor error-on-hit - pi project scoping compares in its dir-encoded space, so hyphenated real paths match --parent-dir (mirror of claude's slug-space fix) - the copilot cwd peek scans a bounded number of lines and accepts top-level cwd keys, matching toolpath-copilot's own tolerance - a derive no longer clobbers the memoized peeked cwd in the record - claude sessions with no recorded cwd fall back to the caller's project for path.base instead of deriving baseless - gemini's peek runs the serde fallback even when the prefix scan declined on a small file (sub-agent dedup stays correct) - the toolpath-pi workspace pin returns to 0.6.1 (rebase damage) - sed-mangled doc comments and the stale six-harness count fixed
…rgeted share stat - strip_thinking also scrubs sub-agent turns embedded in delegations (gemini folds sub-agent chat files in, each turn with its own thinking), recursing into nested delegations - manifest writers serialize on an advisory lock (sync.json.lock); every write is a locked read-merge-save and sync checkpoints merge only the records the run wrote, so concurrent invocations union their records instead of clobbering each other - sync's unchanged gate no longer lets an all-None stamp vouch for a record (mirrors record_is_current) - share's fresh-cache fast path stats its one artifact directly (stamp_one) instead of enumerating the whole artifact type - ArtifactType::parse delegates to clap's ValueEnum instead of duplicating the name table
Claude Code rotates to a new JSONL file on continuation (plan-mode exit, resume, fork) while the chain keeps its oldest segment's id — appends land in the newest file, so statting the head file froze the fingerprint at the first rotation and sync never saw later turns. - toolpath-claude 0.12.1: ClaudeConvo::session_chain is now public (rides the cached chain index list_conversations already builds) - all three stamp sites (enumeration, import provenance, share fast path) share claude_chain_stamp: max segment mtime + summed segment sizes — exactly the files read_conversation merges, so fingerprint and derived doc move in lockstep across rotation-behavior changes - import normalizes its manifest key to the chain head so importing a successor-segment id records the artifact sync will look for
The doc quality gate (-D rustdoc::private-intra-doc-links) rejects public docs linking private items.
|
🔍 Preview deployed: https://0367b02f.toolpath.pages.dev |
The first sync on a machine with years of sessions used to grind silently and, if interrupted, forget everything in the in-progress type. - progress on stderr whenever a sync has pending work: a live \r-updating '<type> done/total' line on a TTY, a plain line every 25 items otherwise; no-op syncs stay silent - manifest checkpoints every 10 writes instead of once per type, so an interrupted run keeps nearly everything it derived (out-of-scope peek memoizations included) - derives run newest-first, so partial progress covers the sessions the user most likely wants
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the local cache fill itself instead of relying on manual
p import. Supersedes #128 (closed when its baseben/querymerged as #114); since then the branch was rebased onto the merged query work and gained the query auto-sync, manifest-index, parent-dir scoping, Copilot, and share-fast-path rounds, plus post-review hardening (privacy strip coverage, manifest locking, rotation-proof claude fingerprints, sync progress/interruptibility).path p cache sync [types…]enumerates sessions across the installed harnesses (claude / gemini / codex / opencode / cursor / pi / copilot) and derives into the cache only what's new or changed. Change detection is stat-level — source mtime + size for file-backed providers, the DB row's updated-at for opencode/cursor — so a no-op sync reads no session bodies and finishes in milliseconds. Claude fingerprints cover the whole session chain (max segment mtime + summed sizes viaclaude_chain_stamp): Claude Code rotates to a new file on continuation while the chain keeps its oldest segment's id, so statting the head file alone would freeze the fingerprint at the first rotation and sync would never see later turns.The manifest (
~/.toolpath/sync.json, atomic 0600 writes) is an artifact index, not just a cache inventory: a record without acache_idmeans "known, not materialized", sop cache rmdowngrades records instead of orphaning them and out-of-band deletions self-heal on the next sync. Writers serialize on an advisory lock (sync.json.lock— a stable inode, since the manifest itself is rename-replaced) and every write is a locked read-merge-save of only the records that run produced, so concurrent invocations union instead of clobbering. Sync checkpoints every 10 writes and derives newest-first, so an interrupted first sync keeps nearly everything and spent its time on the sessions that matter most; pending work reports progress on stderr (live<type> done/totalon a TTY, a line every 25 items otherwise — no-op syncs stay silent). Sync owns refresh semantics (overwrites what it re-derives; manualp importkeeps error-on-hit), and the cache is an archive, not a mirror — sessions deleted upstream keep their documents.path querysyncs its scope before reading —--source claudesyncs only claude,--idprefixes narrow likewise, a bare cache-wide query syncs every type, and--input-only queries never touch the cache. Quiet unless something was actually ingested, a sync failure degrades to querying the cache as-is, and--no-syncopts out. This is the piece that makes the cache an implementation detail: a new user can runpath querywith no setup and get their sessions.--parent-dir <dir>/-donp cache syncandpath queryrestricts ingestion (and the query's reads) to artifacts under a directory. Stat gate first, always: unchanged+cached artifacts skip before any scope check; only artifacts that would cost a derive pay it, with one-line cwd peeks for codex/copilot memoized into the manifest. Claude project matching happens in slug space, and claude derives now sourcepath.basefrom the session's recorded cwd instead of the lossy dir slug.Everything that writes the cache records it — every
p importflow (--session, picker multi-select,--all, most-recent fallbacks) andshareupsert the manifest (claude imports key their record by the rotation-stable chain head), so sync never re-derives what they just produced.shareuploads straight from the cache when the picked session is unchanged since its last sync, statting just that one artifact. Maximal ingest, strip at egress: thinking blocks are always cached for claude and gemini; the privacy decision moves toshare/p export pathbase, which strip thinking from uploads by default with--include-thinkingto opt in — the strip also scrubs sub-agent turns embedded indelegations, recursively. Local cache andresumefidelity keep everything.toolpath-gemini0.6.1: newPathResolver::list_session_entries, andpeek_session_idis now bounded (4 KiB prefix scan with full-parse fallback) — this is what lets sync fingerprint gemini sessions without reading chat bodies.toolpath-claude0.12.1:ClaudeConvo::session_chainis now public (rides the cached chain indexlist_conversationsalready builds) — this is what lets sync fingerprint whole chains without reading bodies.Breaking (pre-1.0):
p import pi --allnow emits one Path document per session (previously a single combined Graph);p import geminiloses--include-thinking(thinking is always ingested; stripping moved to egress).path-cli0.15.0 → 0.16.0.Tests: full
scripts/quality_gates.shgreen (fmt, shellcheck, clippy, test, doc, examples, site); new coverage for sync ingestion, query auto-sync, the manifest index, parent-dir scoping, rotation re-sync, delegation stripping, and progress/ordering.Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.