feat(cli): self-syncing cache — p cache sync, query auto-sync, manifest artifact index

CHANGELOG.md

@@ -2,6 +2,153 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## `path p cache sync` — incremental session ingestion — 2026-07-09
+
+Adds `path p cache sync [types…]`, the first step toward a cache that fills
+itself: it enumerates sessions across the installed agent harnesses and
+derives into the cache only what is new or changed since the last sync, so
+users no longer have to `p import` each session by hand.
+
+- **`path-cli`** (0.16.0):
+  - `path p cache sync [claude|gemini|codex|opencode|cursor|pi]…` syncs the
+    named artifact types; with no arguments it syncs every harness.
+    Enumeration reuses the `path share` aggregation, and per-session
+    derivation goes through the same provider managers, so listing and
+    derivation always agree on provider roots.
+  - The sync manifest at `~/.toolpath/sync.json` maps artifact type →
+    artifact id → `{path?, cache_id, modified?, size?, synced_at}`. Change detection is stat-level — source mtime + size
+    for the file-backed providers, the DB row's updated-at for
+    opencode/cursor — so deciding "nothing changed" reads no session
+    bodies and a no-op sync is milliseconds. The manifest is written
+    atomically (temp file + rename,
+    `0600`) and checkpointed every 10 writes with derives running
+    newest-first, so an interrupted first run keeps nearly everything
+    it derived — and spent its time on the sessions that matter most.
+    Pending work reports progress on stderr (live `<type> done/total`
+    on a TTY, a plain line every 25 items otherwise; no-op syncs stay
+    silent).
+  - `ArtifactType` (in `sync.rs`) is the general enum naming artifact
+    sources: `p cache sync` types, the manifest keys, and import
+    cache-id prefixes all use it (it absorbs the former `HarnessArg`).
+    The `Harness` enum stays as the harness-only layer that
+    `share`/`resume` `--harness` take — you can't resume into a git
+    repo — mapping into the general enum via `Harness::artifact_type()`.
+  - Sync owns refresh semantics: it overwrites cache entries it
+    re-derives (no `--force` needed), while manual `p import` keeps its
+    error-on-hit default. Sessions deleted upstream keep both their cache
+    document and their manifest record — the cache is an archive, not a
+    mirror.
+  - Per-type summary on stderr (`claude   3 new, 1 updated, 240 unchanged`).
+    On a default run, harnesses with no sessions stay silent; explicit
+    types always report. Derivation failures warn and tally as `failed`
+    without aborting the run.
+  - `p import` and `share` record what they write: session derives carry
+    a provenance stub (source stamped before the read), and the cache
+    write upserts the manifest — so sync never re-derives an artifact
+    that import or share just produced. `ArtifactType` gains `Git`:
+    git imports are recorded (repo path + `<repo-tag>-<graph-id>` id)
+    but never re-derived by sync, since repos aren't discoverable.
+    Github and pathbase stay out of the manifest — remote services,
+    not artifact sources.
+  - Every import flow loops the per-session derive helpers — explicit
+    `--session`, picker multi-select, `--all`, and the most-recent
+    fallbacks — so everything entering the cache is recorded.
+    **Breaking**: `p import pi --all` now emits one Path document per
+    session, consistent with every other provider (it previously
+    produced a single combined Graph).
+  - `path query` now syncs before it reads, scoped to the query's own
+    flags: `--source claude` syncs only claude, `--id` prefixes narrow
+    likewise, a bare cache-wide query syncs every type, and
+    `--input`-only queries never touch the cache. Output is quiet
+    unless something was actually ingested (`synced claude: 3 new, 1
+    updated`), a sync failure degrades to querying the cache as-is,
+    and `--no-sync` opts out. This is the piece that makes the cache
+    an implementation detail: a new user can run `path query` with no
+    setup and get their sessions.
+  - The manifest is now an artifact *index*, not just a cache inventory:
+    `cache_id` on a record is optional, and a record without one means
+    "known, not materialized". `p cache rm` downgrades records instead
+    of orphaning them (the next sync re-materializes the doc — this
+    also fixes rm'd docs previously staying gone until the source
+    changed), and sync verifies doc existence before skipping, so
+    out-of-band deletions self-heal too.
+  - `--parent-dir <dir>` / `-d` on `p cache sync` and `path query`
+    restricts ingestion (and the query's reads) to artifacts under a
+    directory. Stat gate first, always: unchanged+cached artifacts skip
+    before any scope check; only artifacts that would cost a derive get
+    the constraint, with a one-line peek for codex (whose cwd lives
+    inside the rollout) memoized into the manifest so it happens at
+    most once per artifact. Claude project matching happens in slug
+    space (its dir slugs are lossy), and claude derives now source
+    `path.base` from the session's recorded cwd instead of the lossy
+    slug string.
+  - Copilot participates in sync like every other harness: an
+    `ArtifactType::Copilot` (stat-only enumeration over
+    `session-state/<id>/events.jsonl`, cwd peeked from `session.start`
+    and memoized for `--parent-dir`), per-session import loops with
+    provenance, and `path query --source copilot` auto-sync.
+  - `share` uploads straight from the cache when the picked session is
+    unchanged since its last sync (manifest stamp matches a fresh stat
+    and the doc exists) — re-deriving would reproduce the same bytes.
+    A grown session steps around the fast path and derives as before.
+  - **Maximal ingest, strip at egress**: the cache always holds the
+    fullest derivation — thinking blocks are ingested for claude and
+    gemini unconditionally (**breaking**: `p import gemini` loses its
+    `--include-thinking` flag), and the privacy decision moves to where
+    it belongs: `share` and `p export pathbase` strip thinking from
+    uploads by default, with `--include-thinking` there to opt in. The
+    local cache and `resume` fidelity keep everything.
+  - Review fixes: `p import <provider> --all` warns-and-skips unreadable
+    sessions again instead of aborting the batch; importing an artifact
+    the implicit query-sync already cached is a friendly no-op instead
+    of an exists-error; pi project scoping compares in its dir-encoded
+    space so hyphenated paths match `--parent-dir`; the copilot cwd
+    peek tolerates `session.start` anywhere in the first lines and
+    top-level cwd keys; a derive no longer clobbers the memoized peeked
+    cwd; claude sessions with no recorded cwd fall back to the caller's
+    `--project` for `path.base`; a small-file gemini peek that declines
+    in the prefix now still runs the full-parse fallback; and a
+    rebase-dropped `toolpath-pi` workspace pin is back at 0.6.1.
+  - PR review round 2: thinking-stripping now also scrubs sub-agent
+    turns embedded in `delegations` (gemini folds sub-agent chat files
+    into them, each turn with its own `thinking`), recursing into
+    nested delegations; manifest writers are serialized by an advisory
+    lock (`sync.json.lock`) and every write is a locked read-merge-save
+    — sync checkpoints merge only the records the run wrote — so
+    concurrent invocations (query auto-syncs, imports) union their
+    records instead of clobbering each other; an all-`None` stamp can
+    no longer vouch for a record in sync's unchanged gate (mirrors
+    `record_is_current` — unknowable freshness re-derives); the `share`
+    fast path stats its one artifact directly instead of enumerating
+    the whole artifact type.
+  - Claude fingerprints cover the whole session chain. Claude Code
+    rotates to a new JSONL file on continuation (plan-mode exit,
+    resume, fork) while `list_conversations` keys the chain by its
+    *oldest* segment — appends land in the newest file, so statting
+    the head file froze the fingerprint at the first rotation and sync
+    never saw later turns. All three stamp sites (enumeration, import
+    provenance, the `share` fast path) now share `claude_chain_stamp`:
+    max mtime across the chain's segments plus the sum of their sizes
+    — exactly the files `read_conversation` merges, so the fingerprint
+    and the derived doc move in lockstep however rotation behavior
+    shifts across Claude Code versions. Import also normalizes its
+    manifest key to the chain head, so importing a successor-segment
+    id records the artifact sync will look for.
+- **`toolpath-gemini`** (0.6.1): new `PathResolver::list_session_entries`
+  returns each session's listing id, inner `sessionId`, and backing
+  file/dir path, and `peek_session_id` is now bounded — it scans the
+  first 4 KiB of a main chat file (identity fields lead the JSON) and
+  falls back to a full parse only when they don't. `list_sessions`
+  delegates to it unchanged. This is what lets `p cache sync` fingerprint
+  gemini sessions without reading chat bodies.
+- **`toolpath-claude`** (0.12.1): `ClaudeConvo::session_chain` is now
+  public — it resolves the full session chain for a session id (oldest
+  segment first), which is what lets `p cache sync` fingerprint a
+  conversation across rotations without reading bodies. It rides the
+  same cached chain index as `list_conversations`, so calling it after
+  a listing costs no extra IO.
+- **`toolpath-cli`** (0.16.0): version bump only (tracks `path-cli`).
+
 ## Derive: resolve duplicate step ids — 2026-07-01
 
 - **`toolpath-convo`** (0.11.1): `derive_path` now guarantees the derived