Cross-harness context-compaction provenance (kind v1.2.0)#108
Open
benbaarber wants to merge 38 commits into
Open
Cross-harness context-compaction provenance (kind v1.2.0)#108benbaarber wants to merge 38 commits into
benbaarber wants to merge 38 commits into
Conversation
|
🔍 Preview deployed: https://cad7b371.toolpath.pages.dev |
ben-emp
force-pushed
the
ben/compaction
branch
3 times, most recently
from
e0d1bab to
7cb45a7
Compare
ben-emp
force-pushed
the
ben/derive-dedup
branch
2 times, most recently
from
1130be3 to
bc7e3b8
Compare
fix(derive): resolve duplicate step ids so paths keep unique ids
…rse, verified against the live CLI (#117) * feat(copilot): add GitHub Copilot CLI provider (preview) + format docs Adds toolpath-copilot 0.1.0, a forward provider that derives Toolpath documents from GitHub Copilot CLI (@github/copilot) sessions under ~/.copilot/session-state/<id>/events.jsonl. - Tolerant events.jsonl parser (schema reverse-engineered, authored without first-hand samples): session/user/assistant/tool/subagent/skill/hook events -> ConversationView -> shared derive_path. Tool names classified into the ToolCategory ontology; file writes with full content get a raw diff. - workspace.yaml git context (root/repository/branch/revision) -> Path.base, via a tolerant key-scan parser (no YAML dependency). - CLI forward path only: path p import/list/show copilot (mirrors codex). - 8-doc on-disk format reference at docs/agents/formats/copilot-cli/, with every claim confidence-tagged (official / reverse-eng / inferred / unverified) and a verify-once-we-have-samples checklist. - Bumps path-cli 0.14.0 -> 0.15.0 (new dep + subcommands). Preview: no projector (so no path p export copilot / path resume / path share) and the events.jsonl schema is unverified against a real session -- see docs/agents/formats/copilot-cli/known-gaps-and-sourcing.md. * fix(copilot): pair id-less tool start/complete instead of double-counting The reverse-engineered events.jsonl schema never confirmed a correlation id on tool.execution_* events (sources report only name/args/success). The prior logic paired solely by id, so an id-less complete fell through to a synthesized carrier — double-counting every tool call and duplicating file mutations. attach_tool_result now treats an id as authoritative when present, and falls back to positional pairing (most-recent result-less invocation in the open turn, preferring the same tool name) when absent. Adds tests for the id-less case and a regression guard for the id-bearing case; documents the correlation uncertainty in events.md + the verification checklist. * fix(copilot): correct schema against a real session (copilotVersion 1.0.67) Verified the reverse-engineered events.jsonl format against a first-hand capture and corrected several guessed field locations that lost data: - session.start: cwd + git context are under data.context.{cwd,gitRoot, repository,branch,headCommit}, not top-level; CLI version is copilotVersion (top-level `version` is an int schema version). Base now carries the commit. - tool.execution_complete: result text is under data.result.content (an object), not a top-level string -- tool results were previously dropped. - assistant.message: capture reasoningText (-> Turn.thinking) and per-message outputTokens (summed as the session output total when no session.shutdown). - tool correlation is via toolCallId (confirmed); positional fallback retained. Also: fix a flaky test (strict-mode reader test mutated a process-global env var, racing the parallel non-strict test -- now threads `strict` explicitly); reshape all fixtures to the real envelope; refresh the docs folder from [inferred]/[unverified] to [observed, 1.0.67] with resolved-vs-open gaps. Verified live: `path show/import copilot` against the real session yields base {uri, ref=<commit>, branch}, bash/view tool results (1574/223 chars), and thinking populated. Envelope + core event types confirmed; subagent/skill/hook/ abort/shutdown/compaction and checkpoints/ remain unverified (not exercised). * feat(copilot): wire Copilot into `path share` `path share` is a separate orchestrator from the `p import/list/show` plumbing and didn't know about Copilot, so Copilot sessions never appeared in it. Its forward direction (derive -> upload to Pathbase) needs no projector, so: - Add Harness::Copilot / HarnessArg::Copilot and thread it through name/symbol/ parse/from_arg/harness_to_arg, the HarnessBundle, the gather-sessions probe (collect_copilot), the not-found classifier, and the "no sessions" status summary. - derive_session dispatch -> new cmd_import::derive_copilot_session. - Resume targets: argv_for gets `--resume <id>`; project_into_harness bails with a clear "no projector yet" message (resume into Copilot can't work without the reverse mapping). Adds share unit tests (gather includes/filters Copilot) and updates the docs (CLAUDE.md, CHANGELOG, crate README) to reflect share support. Verified live: `path share --harness copilot --session <real>` against a dead URL derives the real session and fails only at upload (Connection refused) -- i.e. share now surfaces Copilot end-to-end. path-cli 296 unit + 69 integration pass; clippy clean. * feat(copilot): projector + resume/export + conformance; fix resume loader Adds the reverse path (ConversationView -> Copilot events.jsonl), wires Copilot into resume/export and the cross-harness conformance matrix, and fixes the first real-CLI resume rejection. - New `CopilotProjector` (project.rs): emits the observed events.jsonl shape (session.start context, user/assistant.* with reasoningText + per-message tokens, tool.execution_* with result.content, subagent.*), remapping foreign tool names via a new `native_name`. Envelope `id`/`parentId` are UUID strings — the real `copilot --resume` rejects non-UUID ids ("invalid session event envelope: `id` must be a UUID string"), reported from a live run. - Forward provider carries per-turn token_usage (summed for the total) for cross-harness token survival, and assigns position-stable turn ids so the parent graph is idempotent. - path-cli: `path resume` / `path p export copilot` via `project_copilot` (writes session-state/<id>/{events.jsonl,workspace.yaml} + a session-store.db `sessions` row, fresh uuid, INSERT-only). Fixes the resume picker (Copilot was missing from ALL_HARNESSES; infer_source_harness now knows "copilot"). uuid v4. - Cross-harness conformance: `CopilotHarness` in cross_harness_matrix (test-fixtures/copilot/) — all cells pass. New resume integration test.⚠️ Real `copilot --resume` acceptance is still being verified against live runs; the envelope-id fix clears the first reported error. Projector otherwise validated by the matrix + round-trip. * fix(copilot): offset-bearing ISO timestamps on every projected event Second live `copilot --resume` rejection: "`timestamp` must be an ISO 8601 date-time string with a timezone offset". The projector stamped session.start with an empty timestamp (omitted) and passed through turn timestamps unchecked. - Projector now stamps every event (session.start included) with a valid offset-bearing RFC 3339 timestamp: picks a base (first offset-bearing turn ts, else the view's started_at) and normalizes each turn's timestamp against it. - Document the discovered loader contract: new docs/agents/formats/copilot-cli/writing-compatible.md (UUID envelope ids + offset timestamps, with the verbatim rejection messages), linked from the folder README and known-gaps. Updated as new rejections surface. Verified: projected session.start now carries id=<uuid> and timestamp=2026-07-01T14:31:29.298Z. Crate 59 + matrix + resume tests green. * fix(copilot): emit parentId (null on root) Third live `copilot --resume` rejection: "`parentId` must be a UUID string or null" — the root session.start omitted parentId entirely. The projector now always emits parentId: the previous event's UUID, or null on the root. - project.rs: parentId always present (UUID | null). - docs: writing-compatible.md req 3 upgraded to [observed, 1.0.67] with the verbatim error. Verified: projected session.start now has id=<uuid>, parentId=null, timestamp=<offset ISO>. Crate + matrix + resume tests green. * fix(copilot): emit session.start `startTime` (+ full observed top-level shape) Fourth live `copilot --resume` rejection: "line 1: missing field `startTime`". The projected session.start `data` omitted it. - project.rs: session_start_data now emits `startTime` (offset ISO, same base timestamp as the envelope) plus the rest of the observed 1.0.67 top-level set (contextTier, alreadyInUse, remoteSteerable) and richer context (hostType/repositoryHost/baseCommit) — matching the real session.start so the loader's one-field-at-a-time check doesn't keep rejecting. - docs: writing-compatible.md req 5 [observed, 1.0.67] with the verbatim error. Verified: projected session.start data keys now = {sessionId, version, producer, copilotVersion, startTime, contextTier, context, alreadyInUse, remoteSteerable}. Crate 59 + matrix + resume tests green. * fix(copilot): stamp turnId on turn-scoped events Fifth live `copilot --resume` rejection (line 6): "missing field `turnId`". Copilot requires a `turnId` on assistant-turn-scoped events; the projector omitted it. - project.rs: push_assistant takes a per-assistant-turn index ("0","1",…) and stamps `turnId` on assistant.turn_start/.message/.turn_end and every tool.execution_start/_complete (and subagent.*). session.start/user.message correctly carry none. - docs: writing-compatible.md req 6 [observed, 1.0.67]. The error advanced from line 1 → line 6 as earlier envelope reqs were fixed; this clears the turn events. Crate 59 + matrix + resume tests green. * fix(copilot): messageId on assistant msg/turn_end; non-empty toolCallId; verified in 1.0.67 Two more live `copilot --resume` requirements + the verified round-trip. - messageId (req 7): assistant.message and assistant.turn_end must carry a `messageId` (UUID); turn_end references the message it closes. - toolCallId non-empty (req 8): tool.execution_* and the toolRequests mirror need a NON-EMPTY toolCallId (empty reads as "missing field"). Forward treats empty-string ids as absent; projector synthesizes a stable id when empty. Verified: drove `copilot --resume` locally (isolated COPILOT_HOME + copied config.json) — a projected real session LOADS and resumes (full context). Flipped docs (crate README / CLAUDE.md / CHANGELOG / writing-compatible.md) to verified-in-1.0.67 with the 8-requirement loader contract (verbatim rejections). Crate 60 + matrix + resume tests green; clippy clean. * fix(copilot): subagent.* loader fields — real Pathbase sub-agent session resumes The user's failing doc was a Pathbase graph (resumed by URL) containing sub-agents. `copilot --resume` rejected `subagent.started` line-by-line for missing fields. Self-drove the loop locally (isolated COPILOT_HOME + copied config.json, real `copilot --resume -p`) and cleared them all: - subagent.started / subagent.completed now carry `toolCallId` (synthesized, stable per delegation), `agentName` + `agentDisplayName` (from the delegation id), and `agentDescription` — in addition to the existing `id`/`prompt`/ `result`/`turnId`. Verified end to end: the exact 5817-event Pathbase graph now LOADS and resumes in real copilot 1.0.67 (full context, ↑99.6k tokens). writing-compatible.md req 9 added; CLAUDE.md / CHANGELOG updated (contract now 9 requirements, verified on small + large sub-agent sessions). Crate 60 + matrix + resume tests green; clippy clean. * feat(copilot): project file edits to native edit/create shape so diffs render The user reported edit tool calls not displaying a diff in a resumed session. Referencing a real native copilot `edit`/`create` tool call: Copilot renders the change from `result.detailedContent` (a git-style unified diff), on tools named `edit` (args {path, old_str, new_str}) and `create` (args {path, file_text}). - native_name(FileWrite) now returns `edit`/`create` (was edit_file/create_file). - CopilotProjector detects FileWrite tool calls and re-emits them in the native shape: reshapes args (Claude Edit/Write -> old_str/new_str/file_text), emits a result with a summary `content` + a git-style `detailedContent` diff, and always emits the complete (the diff is the point) even when the IR had none. - docs: file-fidelity.md "Reverse (projection): making an edit render"; CHANGELOG. Verified: the Pathbase graph's edits project to toolName=edit with a `diff --git …` detailedContent, and the session still loads in real copilot 1.0.67. Crate 61 + matrix + resume tests green; clippy clean. * feat(copilot): add toolTelemetry to file edits so diffs render colorized Follow-up: the projected edit showed a flat (uncolorized) diff vs the native edit's colorized one. The native `edit`/`create` complete carries a `toolTelemetry` block whose `codeBlocks` (stringified JSON with fileExt + languageId + line counts) tells Copilot's UI to render a colorized diff view; `metrics.linesAdded/linesRemoved` give the +N/-M summary. Mine omitted it. - file_write_projection now builds `toolTelemetry` (properties/metrics/ restrictedProperties) matching the observed 1.0.67 shape: languageId derived from the path extension, line counts from the diff. - docs: file-fidelity.md; test asserts the telemetry. Verified: projected edit complete now carries toolTelemetry.codeBlocks with the right languageId + line counts, and the session still loads in copilot 1.0.67. Crate 61 + clippy clean. * fix(copilot): build edit diffs with a single header so they colorize The projected edit diff still showed flat (uncolorized). Root cause: the `detailedContent` had a duplicate/empty header — `toolpath_convo::unified_diff` prepends its own `--- a/<path>` / `+++ b/<path>` on top of `similar`'s empty-filename `--- `/`+++ ` header, producing a malformed diff Copilot can't parse into a colorized view. - git_diff now builds the diff with `similar` directly (single header, correct `a/dev/null` for creates). Added `similar` dep to toolpath-copilot. - test asserts exactly one file header (no stray `--- `/`+++ ` lines). - docs: file-fidelity.md. Verified: projected edit detailedContent is now a clean single-header `diff --git …` and the session still loads in copilot 1.0.67. Crate 61 + clippy clean. * fix(copilot): never emit hunkless diffs — root cause of the "flat diff" Diagnosed by capturing Copilot's actual TUI rendering through a pty harness (answering its terminal queries, auto-accepting folder trust, ctrl+o to expand the timeline) and comparing a resumed NATIVE session against the same session re-projected through our pipeline, byte for byte: - Our diffs were already driving the colorized machinery (green/red +N/-M counts, colorized hunk rows) — q7/ske/Y7 in the app bundle all accepted them. - The "flat diff text" was ONE case: a hunkless diff. Creating an EMPTY file produces a ""→"" text diff with headers but no @@ hunk; Copilot's diff view renders parsed hunk rows and hides headers, but with nothing to parse it dumps `diff --git`/`index`/mode lines as raw text. Fixes: - git_diff: an empty-file create now diffs to one added empty line (@@ -1,0 +1,1 @@ / +), matching the native tool's rendering (`+1`). - FileWrite.detailed is now Option: any other hunkless diff omits detailedContent entirely instead of leaking headers. - docs: file-fidelity.md — hunk requirement [observed via pty capture, 1.0.68] + note that the TUI recomputes +N/-M from the diff (arg-diff vs file-state nuance explains +1 vs +1 -1). Verified via pty capture of the re-projected session: 0 raw header lines, `Create foo.txt +1` / `Edit foo.txt +1` colorized, expanded body renders the colorized `1 +` rows. Crate 61 + matrix + resume green; clippy clean. * fix(copilot): remap tool args in the toolRequests mirror — colorized diffs render The colorized diff still didn't render for Claude-origin docs. Reproduced with the user's exact Pathbase graph (sliced to its first edit, projected, and captured through the pty TUI harness): the row rendered as generic "● edit" with file_path/old_string args and a flat markdown diff. Root cause: Copilot's timeline UI builds tool rows from the assistant.message.toolRequests MIRROR, not tool.execution_start. The mirror still carried raw IR args (Claude's file_path/old_string), so arguments.path was absent and the editor-family row (title "Edit <path>", +N/−M counts, Y7 colorized diff body) never engaged — the generic row markdown-rendered the diff as flat text ("basic syntax highlighting"). - New projected_tool(): one (name, arguments) remap per tool call, used by BOTH the mirror and tool.execution_start (they must agree). File writes → edit/create; file reads → view with path + view_range from offset/limit. - Regression test: mirror carries remapped args (path present, file_path absent; view_range mapped) and execution_start agrees. - docs: file-fidelity.md — the mirror is what the UI renders from. Verified via pty capture on the user's doc: colorized line-numbered hunk rows (1 -, 1 +, 3 +) render; zero raw-header/generic-arg leaks. Crate 62 + matrix + resume green; clippy clean. * docs(copilot): sync format docs with the rendering/verification learnings Audit found five gaps between the docs and what the last debugging arc established; close them: - project.rs module doc: stale "resume unverified" -> verified (1.0.67/1.0.68, incl. sub-agent session), pointing at the two contract docs. - events.md: new "Native tool vocabulary" section (bash/view/edit/create arg shapes + result strings, observed); the assistant.message row now warns that the resumed-timeline UI renders tool rows from the toolRequests MIRROR, so a writer must keep the mirror in native vocabulary. - known-gaps-and-sourcing.md: new "Verification methodology" section (the isolated-COPILOT_HOME loader loop + the pty TUI-capture technique with terminal-query replies, trust auto-accept, ctrl+o, and bundle grepping); loader contract + rendering contract + tool vocabulary moved from open questions to resolved. - writing-compatible.md: "loading is necessary but not sufficient" pointer to the rendering contract. - folder README: revision 2026-07-02; grounding now spans 1.0.67 capture + 1.0.67-1.0.68 live loader/rendering verification. * test(copilot): S-tier verification — real elicit fixture, un-dodged matrix, live probe Replaces synthetic coverage with real-capture-driven tests. Ran the feature-elicit prompt through the live copilot CLI (isolated COPILOT_HOME) — all 9 tasks incl. a real sub-agent. Sanitized capture is the matrix fixture + crate tests/fixtures/real-session.jsonl. capture-elicit-fixtures.sh + feature-elicit.md learned copilot. Parser fixes surfaced by the capture (all tested): - session.shutdown real shape: tokenDetails.{...}.tokenCount + model-keyed modelMetrics (reverse-eng usage.inputTokens was wrong); output == Σ per-message outputTokens (verified). - subagent.* are thin markers sharing the task tool's toolCallId; delegations pair by it and the projector preserves the id (was synthesizing — broke idempotency and cross-harness survival). - Codex-grade file fidelity: edit/create completes embed the real file-state diff in result.detailedContent; forward upgrades FileMutation.raw_diff. - toolpath-pi: decode zero wire `input` as None (same absence rule as cache). Tests: real_fixture_roundtrip.rs (forward invariants, projection round-trip, wire serde value-identity on all 64 lines); matrix on the REAL fixture with tokens + sub-agent (all cells green); scripts/verify-copilot-live.sh (loader + context probe — resumed model answered session-specific facts correctly). Docs synced (events.md, file-fidelity.md, known-gaps, CLAUDE.md). Full workspace: 61 test binaries green, clippy clean. * fix(copilot): address review — add `p export copilot`, merge shutdown token totals, bump pi Review feedback on #117 (benbaarber): 1. `path p export copilot` was nonexistent (the projector was reachable only via `path resume`). Added the ExportTarget::Copilot subcommand mirroring the others: --project writes a resume-ready session under ~/.copilot/session-state/<id>/ (+ session-store row); --output / stdout emit the projected events.jsonl without touching ~/.copilot. Factored build_copilot_session out of project_copilot; integration tests for help + output round-trip. (Also dropped the stale "unverified" doc comment.) 2. Session token total dropped input/cache when a shutdown existed: per-message usage only carries `output`, so `summed.or(shutdown)` discarded the shutdown's input+cache (~222k tokens in the real fixture). Now merges — output from the per-message sum, input/cache from the shutdown. New unit test (shutdown_fills_input_cache_when_per_message_output_present) + the real-fixture test now asserts input (54) / cache_read (198884) / cache_write (22998). 3. Bumped toolpath-pi 0.6.0 -> 0.6.1 for the zero-input-decodes-as-None behavior change (Cargo.toml + workspace dep + crates.json + CHANGELOG). Also corrected stale CLAUDE.md claims my earlier S-tier edit silently missed (wrong test counts; "no committed real session"; subagent.*/session.shutdown listed as unverified; the old token-total description). Workspace: 61 test binaries green, clippy clean.
`path query` loads every step in the local cache into one JSON array and
transforms it with an in-process jaq (pure-Rust jq) filter — selection,
projection, ranking, grouping, and top-N are all jaq. Each array element
wraps a Toolpath step (step/change/meta verbatim) with `cache_id`, `path`
(the parent path's id/base/meta), and `dead_end` (off the head's ancestry).
Scope flags choose which docs load: `--source`/`--id`/`--input` (file
selection) and `--project`/`--kind` (content scoping, semver-prefix kind
match). Output mirrors jq: pretty on a TTY, compact when piped (`-c`), raw
strings with `-r`. Malformed cache docs are skipped with a stderr warning;
empty results exit 0, filter errors exit 1.
`path kind` is the cold-start companion: lists the bundled kind specs, or
prints a kind's schema.json (the field reference for writing filters).
`--kind` and `path kind` share one semver-prefix selector, sourced from a
new shared `kinds` module that `schema.rs` now also reads.
BREAKING (pre-1.0): the former `path query` subcommands change. `ancestors`
moves to `path p query ancestors`; `dead-ends`/`filter` become jaq forms
(`map(select(.dead_end))`, `map(select(.step.actor | startswith("agent:")))`).
path-cli 0.14.0 -> 0.15.0; adds jaq-core/jaq-std/jaq-json.
…ache `path query` no longer loads every cached step into one array up front. The executor reads the filter — it parses the jaq into jaq's own AST and picks an execution plan: - **PerFileStream** — element-wise filters (`.[] | g`, distributive for any g) run per document and print as they go; nothing accumulates. - **Decompose** — algebraic aggregations run the filter per file, concatenate the per-file outputs, then run a derived combine over them: `map` → `add`, top-N `sort_by(k) | .[:N]` → `add | sort_by(k) | .[:N]`, `length`/`add` → `add`, `min`/`max` → themselves. A global top-N is a subset of the per-file top-Ns, so the merge is exact and bounded (files × N). - **Slurp** — the always-correct fallback for anything not provably decomposable (`group_by`, `unique`, `as`-bindings, `reduce`, …). Still lean: values are held once, with no whole-cache byte buffer. Recognition is conservative (a non-distributive prefix such as `unique` before a top-N slurps), so the planner never changes an answer. Unit tests assert the streamed output equals the slurp output byte-for-byte across element-wise, top-N, and scalar-reduction filters; integration tests confirm cross-file top-N and sums end to end. `TOOLPATH_QUERY_EXPLAIN=1` prints the chosen plan to stderr. No user-facing flag — it's automatic.
Addresses a code review of the streaming-planner commit: eight cases where the "planner never changes an answer" contract was violated, or an explicitly named input failed silently. - plan.rs: `is_slice_upto` decomposes `.[:N]` only for a literal nonnegative integer N. `.[:-1]` / `.[:length-1]` truncate per file before the merge, so they slurp (finding 1). - plan.rs: drop `min`/`max` from scalar reductions — `[] | min == null` from an empty (or fully scoped-out) document poisons the merge; they slurp (finding 2). - plan.rs: parse-guard the derived reduce — a source-span-recovered combine that doesn't reparse (e.g. a parenthesized `sort_by`) falls back to slurp instead of emitting a broken filter (finding 4). - filter.rs: a Decompose over zero contributing documents runs the main filter over `[]` (matches slurp: `length`→0, top-N→[]) instead of reducing over `[]` (which gave null / "cannot use null as array") (finding 3). - kinds.rs: a present-but-unparseable version segment (`/vgarbage`, `/v1.x`) no longer fails open to "any version" — the selector matches nothing and `path kind` reports no bundled spec (finding 5). - mod.rs: an explicit `--input`/`--id` that won't read/parse, or a `--id` that matches no cached document, is now an error, not a skip-with-warning that returns a wrong answer. Warn-and-skip stays for the whole-cache scan. Stdin (`--input -`) accepts JSONL too (findings 6, 7). The slurp-equality tests now assert the chosen plan is non-slurp, so they can't silently vacate into slurp-vs-slurp (finding 8), plus a regression per finding. Also: BufWriter on the streaming output path, and a test tying the bundled kind URIs to the `toolpath` constants.
…oc fixes Addresses the second review of PR #114 (8 findings). - plan.rs: scalar `add` no longer decomposes — float addition is not associative, so summing per-file partials could change the answer (files [[1e100], [-1e100, 1]]: decomposed 0.0, true sum 1.0). Only `length` (exact integer counts) keeps a scalar combine; the `map`/top-N `add` combines are array concatenation and unaffected (finding 1). - toolpath-cli: bump the out-of-workspace shim to 0.15.0 and its path-cli pin to 0.15.0 so it resolves again; crates.json entry updated (finding 2). - mod.rs: `--id X --source Y` records X as seen before the source filter, so a non-Y document yields an empty intersection instead of a false "no cached document with id" error (finding 3). - mod.rs: `--input` cache_id is the path as given, not `file_stem()`, so same-basename inputs keep distinct identity triples (finding 4). - plan.rs/CHANGELOG/CLAUDE.md: docs no longer claim `min`/`max` (or scalar `add`) decompose; the code comment explains why each slurps (finding 5). - README.md/site/pages/cli.md: CLI reference block shows the real `query` and `kind` surface and `p query ancestors`, not the removed subcommands (finding 6). - mod.rs: stdin double-parse failure reports both the JSON and JSONL errors instead of masking the JSON one (finding 7). - comment cleanup per repo style: drop the Cargo.toml dep rationale and past-state narration in cmd_p_query.rs and test files (finding 8). Below-the-cut: `TOOLPATH_QUERY_EXPLAIN=0` disables explain; `path p` help lists `query`; BUNDLED_KINDS uses the `toolpath::v1` kind-URI constants.
Bare `path query` (or scope flags with no filter) used to default to `.` and dump the entire cache to the terminal. The filter is now a required positional; clap prints usage on omission, and `path query .` says it explicitly.
feat(cli): add `path query` (jaq over the cache) and `path kind`
* chore: make 'just check' green — lint fixes + format sweep
- clippy: replace a redundant closure in cmd_export.rs with the function itself
- shellcheck: use ${HOME} in verify-copilot-live.sh's error message (SC2088)
and find instead of ls for the projected session id (SC2012)
- rustfmt + prettier sweep: GitHub CI only runs test+clippy, so formatting
drifted across merged PRs; this restores 'cargo fmt --check' cleanliness
* ci: run format + shellcheck gates on every PR
GitHub CI previously ran only cargo test + clippy, so rustfmt/prettier/
shellcheck drift merged unchecked (this branch's sweep is the cleanup).
Adds a lint job running scripts/quality_gates.sh format shellcheck, and
pins prettier (3.9.5) in site/package.json so the gate can't flap when
a new prettier version changes formatting rules.
* ci: run the full 'just ci' gate suite as the single CI job
Replaces the bespoke test+clippy steps (and the interim lint job) with
one job running scripts/quality_gates.sh --verbose — the same seven
gates as 'just ci' — so the workflow can't drift from the justfile.
Also disables SC2317 file-wide in quality_gates.sh: the gate_* functions
are dispatched indirectly, which older shellcheck (ubuntu-latest) flags
as SC2317 where newer flags SC2329; only the latter was suppressed,
which is why the first CI run of the shellcheck gate failed.
A path's step ids must be unique, but sources reuse ids across distinct records — Claude Code reuses `uuid` on `attachment` lines, so two unrelated events arrive with the same id — and carrying a duplicate through breaks any consumer that keys on the id (e.g. a store with a UNIQUE (path_id, step_id) constraint). `derive_path` now resolves collisions inline as each step is emitted (via `push_step`): a byte-identical re-emission is dropped, and a same-id-but-different step is re-IDed to `<id>#<n>`. Because resolution happens during emission and updates `turn_to_step`, parent references follow the rename — a later step whose parent matched a renamed duplicate resolves to the renamed step, not the first occurrence. Bumps toolpath-convo 0.11.0 -> 0.11.1.
…fixtures
Document how each agent harness records context compaction, correcting
claims that were based on synthetic/outdated fixtures (verified against
source and freshly captured real sessions):
- claude-code: full compactMetadata shape (preservedSegment/
preservedMessages, postTokens, durationMs) + the duplicate-UUID
re-emission known issue
- codex: real `compacted` payload is {message, replacement_history},
not the synthetic {trigger, preTokens, summary}; trigger is
analytics-only and never persisted
- gemini: compresses in-memory but persists nothing (was "no compaction")
- pi: Compaction entry fields; fromHook is extension-vs-default, not
auto-vs-manual
- opencode: stays one session, contiguous tail_start_id, no id reuse
- README: cross-harness comparison; manual == auto record everywhere,
only the trigger's visibility differs
Capture script: add a compaction second pass per harness (claude
`/compact`, codex tiny `model_context_window`, pi raised
`reserveTokens`, opencode `summarize` route) writing convo-compacted.*,
plus auto-delete of scratch sessions after capture (KEEP_SESSIONS=1 to
opt out, SKIP_COMPACTION=1 to skip). Includes the captured fixtures.
A conversation can carry the same turn id twice — Claude re-emits a block of earlier messages with their original uuids just before a compaction boundary — which produced duplicate step.id values and broke any store with a (path, step_id) primary key (e.g. Pathbase's bulk COPY ingest). derive_path now drops later duplicates, keeping the first occurrence: it carries the true parent lineage, while replayed copies are re-parented into a synthetic linear chain. Parent/head references by id resolve to the kept step, so no remapping is needed. Adds a regression test.
Replace the separate `turns: Vec<Turn>` and `events: Vec<ConversationEvent>` fields with a single ordered `items: Vec<Item>`, where `Item = Turn | Event | Compaction`. Holding the conversation as one ordered stream lets derive_path <-> extract_conversation round-trip losslessly and gives compaction boundaries (populated in a later phase) a true position. Adds the Compaction / KeptRange / CompactionTrigger types (defined, not yet emitted). Reads go through new turns()/events()/compactions() iterator accessors; turns_since now returns Vec<&Turn>. All five providers' to_view build `items` (all turns, then events — preserving the prior layout); gemini and codex disambiguate reused turn ids so the keep-first uniqueness pass doesn't silently drop turns.
derive_path now makes a single ordered pass over ConversationView.items, emitting a `conversation.compact` step for each Item::Compaction at its true position between the turns it separates. The step carries trigger / summary / pre_tokens / kept (each only when present) and resolves its parent through the turn map; later turns that reference the boundary rewire onto it, so the DAG threads through the compaction. extract_conversation reconstructs the Compaction from the step. Synthetic turn/event step ids are byte-identical to the old two-loop layout (per-variant counters), so every provider round-trip passes unchanged. Providers don't emit compactions yet (next phase); this wires the core derive/extract path with unit + round-trip tests.
Each provider's view builder now detects its compaction marker and emits
an Item::Compaction at its true position in the stream, mapped per
docs/agents/formats:
- claude: compact_boundary -> Compaction (trigger from compactMetadata,
pre_tokens=preTokens, kept from preservedSegment head/tail); the
isCompactSummary entry is folded into summary, not emitted as a turn.
- codex: `compacted` rollout item -> Compaction (summary=payload.message,
trigger/pre_tokens=None, kept empty); synthetic fixture updated to the
real {message, replacement_history} payload shape.
- opencode: `compaction` part -> Compaction (auto bool -> trigger,
tail_start_id -> kept range); fixed tail_start_id to deserialize the
camelCase `tailStartID` wire key.
- pi: Compaction entry -> Item::Compaction (summary, pre_tokens=tokensBefore,
trigger=None since fromHook is extension-vs-default, not auto-vs-manual)
replacing the old synthetic System turn.
Per-provider round-trip tests assert the mapping against the real
test-fixtures/<harness>/convo-compacted.* captures and that derive ->
extract preserves each Compaction. gemini unchanged (no compaction on disk).
Bump PATH_KIND_AGENT_CODING_SESSION to .../v1.1.0 and ship the new kind spec + bundled schema documenting the conversation.compact step type (optional trigger / summary / pre_tokens / kept). v1.0.0 stays registered and documented for backward compatibility; the base schema treats meta.kind as a free-form URI, so paths tagged either version validate. Updates the path-cli kind-schema registry, the site kind pages + registry index, and the RFC / CLAUDE.md kind references.
Minor bumps for the crates touched by the items/compaction work, with workspace deps and site/_data/crates.json kept consistent, plus a CHANGELOG entry: toolpath 0.6.0->0.7.0, toolpath-convo 0.10.0->0.11.0, toolpath-claude 0.11.0->0.12.0, toolpath-codex 0.5.0->0.6.0, toolpath-gemini 0.5.0->0.6.0, toolpath-opencode 0.4.0->0.5.0, toolpath-pi 0.5.0->0.6.0, path-cli 0.13.0->0.14.0
The provider projectors (view -> harness on-disk format, used by `path resume` / `path export`) walked turns() and dropped compaction boundaries. They now iterate `view.items` and reconstruct each harness's marker at its true position -- the inverse of the forward mapping: - claude: compact_boundary entry (+ isCompactSummary summary) with reconstructed compactMetadata (trigger / preTokens / preservedSegment). - codex: `compacted` rollout line (payload.message = summary). - opencode: `compaction` part (auto from trigger, tailStartID from kept) plus the synthetic summary message when present. - pi: Entry::Compaction (summary, tokensBefore, firstKeptEntryId from kept), replacing the now-dead turn-extra reconstruction path. Each is verified by a projection round-trip against the real convo-compacted fixtures (view -> project -> re-read -> same Item::Compaction). Forward derive/extract and reverse projection are now both lossless for compaction.
…them Replaces the keep-first dedup (7f05b83): silently truncating a path's steps is surprising, undefined behavior nobody expects. derive_path now returns Result<Path> and fails with ConvoError::DuplicateStepId when two steps would share an id; path-cli surfaces it as a clean error rather than producing a quietly-wrong (empty/truncated) upload. Producing unique ids is the provider's job: gemini and codex already disambiguate their format-reused ids, so they derive fine. A Claude compaction replay (which re-emits earlier messages with their original uuids) now errors loudly -- e.g. `path share` of such a session prints "duplicate step id <uuid>: ..." instead of uploading an empty path. Cascades the fallible signature through the five provider derive wrappers and all path-cli call sites (errors propagate via `?` to anyhow). Also runs rustfmt across the items/compaction work.
…fidelity Reworks compaction so it survives a round-trip through toolpath into a different harness and renders natively there. The portable payload is the summary plus the KEPT SET -- which prior turns survive verbatim into the post-compaction window. - Compaction.kept is now Vec<String> (surviving turn-ids), replacing the contiguous KeptRange (deleted): Claude's kept set is non-contiguous -- a preserved tail PLUS a scattered set of pinned tool results. - Claude forward strips the re-emitted replay block (duplicate-uuid entries before the boundary; Claude records no marker for them, so we detect duplicates) and records kept = preservedMessages ∪ replayed. The real a813677e session now derives cleanly (2805 steps, 0 dup ids). - Each projector renders the kept set in its own form: Claude re-emits the kept turns on-chain before the boundary; opencode/Pi anchor a tail at the earliest kept id; Codex keeps none (wholesale). Verified end-to-end: a Claude compaction projected into Codex emits a native `compacted` rollout line, and into Pi a native `compaction` entry (firstKeptEntryId + tokensBefore).
Claude stamps harness-injected assistant entries (API errors, rate-limit notices) with model "<synthetic>". actor_for_turn passed it straight through as agent:<synthetic>, whose angle brackets violate the actor-id pattern, so real derived sessions failed `path validate`. Attribute these to the harness (tool:claude-code) like System turns instead. The a813677e session now validates.
Folds compaction into the existing A -> IR -> B -> IR -> B translation matrix rather than a bespoke round-trip test. run_cell gains a compaction_survives invariant (boundary count + summary presence survive the A -> B leg), guarded by a persists_compaction() capability so gemini is exempt -- it compresses context in memory but never writes a boundary to the chat file, so there's nothing on disk to round-trip. matrix_translation_compacted runs the full invariant set over each harness's convo-compacted fixture, so the boundary is checked alongside turns, text, tools, and tokens.
A resume test confirmed Claude rebuilds context from the summary plus post-boundary turns only -- everything before the boundary's parentUuid: null is unreachable -- so the re-logged replay block is dead weight on resume. And `kept` already round-trips through compactMetadata.preservedMessages, so the replay is redundant for the derive<->project loop too. Drop it, along with the now-unused turn_to_entry and turn_index plumbing. Projected sessions shrink; resume is unaffected.
main added the toolpath-cursor crate while this branch was changing the shared toolpath-convo API; migrate cursor onto it: - ConversationView.turns/events -> items: Vec<Item> (provider builds Item::Turn; project/derive read via the turns() accessor). - derive_path -> Result<Path> (errors on duplicate step ids); propagate through the cursor path-cli call sites. - bump toolpath-cursor 0.1.0 -> 0.2.0 for the breaking signature change. In the cross-harness matrix, cursor is exempt from compaction survival. Renamed the capability persists_compaction -> roundtrips_compaction: the exemption is about our pipeline (the cursor provider doesn't derive or render compaction yet), not a claim about the format -- Cursor appears to persist summarization, so it's a gap to revisit. Also normalizes rustfmt drift in the recently-added sources.
Cursor compacts (/summarize + auto + Composer self-summarization) and writes a capabilityType:22 boundary marker bubble, but the summary text and kept set live server-side -- not in the local store. Verified against a live /summarize'd session: no latestConversationSummary field, the composer's conversationState protobuf holds only system prompt + tool/skill definitions, and the speculativeSummarizationEncryptionKey payload isn't stored locally. So there's nothing reconstructable to derive; like gemini, we model no compaction for Cursor. The provider recognizes the cap22 marker (Bubble::is_summarization) and skips it -- no turn, no compaction -- rather than surface a content-less boundary or an empty turn. Full finding documented in docs/agents/formats/cursor.md.
The compaction branch was rebased onto main's per-message token-usage work (#106). This makes the merged result build and pass: - update test code for the merged API: Turn's new group_id/ attributed_token_usage fields, ConversationView.items + turns()/ events() accessors, and derive_path's Result return (toolpath-convo, -claude, -codex, -opencode) - fix token-usage double-counting the cross-harness round-trip matrix exposed once compaction fixtures were added: - canonicalize message totals per group_id across the whole turn sequence, not per consecutive run (toolpath-claude) - group consecutive same-id Gemini lines (one split message) so the repeated tokens snapshot counts once; un-fold on the reverse path (toolpath-gemini) - advance Codex's cumulative token_count by a group total once, on the group's last turn (toolpath-codex) - correct the CHANGELOG entry: kind agent-coding-session v1.2.0 and the final crate version list
…on-global one The reader computed one session-level summary and stamped it onto every compaction, so a session with multiple boundaries collapsed all summaries onto the first. Track a pending-boundary cursor and attach each summary message to the boundary awaiting it. Tighten the cross-harness matrix to compare summary text (not mere presence) so this can't regress.
pi drops trigger, defaults pre_tokens to 0, and never leaves kept empty (mandatory tokensBefore / firstKeptEntryId). opencode's trigger is a bool (only auto vs not-auto survives), and each boundary now carries its own summary rather than a session-global one.
The "known limitation" note claimed the compact_boundary marker was dropped on read and isCompactSummary unrecognized — both untrue since the boundary became a first-class Item::Compaction (with the summary folded in). Replace it with what the test now covers, pointing at compaction_view.rs for the boundary assertions.
…oring derive_path is infallible (returns Path, not Result). On a step-id collision it drops a byte-identical re-emission and re-IDs a same-id-but-different step to `<id>#<n>`, so the result is always collision-free without surfacing an error to the caller. Removes ConvoError::DuplicateStepId. The per-provider derive::derive_path / derive_project wrappers (and pi's derive_graph) are infallible too — only the disk-reading entry points (e.g. pi's derive_project) still return Result. toolpath-git's own derive_path does real fallible I/O and is unchanged.
Each fix is backed by a regression test that fails when the fix is reverted. - toolpath-cli: bump the path-cli dep pin 0.14.0 -> 0.15.0. The old `^0.14.0` requirement excluded the actual 0.15.0, so `cargo install toolpath-cli` and the Tier-4 release publish failed to resolve. Invisible to `cargo build --workspace` because the shim is excluded from the workspace. - toolpath-codex: re-emit `reasoning_output_tokens` on projection (and accumulate breakdowns into the running cumulative) so the output->reasoning breakdown survives an IR->Codex->IR round-trip. - toolpath-codex: keep otherwise-empty turns that carry token accounting, so a group-final empty turn that finalize_usage stamped with the group total isn't dropped by the keep-mask (which made Sigma token_usage < session total). - toolpath-opencode: give the compaction summary message a strictly-later timestamp than the boundary message, so the reader's `ORDER BY time_created, id` can't sort the summary first and drop it. - toolpath-opencode: rewrite the kept-tail anchor to the re-minted message id so it resolves on re-read instead of collapsing `kept` to []. Proven by a new real-SQLite wire round-trip test (project -> .db -> read back). - toolpath-convo: derive `head` from `steps.last()` so a dropped final duplicate can't orphan a real step (e.g. a conversation.compact) as a dead end; add the `last_step_id` fallback for compaction parents.
Events and compaction boundaries were chaining backward onto the prior step but nothing chained forward onto them, so with head = last step they fell off the head's ancestry and `query dead-ends` / `render dot` flagged them as abandoned. Splice them into the linear parent chain (only when a step would otherwise chain onto the immediately-preceding turn) so they land on the ancestry, while genuine branches stay dead ends.
derive_path's splice_onto_intervening re-parents turns and compactions through intervening event steps so events land on the head's ancestry. That's right for the DAG but wrong on the wire: extract_conversation copied the spliced parent into Turn.parent_id verbatim, so projectors wrote parent chains through ids that don't exist in the source format — every Claude session with a headerless preamble line (ai-title, last-prompt, file-history-snapshot) exported its first message with parentUuid = "claude-preamble-N" instead of null, and a resumed session would chain through entries Claude never chains through. Providers never build a view in which a turn or compaction parents on an event (Claude even rewrites tool-result wire parents onto the owning assistant turn at read time), so an event step in a turn's parents can only be the splice. extract_conversation now walks past event-derived steps to the nearest turn/compaction ancestor when rebuilding Turn.parent_id and Compaction.parent_id. Events keep their spliced parents — event-to-event chains are legitimate wire data — and derive re-splices on the way back in, so derive -> extract -> derive is stable.
Main's toolpath-copilot (#117) was written against the removed turns/events fields. Build the ordered items stream in to_view (events carry a turn watermark so the real interleaving survives), read through the turns() accessor in the projector, and mark the harness roundtrips_compaction() = false in the cross-harness matrix — Copilot's events.jsonl has no verified compaction encoding, so foreign Item::Compaction is dropped on projection like gemini/cursor.
Main's #114 replaced path-cli's KIND_SCHEMAS const with the BUNDLED_KINDS registry, keying the v1.1.0 entry on PATH_KIND_AGENT_CODING_SESSION — which this branch repoints at v1.2.0. Give v1.1.0 its own constant (PATH_KIND_AGENT_CODING_SESSION_V1_1_0), register v1.2.0 in BUNDLED_KINDS, and update the newest-kind expectations in the kind/query/schema tests. Also restore the compaction changelog entry the rebase dropped and refresh CLAUDE.md's kind-constant note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records context-compaction as first-class provenance: when an agent summarizes and drops earlier context, we capture the boundary, what was kept, and why — instead of silently losing it.
What changed
ConversationView.turns/eventscollapse into one ordereditems: Vec<Item>stream (turns()/events()/compactions()accessors). NewCompaction/CompactionTriggertypes.conversation.compact, placed between the turns it separates so thehead-ancestry walk crosses it in order.agent-coding-session): extends main's v1.1.0 (token usage) withconversation.compact; v1.0.0/v1.1.0 schemas retained.derive_pathresolves duplicate step ids as it emits steps — a byte-identical re-emission is dropped, a same-id-but-different step is re-IDed to<id>#<n>— so it stays infallible (returnsPath) and always yields a collision-free path. The per-providerderive::derive_path/derive_projectwrappers shedResulttoo. Subsumes fix(derive): guarantee unique step IDs per path #111.Per-harness coverage
Claude's post-boundary re-emission (the replay block it re-logs before a boundary) is stripped on read and folded into the boundary's
keptset. On projection the replay block is not re-emitted — a resume test confirmed it's dead weight (Claude rebuilds context from the summary plus post-boundary turns) — andkeptrides incompactMetadata.preservedMessagesinstead, so re-reading reconstructs the same boundary.Tested
group_idcanonicalization across the whole sequence; Gemini split-message grouping; Codex group-once accounting).--all-targetsclean; full suite green.Rebased onto main's token-usage work (#106). Format references under
docs/agents/formats/updated.Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.