Cross-harness context-compaction provenance (kind v1.2.0)

.github/workflows/deploy-site.yml

@@ -6,7 +6,7 @@ on:
   pull_request:
 
 jobs:
-  test:
+  ci:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -24,14 +24,21 @@ jobs:
           key: cargo-test-${{ hashFiles('Cargo.lock') }}
           restore-keys: cargo-test-
 
-      - name: Test
-        run: cargo test --workspace
+      - uses: pnpm/action-setup@v4
+        with:
+          version: 10
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: pnpm
+          cache-dependency-path: site/pnpm-lock.yaml
 
-      - name: Clippy
-        run: cargo clippy --workspace -- -D warnings
+      - name: Quality gates # same gates as `just ci`
+        run: scripts/quality_gates.sh --verbose
 
   deploy:
-    needs: test
+    needs: ci
     runs-on: ubuntu-latest
     permissions:
       contents: read

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## Extract: undo the event splice when rebuilding wire parents — 2026-07-07
+
+- **`toolpath-convo`**: `derive_path` splices events and compactions onto the
+  head's ancestry, which re-parents neighboring steps through event steps —
+  ids that don't exist on the wire (Claude's headerless preamble/snapshot
+  lines carry no `uuid`, so nothing can chain through them).
+  `extract_conversation` now resolves a turn's or compaction's parent past
+  event-derived steps back to the nearest turn/compaction ancestor, so
+  projectors write valid wire chains: a re-exported Claude session's first
+  message keeps `parentUuid: null` instead of naming a synthesized
+  `claude-preamble-N` step, and messages after a snapshot line chain onto
+  the real prior message. Events keep their spliced parents (event-to-event
+  chains are legitimate wire data), and derive re-splices on re-derive, so
+  derive → extract → derive is stable.
+
 ## Derive: resolve duplicate step ids — 2026-07-01
 
 - **`toolpath-convo`** (0.11.1): `derive_path` now guarantees the derived
@@ -12,6 +27,191 @@ All notable changes to the Toolpath workspace are documented here.
   `UNIQUE (path_id, step_id)` constraint). Collisions are resolved before the
   path is returned: a byte-identical re-emission is dropped, and a
   same-id-but-different step is re-IDed to `<id>#<n>` so no data is lost.
+## New provider: GitHub Copilot CLI (preview) — 2026-06-30
+
+Adds **`toolpath-copilot` 0.1.0**, a forward provider that derives Toolpath
+documents from GitHub Copilot CLI (`@github/copilot`) sessions under
+`~/.copilot/session-state/<id>/events.jsonl`.
+
+- Parses the `events.jsonl` stream (`session.*`, `user.message`,
+  `assistant.*`, `tool.execution_*`, `subagent.*`, `skill.*`, `hook.*`,
+  `abort`) into `toolpath_convo::ConversationView` and hands off to the shared
+  `derive_path`. Tool names are classified into the `ToolCategory` ontology;
+  file writes whose args carry full content get a `raw` unified-diff
+  perspective. Tool `start`/`complete` pairing uses a correlation id when
+  present and falls back to positional pairing when absent (the id field is
+  unverified), so an id-less stream doesn't double-count tool calls.
+- Reads git context (root / repository / branch / revision) from the sibling
+  `workspace.yaml` into `Path.base`, via a tolerant key-scan parser (no YAML
+  dependency — the file's schema is itself reverse-engineered).
+- **Preview / schema partly verified.** Copilot CLI's `events.jsonl` format is
+  undocumented. First built from docs + reverse-engineering, then **verified
+  against a first-hand capture at `copilotVersion` 1.0.67** — which corrected
+  several guessed field locations (cwd/git under `session.start.context`, tool
+  results under `result.content`, `copilotVersion`, `reasoningText`,
+  `outputTokens`) and confirmed the envelope + core event types. Types not seen
+  in that session (`subagent.*`/`skill`/`hook`/`abort`/`shutdown`/compaction) and
+  the `checkpoints/` format remain unverified; the parser stays tolerant.
+- Adds the on-disk format reference at `docs/agents/formats/copilot-cli/`
+  (folder, every claim confidence-tagged) and its verification checklist.
+- Wired into the CLI **both directions**: forward (`path p import/list/show
+  copilot`, `path share`) and reverse via a new `CopilotProjector`
+  (`path p export copilot`, `path resume`). Resume/export write
+  `~/.copilot/session-state/<id>/` + a `session-store.db` `sessions` row
+  (INSERTing only a fresh id, never touching existing sessions). **✅ Verified in
+  copilot 1.0.67:** a projected session loads and resumes in the real
+  `copilot --resume`; the loader's writer contract (UUID `id`/`parentId`,
+  offset-ISO timestamps, `turnId`, `messageId`, non-empty `toolCallId`, full
+  `session.start` shape) is documented with verbatim errors in
+  `docs/agents/formats/copilot-cli/writing-compatible.md`. Also validated by the
+  cross-harness conformance matrix (`test-fixtures/copilot/`) + a round-trip test.
+- Bumps **`path-cli` to 0.15.0** (new `toolpath-copilot` dependency, the new
+  subcommands, Copilot in `path share`/`path resume`, and `uuid` `v4`).
+- The Copilot **resume loader contract** was mapped from live `copilot --resume`
+  runs and codified in `docs/agents/formats/copilot-cli/writing-compatible.md`
+  (9 requirements, each with its verbatim rejection): UUID `id`; offset-ISO
+  `timestamp` on every event; present `parentId` (UUID/null); `session-store.db`
+  row; full `session.start` shape incl. `startTime`; `turnId` and `messageId` on
+  turn-scoped events; non-empty `toolCallId`; and `subagent.*` fields
+  (`toolCallId`/`agentName`/`agentDisplayName`/`agentDescription`). Verified on a
+  27-event session **and** a 5817-event session with sub-agents.
+- **S-tier verification**: a real feature-elicit session (shell, create/edit/
+  view, glob+grep, errored read, real sub-agent, reasoning, per-message +
+  shutdown tokens) captured live at 1.0.68 now drives the cross-harness matrix
+  (`test-fixtures/copilot/convo.jsonl` — tokens and sub-agent included, no
+  dodges) and new crate tests (`real_fixture_roundtrip.rs`: forward invariants,
+  projection round-trip fidelity, wire-level serde value-identity). The capture
+  corrected the `session.shutdown` parser (`tokenDetails.{…}.tokenCount`;
+  model-keyed `modelMetrics`), resolved sub-agent semantics (`subagent.*` are
+  thin markers sharing the `task` tool's `toolCallId` — delegations pair by it,
+  ids preserved through projection), and upgraded file fidelity to
+  **Codex-grade** (native `edit`/`create` embed the real file-state diff in
+  `result.detailedContent`; the forward path now consumes it). Live loop
+  committed as `scripts/verify-copilot-live.sh` (isolated-home loader check +
+  resumed-model context probe — the probe answers session-specific questions
+  correctly); `scripts/capture-elicit-fixtures.sh` learned copilot.
+- File edits now **render as diffs** in a resumed session: `FileWrite` tool calls
+  project to Copilot's native `edit`/`create` shape with a git-style unified diff
+  in `result.detailedContent` (mapping a Claude `Edit`/`Write` accordingly). See
+  `docs/agents/formats/copilot-cli/file-fidelity.md`.
+- Adds the missing **`path p export copilot`** plumbing command (the projector
+  was previously reachable only via `path resume`): `--project <dir>` writes a
+  resume-ready session under `~/.copilot/session-state/<id>/`, `--output <file>`
+  or stdout emit the projected `events.jsonl`.
+- Session token total now **merges `session.shutdown` totals** even when
+  per-message `outputTokens` are present: Copilot only reports `output`
+  per-message, so the session's input + cache totals (≈222k in the real
+  fixture) live on the shutdown and were being dropped. Output comes from the
+  per-message sum; input/cache from the shutdown.
+- **`toolpath-pi` 0.6.1**: decode a zero wire `input` as `None` (the wire can't
+  express "unknown", so a zero written by a foreign-source projection must not
+  round-trip to `Some(0)` — same absence rule already applied to its cache
+  fields).
+
+## `path query` — query the local cache with jaq — 2026-06-29
+
+Adds `path query`, a porcelain that loads every step in the local cache
+(`~/.toolpath/documents/`) into a single JSON array and transforms it with
+an in-process jaq (pure-Rust jq) filter. Selection, projection, ranking,
+grouping, and top-N are all just jaq, so one command answers "find every
+turn that mentions `RefCell`", "which sessions touched `cmd_resume.rs`?",
+"the 10 steps that cost the most tokens", and so on.
+
+- **`path query [scope flags] '<jq filter>'`** — each array element is a
+  Toolpath step (`step`/`change`/`meta` verbatim) wrapped with `cache_id`,
+  `path` (the parent path's `id`/`base`/`meta`), and `dead_end` (whether the
+  step is off the head's ancestry). The filter is required, as in jq (`.`
+  emits the array unchanged); running one is the same as piping that array
+  to `jq`. Output mirrors jq:
+  pretty on a TTY, compact when piped (`-c` forces compact), and `-r`
+  prints string results unquoted (pipe a column of ids/paths into another
+  command, or read a turn's text/diff unescaped).
+- **Scope flags.** File selection (before parse): `--source <name>`
+  (cache-id prefix, e.g. `claude`/`git`), `--id <cache-id>` (repeatable),
+  `--input <file>` (`-` for stdin, repeatable). Content scoping (per path):
+  `--project <path>` (matches a `file://` base) and `--kind <selector>`
+  (semver-prefix match, e.g. `agent-coding-session/v1`). Everything else is
+  a jaq predicate on the real structure.
+- **`path kind`** — the cold-start companion: lists the kinds the binary
+  bundles a spec for; `path kind <kind>` prints that kind's bundled
+  `schema.json` (the per-field type + semantics reference for writing
+  filters). A trailing `/<version>` pins a version with the same prefix
+  rule as `--kind`.
+- **Streaming executor (no flag).** So the whole cache needn't sit in memory,
+  the executor *reads the filter*: it parses the jaq into its AST and, when it
+  can prove the shape decomposable, runs per document with a bounded merge —
+  element-wise `.[] | …` filters stream one doc at a time, and algebraic
+  aggregations split into a per-file partial + combine (`map(…)`, top-N
+  `sort_by(k) | .[:N]`, `length`). A global top-N is a subset of the per-file
+  top-Ns, so the answer is identical. Anything not provably decomposable falls
+  back to the whole-array path — including scalar `add` (float sums
+  re-associate), `min`/`max` (empty partitions yield null), `group_by`, and
+  `unique` —
+  which is still lean (values held once, no whole-cache byte buffer). The
+  planner never changes an answer — validated by tests asserting streamed
+  output equals slurp byte-for-byte. `TOOLPATH_QUERY_EXPLAIN=1` prints the
+  chosen strategy to stderr.
+
+**Breaking** (pre-1.0): the former `path query` subcommands change.
+`ancestors` moves to `path p query ancestors`; `dead-ends` and `filter`
+are gone in favor of jaq forms — `path query 'map(select(.dead_end))'` and
+`path query 'map(select(.step.actor | startswith("agent:")))'`. `path-cli`
+0.14.0 → 0.15.0; adds the `jaq-core`/`jaq-std`/`jaq-json` dependencies.
+
+## Conversation items IR + compaction provenance — 2026-06-22
+
+`ConversationView` now exposes a single ordered `items` stream (the new
+`Item` enum) in place of the separate `turns`/`events` lists, giving every
+provider one timeline to populate and every consumer one timeline to walk.
+On top of that, compaction is now first-class provenance: when an agent
+summarizes and drops earlier context, we record what was kept and why
+instead of silently losing the boundary.
+
+- **`toolpath-convo`** (BREAKING): `ConversationView.turns`/`events` are
+  unified into `ConversationView.items: Vec<Item>`, with `turns()`,
+  `events()`, and `compactions()` accessors for walking the stream. New
+  public types `Compaction` and `CompactionTrigger` model a
+  context-compaction boundary — `kept` is the set of prior turn ids that
+  survived, and `trigger` records what caused it. `derive_path` resolves
+  step-id collisions as it emits steps — a byte-identical re-emission is
+  dropped, a same-id-but-different step is re-IDed to `<id>#<n>` — so it
+  is infallible (returns `Path`, not `Result`) and the result is always
+  collision-free. The per-provider `derive::derive_path` / `derive_project`
+  (and pi's `derive_graph`) wrappers are likewise infallible now; only the
+  disk-reading entry points (e.g. pi's `derive_project`) still return
+  `Result`.
+- **New step type**: `conversation.compact` records a compaction event as
+  a Step in the derived `Path`.
+- **Per-provider compaction population**: `toolpath-claude`,
+  `toolpath-codex`, `toolpath-opencode`, and `toolpath-pi` emit
+  `Compaction` items from their on-disk compaction signals.
+  `toolpath-gemini` and `toolpath-cursor` participate in the items IR but
+  persist no compaction: Gemini records none, and Cursor's `/summarize`
+  writes only a boundary marker (the summary and kept set live server-side
+  and are unrecoverable from local data), so the marker is skipped on read.
+- **`agent-coding-session` kind → v1.2.0**: the kind constant is now
+  `https://toolpath.net/kinds/agent-coding-session/v1.2.0`, extending
+  v1.1.0 (token usage) with the `conversation.compact` step type. The
+  v1.2.0 schema is bundled in both `toolpath` and `path-cli` alongside the
+  retained v1.0.0/v1.1.0 schemas; producers (the shared
+  `toolpath_convo::derive_path` and every provider built on it) emit the
+  new URI.
+- **Token-usage idempotency fixes** (surfaced by the cross-harness
+  round-trip matrix once compaction fixtures were added): message totals
+  are canonicalized per `group_id` across the whole turn sequence rather
+  than per consecutive run, so a group interrupted by an intervening turn
+  no longer double-counts on re-read (`toolpath-claude`); consecutive
+  same-id Gemini lines (one split message) now share a `group_id` so their
+  repeated `tokens` snapshot is counted once (`toolpath-gemini`); and
+  Codex's cumulative `token_count` advances by a group's total once, on the
+  group's last turn (`toolpath-codex`).
+
+Crates bumped: `toolpath` 0.8.0, `toolpath-convo` 0.12.0,
+`toolpath-claude` 0.13.0, `toolpath-codex` 0.7.0, `toolpath-gemini`
+0.7.0, `toolpath-opencode` 0.6.0, `toolpath-pi` 0.7.0,
+`toolpath-cursor` 0.3.0, `path-cli` 0.15.0, `toolpath-cli` 0.15.0.
+Unchanged: `toolpath-git`, `toolpath-github`, `toolpath-dot`,
+`toolpath-md`, `pathbase-client`.
 
 ## Token usage: once per message, with per-step attribution + kind v1.1.0 — 2026-06-17