From fb5b1fe055d34337fd7e878cf8c8e5584c6cc4b3 Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 10:56:09 -0700 Subject: [PATCH] docs: plan hot path optimizations --- .../260722-digest-guarded-coordinate-reads.md | 79 ----- .../260723-doctor-maintenance-hot-paths.md | 334 ++++++++++++++++++ dev/plans/260723-everyday-query-hot-paths.md | 295 ++++++++++++++++ dev/plans/260723-indexing-hot-paths.md | 265 ++++++++++++++ .../260723-verified-bounded-session-reads.md | 205 +++++++++++ dev/plans/README.md | 25 +- 6 files changed, 1117 insertions(+), 86 deletions(-) delete mode 100644 dev/plans/260722-digest-guarded-coordinate-reads.md create mode 100644 dev/plans/260723-doctor-maintenance-hot-paths.md create mode 100644 dev/plans/260723-everyday-query-hot-paths.md create mode 100644 dev/plans/260723-indexing-hot-paths.md create mode 100644 dev/plans/260723-verified-bounded-session-reads.md diff --git a/dev/plans/260722-digest-guarded-coordinate-reads.md b/dev/plans/260722-digest-guarded-coordinate-reads.md deleted file mode 100644 index a2363a4..0000000 --- a/dev/plans/260722-digest-guarded-coordinate-reads.md +++ /dev/null @@ -1,79 +0,0 @@ -# Add digest-guarded coordinate reads - -## Goal - -Let callers hydrate a manifest-selected session only when the retained canonical -public document still has the expected digest. Add an optional guard to existing -`show` and `export` reads so a valid mismatch fails before entry bounds are -resolved or transcript evidence is returned. Existing unguarded behavior and -successful human, JSON, and JSONL output remain unchanged. - -The guard is content-addressed rather than an archive or lease: identity and the -complete public-document digest must match, while current capture attribution, -freshness, source state, adapter version, and root resolution may differ. A -mismatch never retrieves an older body or automatically retries against the new -revision. - -## Changes - -1. `src/domain/public-session-document.ts` and `src/cli/program.ts` — admit one - canonical lowercase SHA-256 digest value for the optional - `--expected-document-digest` flag on both `show` and `export`. Convert it to - the existing `SessionDocumentDigest` value with the fixed - `sha256-sessions-document-jcs-v1` scheme before library inspection. Malformed - input is usage failure with exit `2`; the flag is orthogonal to all current - show/export selection modes. -2. `src/application/show-session.ts`, `src/application/export-session.ts`, and a - focused shared application helper — accept an optional expected document - digest, reuse the existing single fully verified `getSession` result, and - compare the returned summary digest before resolving actual entry bounds or - selecting presentation output. Missing identity remains `session-not-found`; - a valid mismatch becomes a distinct `document-digest-mismatch` operational - failure with exit `1`, empty stdout, and no expected or current digest in the - diagnostic. Mismatch takes precedence over an out-of-document coordinate. -3. `src/application/library-error.ts` and the CLI composition in - `src/bin/sessions.ts` — carry the guarded input through the provider-neutral - application boundary and render the fixed sanitized mismatch message. Do not - add a machine-readable error envelope or change successful schema-1 records. -4. Keep `SessionIndexReader.getSession` and the SQLite schema unchanged. The - current `SqliteReadSnapshot` already returns one immutable operation or - discards it on concurrent file change, while `getSession` reconstructs the - complete canonical document and verifies stored/computed digests and metrics. - This milestone preserves that proof and makes no cheaper-physical-read claim. -5. Update `docs/project-intent.md`, `docs/architecture-memo.md`, - `docs/getting-started.md`, `docs/contributing/architecture.md`, `docs/privacy.md`, - `docs/reference/cli-contract.md`, `docs/reference/structured-output.md`, and - the packaged Sessions skill evidence protocol/search guidance to distinguish - current guarded behavior from deferred historical storage and physical read - optimization. Replace manual post-hydration digest comparison with the guard - for manifest-driven reads. -6. Extend the digest parser, show/export application, CLI, one provider-neutral - workflow integration, and packaged-skill contract test seams. Prove matching - guarded output equals unguarded output; revision A fails after replacement by - B with no stdout; B returns the exact requested entry/range; mismatch - precedes coordinate absence; a valid guard on an absent identity remains - `session-not-found`; attribution-only changes do not invalidate an equal - document; and neither guarded command resolves a provider. Prove the guarded - missing-session precedence once through a parameterized show/export - application test rather than another integration layer. Rely on the - existing SQLite digest-corruption/concurrent-snapshot tests and closed - structured-output tests because those owners do not change. - -## Verify - -- Run focused digest parser, show/export application, CLI, provider-neutral - workflow, and skill contract tests covering the changed surfaces. -- Run `pnpm check`. -- If compiled-entrypoint wiring is not already covered by the focused workflow, - use an isolated generated library or the existing synthetic distribution smoke; - routine proof must not depend on private provider or contributor library state. - -## Boundaries - -- Do not add a new read command, structured-output schema, storage migration, - historical revision retention, manifest lease, batch hydration, or automatic - retry/re-key behavior. -- Do not expose current or expected digests in mismatch diagnostics, reopen a - provider, change presentation bounds, or weaken complete-document validation. -- A future partial-read optimization requires separate measurement and a design - that preserves the complete canonical digest proof. diff --git a/dev/plans/260723-doctor-maintenance-hot-paths.md b/dev/plans/260723-doctor-maintenance-hot-paths.md new file mode 100644 index 0000000..4755103 --- /dev/null +++ b/dev/plans/260723-doctor-maintenance-hot-paths.md @@ -0,0 +1,334 @@ +# Bound doctor and maintenance work without weakening integrity + +## Goal + +Make `doctor` and routine maintenance use bounded transient work while keeping +Sessions' exact, fail-closed evidence contracts. + +The first priority is doctor FTS verification. Today +`ftsProjectionSemanticContentIsValidReadOnly` builds a second complete FTS5 +projection from every retained canonical content value in memory, then compares +the complete expected and actual vocabularies. Its term ranges normally target +1,000,000 instances, but one term above that target is still compared as one +unbounded range. On a large retained library this duplicates a corpus-sized +search structure before useful progress is visible and is the most likely cause +of reports that doctor appears to hang. + +The second priority is orphan repair. It currently pages all content rows, +tests reachability row by row, and executes a fenced zero-deletion batch for +windows containing only referenced content. The desired query should page only +orphan candidates while preserving the transaction-time reachability recheck. + +Carrying certified cleanliness through a successful compact is a possible third +optimization, not accepted implementation. Compaction deliberately invalidates +the previous clean proof today. It advances only if provider-free measurement +shows that the next index's full validation is a material cost and a separate +ADR proves that compaction can carry, but never manufacture, cleanliness. + +Doctor remains an immutable, exact whole-library audit. Repair remains +uncertified maintenance. The program changes no provider behavior, query +result, public command grammar, structured output, exit code, or retention +policy. The first two delivery slices land as separate PRs. Compact +certification, if its gate passes, receives a new plan and PR. + +## Changes + +### 1. Establish the doctor measurement and work contract + +1. Add `scripts/measure-doctor.ts` and `measure:doctor` in `package.json`. + Generate provider-free libraries through the production SQLite writer and + run each measured cohort in a fresh child process so seeding and earlier + cohorts do not contaminate peak RSS. +2. Include small and large corpora, shared and unique text, zero-token text, + multibyte text, one content value above the byte interval target, and a + repeated term that crosses a reduced measurement-only instance target. Let + the production writer allocate content IDs; keep sparse and signed-ID + boundary coverage in low-level deterministic FTS tests rather than mutating a + measured production-seeded database. Never resolve an ordinary provider root + or copy contributor data. +3. Make a document-interval feasibility probe the first doctor deliverable. + Against equal generated databases, run actual + `fts5vocab(...,'instance')` queries over one, two, and many document-ID + intervals, capture normalized `EXPLAIN QUERY PLAN` output, and alternate + elapsed/RSS measurements. Node's current SQLite API has no stable statement + status or progress-handler seam, so do not promise VM-step or internal + virtual-table traversal counters. +4. Stop before the production refactor and documentation changes if the + interval query repeatedly scans the complete actual vocabulary or elapsed + work scales with corpus size multiplied by interval count. RSS and elapsed + scaling are evidence, not deterministic proof. Record a failed feasibility + result and preserve the current exact audit while a different bounded design + is prepared. +5. If the feasibility gate passes, add a private best-effort work observer in + `src/infrastructure/sqlite/fts-projection.ts`, composed from + `src/infrastructure/sqlite/sqlite-index-health.ts:inspectDatabaseHealth`. + Keep it out of the application health port and public doctor result. + Allowlisted aggregate fields are: + - canonical rows and UTF-8 bytes inspected; + - expected interval count and maximum interval rows/bytes; + - oversized-content interval count; + - term summaries and ordinary term ranges compared; + - oversized terms and coordinate ranges compared; + - actual/expected instances compared and maximum range size. +6. Observer failure must not affect health, cleanup, stdout, or the exit code. + It must never receive text, terms, content IDs, hashes, identities, paths, + timestamps, SQL text/errors, or lease values. +7. The script reports only generated corpus sizes, database bytes, normalized + aggregate plan facts, phase durations, child-process peak RSS, final health, + main-file/sidecar equality, and the allowlisted counters. It uses + mode-`0700` temporary roots, mode-`0600` owned files, cleans on success and + failure, and stays outside `pnpm check`. +8. Add a script-contract test that rejects private fields, proves child and + temporary-file cleanup after a forced failure, and requires healthy exact + results. Elapsed time and RSS are report-only; semantic equality, immutable + persistent state, bounded interval counters, and complete cleanup are hard + assertions. + +### 2. Verify expected FTS content in bounded exact intervals + +1. Refactor + `src/infrastructure/sqlite/fts-projection.ts:ftsProjectionSemanticContentIsValidReadOnly` + so it never retains one corpus-sized expected projection or a corpus-sized + interval list. Keep `PRAGMA temp_store = MEMORY` and fail closed unless + SQLite confirms memory-only TEMP storage. +2. Replace `loadExpectedDoctorProjectionInsideSavepoint` with interval-local + helpers such as `scanDoctorContentIntervals`, + `loadDoctorExpectedInterval`, and `dropDoctorExpectedProjection`. + Construct intervals by keyset-reading `content_id` and exact + `length(CAST(text AS BLOB))`, then load the corresponding canonical + `content_id,text` rows in signed-ID order. +3. Use fixed private admission targets of at most 512 canonical rows and exactly + `16 * 1024 * 1024` UTF-8 bytes per expected interval. A single canonical + value above the byte target is processed alone, counted as oversized, and + never truncated, skipped, split into invented canonical rows, or spilled to + disk. +4. For each interval: + - create a fresh contentless TEMP FTS5 table plus only the vocab tables + needed for that comparison; + - preserve the original signed row IDs; + - load inside a private savepoint that composes with an existing outer + transaction; + - compare docsize in both directions; + - compare exact term/document/column/offset instances in both directions; + - close iterators, roll back or release the savepoint, and drop every TEMP + object before advancing. +5. Cover actual document IDs exactly with half-open signed-ID intervals. The + first interval includes the actual prefix through its last canonical ID; + later intervals cover `(previousLastId,lastCanonicalId]`; after the final + interval, the actual docsize and vocabulary tail must be empty. This detects + extra actual documents before the first canonical row, between sparse + canonical IDs, and after the last row. An empty canonical table requires an + empty actual projection. +6. Derive each interval's actual and expected term summaries from the + `instance` vocabulary restricted by that interval's document bounds, grouped + by term with exact distinct-document and instance counts. The `doc` column of + the current `fts5vocab(...,'row')` table is a document count, not a document + ID; never filter or reuse whole-library row-vocabulary summaries as if it + were an interval coordinate. +7. The feasibility gate in slice 1 must already have shown that this exact + document-bounded access shape does not multiply whole-vocabulary work. + Bounded memory alone is not sufficient justification for accidental + superlinear CPU. +8. Preserve primary-error precedence. Load, comparison, observer, iterator, or + cleanup failure makes semantic health false; cleanup noise must not replace + the original failure. Repeated calls and calls inside the current immutable + snapshot must leave no TEMP or persistent artifact. + +### 3. Stream term ranges and partition an oversized term + +1. Replace + `src/infrastructure/sqlite/fts-projection.ts:matchingDoctorTermRanges` and + the retained `DoctorTermRange[]` with immediate, streaming comparisons. + Walk actual and expected term summaries in UTF-8 binary order. Require exact + term, document-count, and instance-count equality before comparing + positions. +2. Accumulate complete ordinary terms only until either side reaches the + existing 1,000,000-instance target, compare that range in both directions, + then discard its bounds. Never sample terms, documents, or positions and + never replace exact instance equality with aggregate counts or hashes. +3. When one term exceeds the target, partition its exact ordered coordinate + space `(doc,col,offset)`. Build each half-open upper boundary with bounded + lookahead from both actual and expected vocabularies and choose the earlier + target-th coordinate, so neither side can place more than the target in one + comparison. Compare actual-only and expected-only rows before advancing. +4. Cover the first coordinate, signed 64-bit document IDs, boundaries that fall + on different sides, and the final tail. Equal total counts are not proof: + shifted documents, columns, or offsets must still fail. +5. Extend `test/infrastructure/sqlite-fts-projection.test.ts` with: + - healthy equality across row/byte intervals, sparse IDs, zero-token rows, + multibyte terms, and one oversized canonical value; + - missing, extra, and malformed docsize in the first, middle, and final + interval; + - extra actual documents before, between, and after canonical IDs; + - wrong terms with equal token counts, changed positions, cross-document + swaps, and unequal counts under the same term universe; + - a healthy oversized term spanning several coordinate ranges; + - actual-only, expected-only, shifted-document, and shifted-offset damage on + both sides of boundaries; + - an adversarial distribution that would exceed the target if boundaries + came from only one side; + - invalid private limits, load/comparison/observer/drop failures, outer + transaction composition, and repeated invocation. +6. Extend `test/infrastructure/sqlite-index-health.test.ts` to require the same + public `ReadyIndexHealth` decisions for healthy, docsize-damaged, and + semantic-damaged libraries. Extend `test/doctor-no-persistence.test.ts` with + a worker stopped after its first interval; the parent must see unchanged + main-file identity/stat, no WAL/SHM, and no persistent TEMP artifact. +7. Reconcile the current-behavior descriptions in + `docs/reference/cli-contract.md`, `docs/privacy.md`, + `docs/contributing/storage.md`, `docs/contributing/architecture.md`, + `docs/contributing/testing.md`, and `docs/architecture-memo.md`. Describe + exact interval coverage, the one-oversized-content exception, coordinate + partitioning, and the memory-only TEMP rule. Do not make the private work + observer part of the public CLI contract. + +### 4. Page only orphan repair candidates + +1. Replace + `src/infrastructure/sqlite/sqlite-index-repair-orphans.ts:readContentWindow` + with `readOrphanCandidateWindow`. Use one keyset anti-join over + `sessions_content_values`, ordered by signed `content_id`, with + `NOT EXISTS` against `sessions_content_occurrences` and a fixed candidate + limit. Return `length(CAST(content.text AS BLOB))` as exact candidate bytes. + The first page has no cursor predicate; later pages use + `content_id > :cursor`. +2. The existing `scanLimit` becomes a private orphan-candidate page bound + named `candidateLimit`. Rename the composition option in + `src/infrastructure/sqlite/index-maintenance.ts:SqliteIndexMaintenanceOptions` + from `repairScanLimit` to `repairCandidateLimit` and update its + `repairOrphans` wiring and focused tests. Do not add a public limit, cursor, + or partial result. Use the existing + `sessions_content_occurrences_content_idx`; record `EXPLAIN QUERY PLAN` or a + stable statement-observer assertion without tying tests to elapsed time. +3. Apply the byte admission to orphan candidates only. If the next candidate + would exceed the byte target after at least one selection, advance only + through the last selected ID so the skipped candidate is seen on the next + page. Process one oversized candidate alone. +4. Keep `assertCandidateReady` inside the same leased immediate transaction + immediately before deletion. Preserve + `deleteUnreferencedContentCandidates`, exact returned/deleted-row equality, + `assertCandidateDeleted`, FTS-trigger deletion, renewable lease fencing, and + aggregate decimal row/byte totals. +5. Return exhausted immediately when the candidate query finds no row. Preserve + the initial checkpoint/fence and one checkpoint/fence after every non-empty + committed batch, but remove zero-deletion batch progress and checkpoints. A + healthy no-orphan library performs one candidate query and returns + `unchanged`. +6. Orphan repair remains uncertified. It must not initialize or advance + `sessions_index_generation_receipt`, mark the repair generation clean, or + publish a clean proof. The next index still selects full validation unless + independently eligible through an already accepted proof path. +7. Add a private synchronous test seam between candidate selection and the + readiness assertion so a focused test can make a selected candidate + referenced before deletion. The production composition supplies no hook. + The assertion remains authoritative; this seam must not enter the public + maintenance port or permit asynchronous work inside the immediate + transaction. +8. Extend + `test/infrastructure/index-maintenance-repair-orphans.test.ts` with a large + referenced-only library, sparse orphans separated by referenced regions, + row and byte bounds, one oversized candidate, signed-ID order, retained + shared evidence, a candidate that becomes referenced, missing FTS state, + later-batch failure/restart, raw worker exit, live refusal, expired takeover, + stale-owner fencing, and exact totals. Require the receipt row and sequence + to remain unchanged and no new clean seal/proof to be published. Any repair + that acquired a writer generation—unchanged, repaired, post-acquisition + failed, or crashed—makes the prior proof ineligible and sends the next index + through full validation. A refusal or failure before acquisition preserves + the prior generation and proof eligibility. +9. Update `docs/reference/cli-contract.md`, + `docs/contributing/storage.md`, `docs/contributing/architecture.md`, + `docs/contributing/testing.md`, and `docs/architecture-memo.md` to say repair + pages orphan candidates. Public no-limit/no-cursor behavior, exact totals, + restart semantics, and the post-repair doctor contract remain unchanged. + +### 5. Measure, then gate certified cleanliness through compact + +1. Do not implement this slice with bounded doctor or orphan paging. First + extend `scripts/measure-indexing.ts` with equal clean control and compacted + clones. Create reusable pages through supported index mutations before the + clean writer closes, then clone only after `readWriterCleanProof` confirms a + valid stat-bound proof. Never mutate a clone directly after clean close. + Record compact duration, reclaimed pages/bytes, next writer-open mode, + full-validation phase calls/durations, and canonical/query/health equality. + Require the untouched control's next index to select `fast`, current compact + to make its clone select full validation, and both to retain equal canonical, + FTS, capture, and query state. +2. Proceed only when accepted evidence shows post-compact full validation is a + material next-index cost and carrying an existing valid proof avoids that + work. If not, leave compact correctly uncertified and close the candidate. +3. If the gate passes, create a new executor plan and + `docs/decisions/0012-preserve-certified-cleanliness-through-compaction.md`. + The ADR must narrow the claim: + - compact may carry forward an already valid clean proof but cannot create + cleanliness from an unproven library; + - only supported SQLite compaction and coordination mutations are covered; + - crash recovery remains uncertified and direct same-user SQLite edits + remain outside the threat boundary; + - doctor remains the explicit semantic audit. +4. The later plan must consume the exact stat-, schema-, instance-, generation-, + lease-, and sidecar-bound proof before writer acquisition, publish a new + proof only after all compact batches, final checkpoint, exact-owner release, + close, hardening, and sidecar absence succeed, and make every uncertainty + fall back to full validation. It must not extend certified recovery receipts + to compact or accidentally certify `repair`, `forget`, `clear`, failure, or + crash. +5. Required later tests include absent/malformed/stale proof, stat and schema + mismatch, sidecars, no-op and page-reclaiming compact, every failure/crash + boundary, lease takeover, retained canonical/FTS/query equality, and proof + privacy. Stop if certification requires rescanning canonical or FTS content; + that would erase the intended benefit. + +## Live smoke protocol + +Run deterministic tests and generated measures first. A live doctor smoke is +read-only but still requires explicit authorization because it reads the +ordinary retained library. + +1. Require `sessions paths` state `ready`. Stop on recovery, migration, unsafe, + incompatible, or live-writer state; do not recover or index as part of the + smoke. +2. Run one doctor process at a time. Do not open a second SQLite connection + while its immutable snapshot is active. +3. Record the main database device/inode/mode/link count/size/mtime/ctime and + WAL/SHM absence privately before and after. +4. Redirect stdout/stderr into an owned mode-`0700` temporary directory, capture + OS peak RSS, and run one cold then two serial warm compiled doctor processes + with `SESSIONS_DOCTOR_TIMINGS=1`. +5. Report only health, allowlisted phase durations/counters, peak RSS, database + bytes, and persistent-file equality. Remove the temporary directory and + never print paths, identities, terms, hashes, source metadata, or text. +6. Never smoke destructive repair or compact against the ordinary retained + library under this plan. Use a disposable provider-free library containing + retained rows, sparse orphans, and reusable pages. Real repair or compact + needs separate explicit authorization after doctor identifies a need. + +## Verify + +- For bounded doctor, run the focused FTS projection, index health, + no-persistence, immutable snapshot, and measurement-contract tests, then + `pnpm measure:doctor`. +- For orphan paging, run the repair-orphans, writer-coordination, FTS repair, + clean-proof, and lifecycle tests. +- Require exact semantic/corruption parity, bounded work counters, cleanup, + crash/restart behavior, receipt/proof gates, and immutable persistent state. + Elapsed time and RSS are supporting evidence, not correctness gates. +- Run `pnpm check` before publishing each independently reviewable slice. + +## Boundaries and stop conditions + +- Do not sample evidence or replace exact bidirectional term/position equality + with counts, hashes, or FTS `integrity-check`. +- Do not allow SQLite TEMP state to spill transcript-derived data to disk. +- Do not persist doctor work counters or add them to normal doctor JSON/stderr. +- Do not add automatic doctor, repair, compact, recovery, or index execution. +- Do not add public repair paging, progress tokens, resume cursors, or partial + results; do not rebuild FTS from orphan repair. +- Do not certify repair, forget, clear, failed compact, or crashed compact. +- Do not claim secure erasure, partial-page repacking, or file shrink beyond + observed whole-page reclamation. +- Stop and redesign bounded doctor if interval cleanup cannot remain + memory-only, exact coverage weakens, or actual-vocabulary work becomes + superlinear in interval count. +- Stop compact-certification work if its measurement gate fails or exact proof + requires a new whole-library audit. diff --git a/dev/plans/260723-everyday-query-hot-paths.md b/dev/plans/260723-everyday-query-hot-paths.md new file mode 100644 index 0000000..886326f --- /dev/null +++ b/dev/plans/260723-everyday-query-hot-paths.md @@ -0,0 +1,295 @@ +# Optimize everyday query hot paths without changing evidence semantics + +## Goal + +Make routine `list`, `entries`, and `search` reads scale with the requested page +wherever their contracts permit it. Query execution should select the smallest +deterministically ordered page of compact coordinates, batch-hydrate and verify +only that page, and calculate exact query-wide evidence once when support or +lineage requires it. + +The current broad entry selector is the first priority. On an aggregate-only +live library with 4,544 sessions and 3.57 million entries, a 50-entry page took +roughly 4.5–6 seconds warm because SQLite scanned `sessions_entries` first and +built a temporary ordering B-tree before applying `LIMIT`. An equivalent +source→tracking→canonical→entry traversal used existing indexes, required no +temporary sort, and selected 51 coordinates in about 0.1 milliseconds warm. +The existing generated measure uses only five entries per session and does not +expose this skew. + +Search is the second priority. It currently scans overlapping FTS results for +aggregate support, distinct matching identities, and ranking, then performs +point reads for selected snippets, contexts, and summaries. Exact support and +ranking inspect the complete qualifying result by design, so broad search +cannot become page-only work; the target is one query-wide match computation +plus bounded page hydration, not approximate evidence. + +Every delivery slice below is independently reviewable and lands in order. +Acceptance requires deep-equal repository results, byte-identical structured +output except for the intentionally versioned opaque cursor token in slice 7, +unchanged cursor field presence and continuation results, unchanged capture +scope, support, ranking, roots, corruption handling, snapshot behavior, and +cursor failures, plus deterministic work-shape or repeatable aggregate +measurements. Machine time is never the sole correctness gate. + +## Changes + +### 1. Bound entry coordinate selection + +1. `src/infrastructure/sqlite/sqlite-session-entry-query.ts:readEntryRows` — + make source→tracking→canonical→entry traversal explicit with order-preserving + `CROSS JOIN` boundaries and the existing equality predicates. Preserve every + `entryInventoryWhere` filter, the filtered `first|last` subquery from + `selectionClause`, binary source/instance/native ordering, ascending entry + ordinal, `LIMIT + 1`, and cursor behavior. Keep hydration after page slicing + and use the existing unique and primary-key indexes; this slice adds no + migration or durable index. +2. `test/contracts/session-query.contract.ts` — traverse every entry page in + `all`, `first`, and `last` modes and compare the concatenated result with the + one-page canonical result. Cover source, instance, native ID, source state, + workspace, activity, entry time, actor, origin, kind, and tool filters; + binary-order identity edges; roots; content counts and previews; capture + scope; cursor mismatch; and cursor staleness. +3. `test/infrastructure/sqlite-session-entry-query.test.ts` — isolate plan-shape + risks that the shared contract does not: selected versus unselected corrupt + rows, each filter family under the explicit traversal, and `first|last` + applying entry filters inside the per-session selection rather than after + ordinal selection. +4. `scripts/measure-entry-query.ts` — replace or supplement the uniform profile + with an owned on-disk corpus containing thousands of sessions, hundreds of + thousands of entries, and at least one session with tens of thousands of + entries. Use mostly textless entries plus bounded shared text so coordinate + work dominates. Measure broad and filtered `all`, `first`, `last`, origin, + tool, early-page, and deep-page cases after a warm-up; report seeding, + coordinate selection, full hydration, aggregate counts, and repeated + semantic equality separately. +5. Add a private, best-effort timing/work observer around `readEntryRows` and + `hydrateEntries`, accepted only by the infrastructure entry-query function + and composed only by the generated measure. Observer failure must not change + query behavior or public output. Factor the exact production coordinate SQL + into one focused internal builder so the measure can run + `EXPLAIN QUERY PLAN` against the statement that ships rather than a copied + query. +6. The measure records normalized plan facts: outer scan, indexes used, and + temporary ordering B-tree presence. These are diagnostics, not Vitest + assertions tied to one SQLite version. Promote the SQL change only when the + generated result is identical and the broad plan avoids the entry-first full + scan and full temporary sort on supported development runtimes. +7. Update `docs/contributing/entries.md` and + `docs/contributing/testing.md` with the bounded-coordinate design, generated + skew profile, structural proof, and report-only elapsed evidence. + +### 2. Measure planner statistics as a separate complement + +1. Add `scripts/measure-query-planner-statistics.ts` and an opt-in package + command. Create equal generated databases from the skewed corpus, apply + explicit `ANALYZE` and `PRAGMA optimize` variants only to experimental + clones, and compare plans, alternating warm elapsed times, statistics rows, + and database byte growth. +2. Cover broad and narrow entries after the explicit traversal, list identity + and activity filters, broad and selective `all|any` search, and manifest + selection. Record whether any benefit remains beyond the local query-shape + fix and how often statistics would need refresh as the corpus changes. +3. Do not invoke planner-statistics commands from readers, migrations, index, + doctor, or maintenance in this slice. If no repeatable benefit remains, if + selective queries regress, or if refresh/storage cost is material, close the + experiment without a runtime policy. If evidence justifies persistence, + create a separate plan covering certified writer mutation, receipt sequence, + recovery, refresh cadence, schema compatibility, and index-time cost. + +### 3. Batch selected search content and snippets + +1. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchContent` — + replace one execution per selected content ID with one page-bounded selected + content CTE. Deduplicate IDs, retain the explicit integer FTS rowid + restriction, and require exactly one canonical text, digest, and snippet per + requested ID. Preserve full-text hash validation and marker-collision + handling; if any selected text contains the candidate markers, retry the + complete selected page with the next deterministic candidate. Unselected + content must not influence marker selection. +2. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — + build the content map once and make snippet projection free of content point + reads. Leave context, summary/root work, and `readMatchedTerms` unchanged in + this slice; `any` mode is already bounded by selected coordinates and at most + 32 admitted terms. +3. Extend `test/infrastructure/sqlite-session-query.test.ts` with limits 1, 20, + and maximum; shared/repeated content IDs; marker collisions; multibyte text; + and empty bodies. Missing or duplicate selected content, FTS disagreement, + malformed text, and digest mismatch must still fail the whole read, while an + unselected corrupt row stays unhydrated. +4. Extend `scripts/measure-search-query.ts` with page-size profiles, + selected-content counts, statement counts, and alternating warm elapsed + time. Update `docs/contributing/search.md` and + `docs/contributing/architecture.md` to claim only batched selected-page + content/snippet hydration. + +### 4. Batch selected search context + +1. `src/infrastructure/sqlite/sqlite-query-context.ts` — replace per-hit + `readSearchContext` calls with a page operation keyed by each selected + `(sessionId, primaryOrdinal)`. One query reads direct tool-call/result + coordinates for all primaries and retains enough ordered candidates to prove + `linkedContextTruncated`. +2. Keep a separate per-primary membership map containing each selected physical + coordinate and its adjacent/linked flags. Deduplicate SQL hydration by + physical `(sessionId,entryOrdinal)`, then read those unique coordinates and + ordered text segments in fixed chunks below SQLite's minimum supported + bind-variable limit. Merge chunk results into the per-primary map + deterministically, preserving direct-link-only expansion, ordinal order, + linked caps, flags, and 512-byte UTF-8 body truncation. Query work is one + primary-coordinate query plus + `ceil(unique physical context coordinates / chunk limit)` hydration queries, + not an infeasible fixed two-query claim. +3. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — build + the context map once and remove per-hit context point reads, leaving the + content map from slice 3 and separately owned summary/root work unchanged. +4. Extend `test/infrastructure/sqlite-session-query.test.ts` with context 0 and + maximum at the maximum result limit; enough adjacent/linked coordinates to + cross several chunks; links in both directions; linked overflow; + adjacent/linked overlap; multibyte truncation; empty bodies; multiple hits in + one session; and corruption/failure in a later chunk. Require exact output + equality and all-or-nothing failure. +5. Extend the search measure with selected-context counts, expected/actual chunk + counts, and alternating warm elapsed time. Document chunked selected-page + context hydration without claiming query-wide bounded search work. + +### 5. Prototype one query-wide search match set + +1. In an isolated change, prototype one + `WITH matching_segments AS MATERIALIZED` statement derived from the current + literal FTS query, joins, and exact filters. It emits tagged result sections + for aggregate occurrences/distinct content, distinct matching identities + needed for root support, and the ranked `LIMIT + 1` coordinate page using the + current BM25, activity, binary identity, and ordinal order. It creates no + durable or transcript-bearing table and never derives query-wide support from + the visible page. +2. Factor only shared joins and filter parameters needed to compare baseline and + prototype; do not add a general SQL builder. Root resolution continues over + the complete matching identity set and page hydration uses slice 3. +3. Add a dedicated measure or extend `measure:search-query` to alternate + baseline and prototype over selective/broad terms, `all|any`, repeated + occurrences with low unique content, many identities and lineage states, + limits 1/20/maximum, and first/deep pages. Report qualifying FTS traversals, + match-set time, hydration time, total time, and peak RSS. +4. Run both strategies against the same generated fixtures in + `test/infrastructure/sqlite-session-query.test.ts` and + `test/contracts/session-query.contract.ts`; require deep equality of hits, + support, roots, snippets, matched terms, context, capture scope, and cursors. +5. Define “one qualifying FTS traversal” structurally as one normalized + `EXPLAIN QUERY PLAN` virtual-table `MATCH` scan and one execution of the + match-set statement; Node's SQLite API does not expose a reliable FTS row + traversal counter. Promote only if the shape is legal for FTS5/BM25, meets + that structural proof, repeatedly improves broad search by roughly 20% or + more, keeps selective regression within 10%, and keeps peak RSS growth within + 25% in the alternating generated measure. These are prototype decision + thresholds, not CI budgets. If the gate fails, remove the production + prototype and retain the exact multi-scan path plus the independently useful + page batching. + +### 6. Batch selected summaries + +1. `src/infrastructure/sqlite/sqlite-session-state.ts` — add a bounded + `readSessionSummariesBatch` keyed by canonical session ID and expected + identity. Bind only `(sessionId,kind,instanceId,nativeId)` for each distinct + selected session: at the 200-row public maximum this is 800 variables, below + SQLite's minimum supported limit. Join canonical, tracking, and source rows + once, validate one result per requested session, and restore page order in + JavaScript from the validated session-ID map rather than binding an input + ordinal. Reuse the existing freshness and retained summary decoders; reject + missing, duplicate, malformed, or identity-inconsistent rows as + `corrupt-data`. Refactor shared decoding so point and batch reads cannot + drift. +2. `src/infrastructure/sqlite/sqlite-session-query.ts:listSessions` — carry the + compact session ID through the ordered page and batch selected summaries + after slicing. Do not enlarge the pre-limit sort with summary payload. +3. `src/infrastructure/sqlite/sqlite-session-entry-query.ts:hydrateEntries` and + `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — + deduplicate selected session IDs and replace per-session summary cache misses + with one bounded batch. +4. Add corruption-parity tests and SQLite authorization/work-shape assertions + comparing page size 1 with maximum. The number of summary SELECT + authorizations must remain constant per page rather than grow with distinct + selected sessions. Extend entry/search measures with distinct-session + profiles. + +### 7. Add compatible keyset continuation for list and entries + +1. `src/infrastructure/sqlite/query-cursor.ts` — add cursor v2 while retaining + command, complete query fingerprint, random library instance, writer + generation, and cumulative next offset. Add bounded provider-neutral numeric + anchors: canonical `session_id` for list and + `(session_id,entry_ordinal)` for entries. Do not place raw source instance or + native IDs in the opaque base64url token. Preserve canonical encoding, the + 2,048-byte bound, allowlisted payload shape, usage-versus-stale failures, and + opacity. +2. Decode existing v1 cursors for all commands. New list/entries continuations + emit v2; search continues to emit and consume the existing offset-only v1 + format. A v1 list/entries page uses the current offset path once, then emits + v2 from its last returned row. New v2 list/entry pages use the numeric anchor + while retaining cumulative offset for the existing internal contract. +3. `src/infrastructure/sqlite/sqlite-session-entry-query.ts` — add a strict + anchor-resolution CTE that requires the numeric coordinate to exist and + qualify under the same entry filter and `first|last` selection, recovers its + binary source/instance/native order fields, and applies the lexicographic + after-anchor predicate. `src/infrastructure/sqlite/sqlite-session-query.ts` + does the same for list: coordinate selection carries `canonical.session_id` + and validated effective activity, and anchor resolution recovers null-last + activity plus the binary identity ties inside the same immutable snapshot. + Revision mismatch is `stale-cursor`; a structurally malformed v2 payload or + matching-revision anchor that is absent/non-qualifying is `invalid-cursor`; + an existing anchor row with malformed stored fields is `corrupt-data`. Emit + an anchor only when the extra row proves another page. +4. Extend `test/infrastructure/sqlite-query-primitives.test.ts` for v1/v2 + admission and failure cases. Extend the shared query contract to traverse + page size one across equal and null activity, binary Unicode identities, + every entry selection/filter family, a v1-to-v2 transition, recreated + libraries, and later writer generations. Concatenated keyset traversal must + equal canonical one-page results with no gaps or duplicates. +5. Extend generated measures with early and deep pages. The structural gate is + production v2 SQL with no `OFFSET`, an anchor-resolution CTE, a strict + after-anchor predicate, and normalized `EXPLAIN QUERY PLAN` evidence of + indexed continuation; deep-page elapsed time is supplemental. Update + architecture, entry, and search docs; the public syntax, cursor opacity, + mismatch rules, and stale rules remain unchanged. + +### 8. Measure and, only then, lazy-load CLI composition + +1. After slices 1, 3, 4, 6, and 7 land and the search prototype is closed, add + `scripts/measure-cli-startup.ts`. Against compiled `dist` and generated + temporary state, alternate bare Node, version, top-level help, command help, + and one provider-free read; report median/p95 aggregates without touching + contributor providers or the ordinary library. +2. If the compiled evidence is material, refactor `src/bin/sessions.ts` to use + type-only imports plus memoized dynamic imports inside the command callbacks + that need concrete providers, lifecycle, maintenance, or application + services. Keep provider kinds available for grammar/help and preserve the + composition-root rule that only `src/bin/` selects concrete adapters. +3. `test/cli.test.ts`, `pnpm smoke:dist`, and `pnpm smoke:package` must preserve + exact grammar, help, output, errors, signals, provider laziness, and packaged + module resolution. If callback-level imports do not materially reduce + packaged startup, stop without splitting `program.ts` or adding a custom + dispatcher. + +## Verify + +- Run the focused shared query contract, SQLite entry/search, cursor primitive, + CLI, and applicable measurement-contract tests after their owning slice. +- Run `pnpm measure:entry-query` and `pnpm measure:search-query`; run experimental + planner, search-materialization, and startup measures only in their slices. +- Run `pnpm typecheck`, `pnpm deps:check`, and `pnpm check` before publishing + each independently reviewable slice. + +## Boundaries + +- Do not change human, JSON, or JSONL fields, order, limits, byte bounds, exit + codes, or fail-before-output behavior. +- Do not approximate support, rank, matched terms, snippets, context, capture + scope, or lineage, and do not hydrate transcript text before page selection. +- Do not add a daemon, cross-process cache, package API, batch command, network, + telemetry, or provider-specific query behavior. +- Do not add durable indexes or migrations in the primary query slices. + Planner statistics remain experimental until separately justified. +- Do not fold full-document reads, manifest copying, doctor, or lineage-closure + work into this program. +- Do not place contributor paths, identities, transcripts, or private corpus + details in plans, fixtures, docs, or measurement output. diff --git a/dev/plans/260723-indexing-hot-paths.md b/dev/plans/260723-indexing-hot-paths.md new file mode 100644 index 0000000..8f84631 --- /dev/null +++ b/dev/plans/260723-indexing-hot-paths.md @@ -0,0 +1,265 @@ +# Reduce indexing work without weakening evidence guarantees + +## Goal + +Reduce corpus-sized provider discovery work and statement-per-content +replacement work while preserving the indexing contract: discovery remains +complete before absence is inferred; provider histories remain read-only; +source mutation proofs remain exact; candidate and failure order stays +deterministic; failed refreshes preserve last-good canonical state; and leases, +recovery receipts, canonical readback, document digest/metrics checks, and +affected canonical/FTS parity remain authoritative. + +Routine unchanged indexing is not the main target. The existing deterministic +2,000-session baseline already uses 16 freshness reads, 16 unchanged writes, +and zero stable transcript reads; a local generated run completed in roughly +53–59 milliseconds. Current structural opportunities are: + +- Cursor maps every materialized agent with `catalog.stores.find`, producing + O(agent × store) lookup work. +- Codex describes each independent rollout serially after capturing one state + snapshot. +- Canonical replacement resolves exact `(digest,text)` content once per text + occurrence, even when the same pair has already been resolved in the current + transaction. +- Obsolete-content deletion and affected FTS parity execute one statement per + content ID. + +Land phases 1–5 as ordered, independently reviewable changes. Phase 6 is an +evidence gate, not pre-authorized implementation: replacement transaction +batching, WAL tuning, and unchanged compare-and-set work require a separate +accepted plan only if the new measurements show one remains dominant. + +## Changes + +### 1. Add provider-free discovery and replacement measures + +1. Add `scripts/measure-index-discovery.ts` with independent generated Cursor + and Codex cohorts. It must not resolve a user provider or ordinary provider + path. + - Cursor mapping covers 2,000-agent/2,000-store and + 10,000-agent/10,000-store catalogs, forward/reverse store order, + all-present, half-missing, repeated agent IDs, and duplicate directory + names. Wrap the generated store array with a numeric-access counter so the + scaling contract is observable without a production diagnostic API. + - Codex covers 1, 8, 9, and 2,000 generated rollout descriptors in a random + temporary root, including ready, missing, and invalid descriptors. Record + serial production descriptor semantics, call count, result order, and + elapsed discovery as the baseline. Concurrency, scheduling, and selected + failure counters begin only after slice 3 introduces a controlled internal + seam. + - Emit only aggregate input/result counts, work counters, elapsed time, and a + semantic equality flag. Never print IDs, names, locators, paths, hashes, or + fixture text. Remove temporary roots in `finally`. +2. Add `scripts/measure-index-replacement.ts`. For every matrix row, create an + isolated temporary Sessions library and use the production lifecycle, + writer, and session-index port to start a run, seed shared content, apply + replacements, finish, observe aggregate WAL bytes before close, close + normally, and verify complete health and clean proof. +3. The replacement matrix covers: + + | Shape | Sessions | Text occurrences | Unique exact pairs | Purpose | + | ------------------- | -------: | ---------------: | -----------------: | -------------------------------- | + | One wide unique | 1 | 10,000 | 100% | Lookup/insert scaling | + | One wide mixed | 1 | 10,000 | 50% | Local reuse | + | One wide repetitive | 1 | 10,000 | 10% | Repeated-pair elimination | + | Many small | 128 | 256 | 100% | Per-replacement transaction cost | + | Many medium | 128 | 8,192 | 50% | Transaction versus row work | + | Shared/reintroduced | 2+ | 10,000 | mixed | Cleanup and sharing proof | + + Require exact canonical documents, public digests, metrics, occurrences, + representative search/entry results, run counts, receipt mutations, FTS, + and post-close health. Use one disposable warm-up library and a fresh, + independently but identically seeded production library for every timed + iteration. Do not copy a closed clean seed: the proof binds file identity and + stat. Assert equal schema, PRAGMA/security configuration, seed semantics, + writer-open mode, receipt/generation baseline, page size, and freelist shape + before timing. Report aggregate counts, unique/preexisting/missing pairs, + inserted/deleted rows, affected IDs, replacement calls, receipt mutations, + database/WAL bytes, and timing min/median/max. Keep collision, + duplicate-exact-row, FTS mismatch, and forced rollback as correctness tests + rather than timed rows. + +4. Add `measure:indexing:discovery` and `measure:indexing:replacement` to + `package.json`. Document both in `docs/contributing/testing.md` as generated, + provider-free, aggregate-output diagnostics outside `pnpm check`; distinguish + them from explicitly authorized live provider measures. Update + `docs/contributing/indexing.md` with discovery-, occurrence-, unique-content-, + and transaction-shaped work ownership. +5. Keep `scripts/measure-indexing.ts` unchanged as the stable, certified + recovery, full-validation, missing-reconciliation, query, health, run, and + clean-proof control. Add a small + `scripts/support/indexing-measurement-runner.ts` owner for temporary-root + creation, sanitized aggregate output, cleanup, and injected test failure. + Cover it in `test/indexing-measurement.test.ts` with a tiny generated cohort, + following only the sanitized spawn-failure pattern in + `test/content-storage-measurement.test.ts`. The test rejects private output + fields and proves cleanup without running the full performance matrix inside + `pnpm check`. + +### 2. Index Cursor agent stores once per catalog + +1. `src/adapters/cursor/discovery.ts:mapCursorDiscovery` — build one + `Map` with a single pass over each + admitted catalog before iterating materialized agents. On duplicate + `directoryName`, retain the first store and do not overwrite it, preserving + current `.find` semantics. Continue validating directory/main/WAL kinds and + using `claimedStores` by physical component key. +2. Preserve materialized-agent, candidate, and issue order; native-ID conflicts; + unknown-catalog handling; claimed-store outcomes; catalog snapshotting; + inventory passes; provider reads; fingerprints; and source-change detection. + This phase changes only the in-memory join from O(A×S) to O(A+S). +3. `test/adapters/cursor/discovery.test.ts` — cover 2,000 and 10,000 + agent/store pairs with numeric store reads equal to store count; reverse + order; duplicate names selecting the first store; repeated agents retaining + `claimed-agent-store`; missing/invalid stores; unknown catalogs; duplicate + native IDs; and exact binary candidate/issue order. +4. Update the discovery measure to require one store-array pass per catalog and + exact equality with the recorded reference mapping. Do not change adapter + version or public behavior. + +### 3. Describe Codex rollouts with ordered bounded concurrency + +1. Add `src/adapters/shared/ordered-concurrency.ts` by extracting the current + Cursor helper into a provider-neutral internal adapter owner. Inputs receive + original ordinals; active operations never exceed the caller limit; results + return in input order; no new work starts after a visible failure; + already-started work settles; and multiple failures select the lowest input + ordinal. The helper does not log, expose input values, or try to cancel an + in-progress filesystem call. +2. Move `src/adapters/cursor/inventory.ts` to the shared helper with its existing + limit of eight and remove the old Cursor-owned helper. Move its tests to + `test/adapters/shared/ordered-concurrency.test.ts`; cover empty, one, limit, + limit+1, and large inputs, inverted completion, peak activity, no later + scheduling, awaiting started work, and lowest-ordinal error selection. +3. `src/adapters/codex/source.ts` — add an internal + `describeCodexRolloutsInOrder(threads, describe)` seam and replace the serial + loop with the shared helper at a Codex-owned limit of eight. Production + passes `describeRollout`; tests and the generated measure pass controlled + descriptor operations. Each state thread receives exactly one descriptor + call; `freezeCodexSession` stays paired with that thread; the result and + generation `Map` retain state-thread order; and `currentGeneration` publishes + only after every descriptor succeeds and the discovery-sequence guard passes. +4. Keep complete state snapshotting before descriptors, successful discovery + over every state thread, read-time before/after rollout verification, and + per-generation freshness unchanged. Do not cache descriptors across commands + or generations and do not read transcript bodies during stable discovery. +5. `test/adapters/codex/source.test.ts` — cover 2,000 generated threads, exact + state order, one descriptor per thread, mixed descriptor states, inverted + completion, stable fingerprints, failure without partial generation + publication, prior-generation invalidation, and provider-file byte equality. + Slice 3 extends the discovery measure with controlled start/settle ordinals, + peak concurrency, and selected failure ordinal, and requires calls equal to + thread count, peak concurrency `min(8, count)`, serial semantic equality, and + deterministic lowest-ordinal failure choice. + +### 4. Resolve exact content once per replacement + +1. Add `src/infrastructure/sqlite/sqlite-content-batch.ts` with a private fixed + batch limit of 128. Three parameters per content pair remain below SQLite's + minimum supported bind-variable limit; cleanup and parity reuse the row + bound. +2. `src/infrastructure/sqlite/sqlite-session-document.ts:insertEntries` — + collect first-seen unique exact content pairs before occurrence insertion. + Use a nested map keyed first by validated digest and then exact binary text; + digest alone or an ambiguous concatenated key is never identity. Keep + occurrence-to-pair links in canonical order and retain no cache beyond the + current replacement call. +3. Resolve unique pairs in 128-row + `VALUES(input_ordinal,digest,text)` chunks joined to + `sessions_content_values` on digest and `text COLLATE BINARY`. Order matches + by input ordinal and `content_id`; more than one exact stored row is + `corrupt-data`. Insert each missing exact pair once with the existing guarded + `INSERT ... RETURNING`, keep signed SQLite IDs as `bigint`, then insert + entries and occurrences in canonical order using resolved IDs. +4. Preserve the schema, digest index, exact-duplicate guard, FTS triggers, + sharing across sessions, canonical readback, digest/metrics proof, one-session + transaction, lease, and receipt boundary. +5. `test/infrastructure/sqlite-session-index.test.ts` — cover 257 pairs across + chunks; 257 repetitions producing one content row and 257 occurrences; + unequal text under a forced digest collision; preexisting, missing, repeated, + shared, and reintroduced pairs; duplicate exact stored rows; and failure + during insertion. A failed insertion rolls back all new content and + occurrences, preserves last-good canonical state, leaves the failed + replacement receipt unadvanced, and records only the existing separate + `repository-write` mutation. +6. Update the replacement measure to require inserted content rows equal + missing unique exact pairs rather than occurrences. Update + `docs/contributing/storage.md` to describe transaction-local exact reuse, not + a durable cache. + +### 5. Batch cleanup and affected FTS parity + +1. `src/infrastructure/sqlite/sqlite-content-maintenance.ts:deleteUnreferencedContentCandidates` + — validate signed IDs, deduplicate in first-seen order, and replace one delete + per ID with one conditional delete per 128-ID `VALUES` chunk. Delete only + bound IDs that remain unreachable from all occurrences, use returned IDs or + exact checked change counts, and never scan/delete unrelated orphan content. + Keep the helper shared by replacement, forget, and repair. +2. `src/infrastructure/sqlite/fts-projection.ts:assertFtsProjectionContentParityForIds` + — validate and deduplicate IDs, then compare canonical and FTS docsize + presence once per chunk. Canonical-only and projection-only rows fail; + both-absent rows after cleanup remain valid. Full doctor content/semantic + checks remain unchanged. +3. Extend FTS and session-index tests across zero, negative, maximum signed, + duplicate, and more than 128 IDs; obsolete/shared/reintroduced/unrelated + content; and failure in a later chunk. Any cleanup or parity failure rolls + back the whole replacement, preserves last-good state, and does not advance + the failed transaction receipt. +4. Extend forget and repair tests to prove exact target deletion, shared-content + retention, payload/window bounds, deleted row/byte counts, lease assertions, + FTS triggers, and atomic rollback under the shared batch helper. +5. The replacement measure records affected IDs and expected cleanup/parity + chunks and requires full semantic equality and health. Document bounded + set-based cleanup/parity in `docs/contributing/storage.md`. + +### 6. Re-measure and gate later architecture + +1. After phases 1–5, run the three provider-free index measures and record + before/after work counts and aggregates in `docs/architecture-memo.md`, with + facts separated from hypotheses. Live Codex/Cursor measures remain optional, + supplemental, and require the existing provider-read flag and authority. +2. Replacement transaction batching advances only through a separate plan when + many-small/medium matrices show time dominated by replacement calls rather + than row work. That plan must resolve bounded batch size, interruption, + per-session rollback/replay and failure classification, result order, + committed crash boundaries, and exactly one receipt advance per newly + defined certified mutation. A whole-source transaction is prohibited. +3. WAL/checkpoint tuning advances only when WAL/checkpoint measurements remain a + dominant cost and a generated crash experiment proves equal canonical, FTS, + run, receipt, clean-proof, and recovery outcomes. Do not weaken + `synchronous`, foreign keys, secure-delete, defensive mode, lease fencing, + page reclamation, or sidecar recovery; consider only bounded connection-local + checkpoint scheduling first. +4. Unchanged conditional compare-and-set advances only when `unchangedWrite` + remains dominant after discovery work and a prototype can atomically require + exact source/native identity and last-good fingerprint while asserting full + input cardinality. Keep both the application freshness read and a + transaction-time TOCTOU guard; one stale/missing/mismatched row rolls back the + entire existing 128-item batch. +5. If none of these gates is met, mark all three deferred and end this program. + +## Verify + +- Run the focused Cursor discovery, shared concurrency, Codex source, SQLite + session-index/FTS, forget, repair, writer coordination, and lifecycle tests + after their owning phase. +- Run `pnpm measure:indexing`, `pnpm measure:indexing:discovery`, and + `pnpm measure:indexing:replacement`; elapsed values remain report-only while + exact work counts and semantic equality are required. +- Run `pnpm check` before publishing each independently reviewable phase. + +## Boundaries + +- Do not skip complete discovery, infer absence from an incomplete scan, cache a + provider generation across commands, or remove source mutation verification. +- Do not parallelize transcript normalization or changed replacements in phases + 1–5. +- Do not change adapter versions, fingerprints, canonical schema, public output, + or failure codes. +- Do not remove post-replacement reconstruction, document digest/metrics proof, + affected FTS parity, lease fencing, or certified receipt behavior. +- Do not persist transcript-derived caches or emit transcript text, identities, + locators, hashes, or paths from diagnostics. +- Do not change the existing replacement mutation boundary or tune WAL without + the separate evidence-gated plan required by phase 6. diff --git a/dev/plans/260723-verified-bounded-session-reads.md b/dev/plans/260723-verified-bounded-session-reads.md new file mode 100644 index 0000000..5043f54 --- /dev/null +++ b/dev/plans/260723-verified-bounded-session-reads.md @@ -0,0 +1,205 @@ +# Stream complete proof for bounded session reads + +## Goal + +Reduce peak memory and object construction for bounded `show` and `export` +without weakening the proof attached to a digest-guarded read. The current path +loads every relation, entry, segment, locator, metadata object, and text value +into one `SessionDocument`, validates it, projects the complete public document, +and recomputes its digest before selecting at most 50 default entries or one +explicit range of at most 200 entries. A supplemental aggregate-only live +measurement found that one 56,706-entry retained session took about 2.1 seconds +to return one entry. + +The new bounded path must still visit and validate the complete canonical row +sequence, recompute the existing `sha256-sessions-document-jcs-v1` digest, and +compare complete document metrics inside one immutable reader operation. With +nested entry/segment streaming, retained state is bounded by the selected entry +window, the bounded relation prefix, unique relation keys needed for duplicate +validation, and the largest decoded scalar or metadata object currently being +checked. Unselected segment text and objects are released individually. +Successful human, JSON, and JSONL output must remain byte-for-byte equal to the +current path. + +This is an allocation and memory optimization, not a sublinear-I/O claim. +Unguarded and guarded reads keep the same failure precedence: stored corruption +fails closed; an absent identity remains `session-not-found`; a valid expected +digest mismatch precedes entry-bound failure; and a matching digest permits the +existing coordinate result. + +## Changes + +1. `src/domain/public-session-document.ts` and a focused new + `src/domain/public-session-document-stream.ts` — add a closed, stateful public + document digest writer for the existing schema and digest scheme. It must + emit the exact RFC 8785 byte sequence produced by + `digestPublicSessionDocument(projectPublicSessionDocument(document))` while + consuming header fields, then nested `beginEntry`/`writeSegment`/`endEntry` + events, then relations. +2. The stream state follows RFC 8785 UTF-16 property-key order exactly: + optional `createdAt`, `documentSchema`, `entries`, `lineageCoverage`, + `relations`, optional `title`, optional `updatedAt`. Entries and their + segments are therefore visited before relations. Refactor the existing + private `projectEntry` and `projectContentSegment` into shared closed + projection helpers, and use `writeCanonicalJson` for projected keys, + scalars, and complete leaf objects; do not independently implement JSON + string escaping or follow JavaScript insertion order in SQLite code. +3. The writer enforces contiguous ordinals, valid nested state transitions, + declared counts, and one finalization. It exposes only the final + `SessionDocumentDigest`, never intermediate transcript fragments. + Differential domain tests cover empty documents, every optional field, every + segment variant, control characters, non-BMP Unicode, the exact top-level + key order, mixed relation/entry counts, and invalid call order. +4. Add `src/domain/session-document-stream.ts` as the provider-neutral + incremental validation and metrics owner. Refactor reusable scalar and + canonical-field checks from `src/domain/session-validation.ts` rather than + maintaining a second interpretation of timestamps, identities, relations, + tool fields, source locators, source metadata, content hashes, and omitted + segments. Give it the stored entry count, validate related-entry bounds + without retaining prior entries, keep only the relation duplicate-key set + plus count/byte accumulators, and produce `SessionDocumentMetrics` at + completion. It accepts nested entry/segment events so one unselected + high-segment entry is not accumulated. Existing provider admission remains + on `validateSessionDocument`; generated differential tests prove both + validators accept the same valid documents and every stable stored-corruption + fixture still fails closed. +5. `src/application/ports/session-index.ts` — add a focused + `SessionSelectionReader extends SessionIndexReader` capability with one + read-only `getSessionSelection` operation. Change only + `src/application/ports/index-lifecycle.ts:IndexReader.sessions` to the + extended capability; keep `SessionIndexWriter` extending the unchanged base + reader so indexing implementations and writer fakes do not acquire a + presentation-oriented method. +6. The selection request carries a non-negative relation limit and inclusive + `fromEntry`/`toEntry` coordinates; it never converts them to a half-open end + before the verified entry count is known. Values up to + `Number.MAX_SAFE_INTEGER` remain admitted. The result carries verified public + header fields, selected canonical relations and entries, complete document + metrics, verified canonical digest, and a verified half-open actual window + `{start,end}`. Compute it only after the total is known: + `start = min(fromEntry,total)`; if `toEntry >= total`, use `end = total`, + otherwise use `end = toEntry + 1`. Empty intersections are `{0,0}` or + `{total,total}`. A coordinate beyond the document remains admitted so the + application can preserve + digest-mismatch-before-coordinate precedence. Keep `getSession` unchanged for + full export, writer replacement assertions, and complete-document callers. +7. `src/infrastructure/sqlite/sqlite-session-document.ts` — implement + `readCanonicalDocumentSelection` with ordered `StatementSync.iterate()` scans + restricted to one `session_id`. Visit entries and segments first, then + relations, matching the top-level canonical key order. Validate and hash + every canonical row exactly once and retain only relations below the limit + and entries intersecting the requested inclusive coordinates. +8. Refactor the existing SQLite-specific decoders in that file—stored integer + and ordinal handling, relation and entry headers, nullable variant columns, + metadata JSON, segment/content row shape, and content digest/text checks—into + shared row-level helpers used by both + `readCanonicalDocumentRecord` and the new selection reader. Sharing only + domain scalar validators is insufficient. Unselected locators, metadata, and + text are still parsed, type-checked, hash-checked, counted, hashed into the + public document when applicable, and released. +9. Reuse `iterator-cleanup.ts` so success, corruption, digest mismatch, and + SQLite failure close every iterator without masking the primary error. Add an + optional internal selection-work observer, excluded from the application + port, that reports only visited rows and current/maximum retained entry, + segment, relation, and scalar-object counts. Wrap every notification and + swallow observer exceptions; a throwing observer must produce the same + successful result or canonical failure as no observer. +10. In the same SQLite owner, decode stored digest and metrics before scanning + but trust neither as proof. Return only after the streamed digest equals the + stored digest and every accumulated metric equals the stored row. Missing + metrics, noncontiguous rows, orphan segment coordinates, malformed metadata, + invalid variants/references, and digest or metric disagreement remain + `corrupt-data`. +11. Add the capability through + `src/infrastructure/sqlite/sqlite-session-index.ts` and immutable composition + in `src/infrastructure/sqlite/database.ts`. Implement + `getSessionSelection` with the existing `getSession` proof sequence: resolve + tracking, stream and verify the canonical selection, read the summary, + compare the verified canonical digest with `summary.documentDigest`, and + only then return. Keep `getSession` unchanged. Before/after snapshot and + sidecar checks remain unchanged. +12. `src/application/session-presentation.ts` — extract a selection-based + presentation function that accepts verified public header fields, selected + relations and entries, complete relation/entry totals, and the actual entry + window. Preserve every current title, relation, entry, segment, per-segment + text, aggregate text, omitted-content, and truncation calculation. + `selectSessionTranscript` remains the full-document compatibility wrapper so + full export and existing callers do not fork presentation semantics. +13. `src/application/show-session.ts`, + `src/application/export-session.ts`, and + `src/application/guard-session-document.ts` — route bounded modes through + the selection capability and generalize `requireExpectedSession` over any + verified result containing a summary; do not duplicate guard logic. + Default bounded reads ask for inclusive coordinates + `0..MAX_SELECTED_ENTRIES - 1`; focused show computes its requested context + with a saturating safe upper coordinate; explicit ranges pass their admitted + inclusive values without `toEntry + 1`. Compare the optional expected digest + only after repository and summary verification, then call the existing + coordinate resolver against the verified total. `export --full` continues + through `getSession`. +14. Add a focused selection-reader contract rather than extending the writer + contract. Extend + `test/infrastructure/sqlite-session-index.test.ts`, + `test/infrastructure/sqlite-reader-lifecycle.test.ts`, + `test/contracts/provider-workflow.contract.ts`, + `test/application/guarded-session-read.test.ts`, + `test/application/show-session.test.ts`, + `test/application/export-session.test.ts`, `test/cli-render.test.ts`, and + `test/cli-structured-output.test.ts`. Pass old full-document and new + selection results through the same human and JSON/JSONL render/encode + functions and compare exact strings. +15. Mechanically update the immutable reader fixtures in + `test/application/list-sessions.test.ts`, + `test/application/list-session-entries.test.ts`, + `test/application/search-sessions.test.ts`, and + `test/application/create-session-manifest.test.ts` to use + `SessionSelectionReader` or a shared reader fixture. Do not add the selection + operation to writer fixtures. +16. Cover empty/short documents, default bounds, focus at both ends, the + 200-entry range maximum, maximum context, `Number.MAX_SAFE_INTEGER` + coordinates, a request beyond the end, matching/mismatching/unguarded reads, + attribution-only changes, relations above the display limit, one unselected + entry with many segments, and every stored-corruption class. Combined-failure + assertions require canonical corruption before expected-digest mismatch, + missing identity before mismatch, valid mismatch before entry absence, and + summary corruption as `corrupt-data`. +17. Add `scripts/measure-session-read.ts`, a + `measure:session-read` package script, a small script-contract test, and the + aggregate contract in `docs/contributing/testing.md`. Generate one canonical + session at 100, 10,000, and 50,000 entries with controlled text bytes and + compare complete-document selection with streamed selection for one entry and + a 200-entry range. Pre-seed one closed generic database, then run baseline and + streamed reads sequentially in separate child processes so seeding and the + other strategy do not pollute peak RSS. +18. Report only generated counts, logical bytes, elapsed time, read-phase peak + RSS, visited rows, maximum retained selected entries/segments/relation keys + and current scalar objects, digest/metric equality, and output equality. + Require exact equality and counters matching the stated retention model; + keep elapsed time and RSS report-only. +19. Update `docs/architecture-memo.md`, + `docs/contributing/architecture.md`, `docs/contributing/search.md`, and + `docs/contributing/testing.md` when implementation lands. Label the path as + complete streamed verification with bounded retention, state that SQLite + I/O remains whole-document, and keep proof-addressable partial reads and + streamed full export evidence-gated. + +## Verify + +- Run the focused public-digest stream, incremental validation, SQLite selection, + guarded-read, show/export, structured-output, and measurement-contract tests. +- Run `pnpm measure:session-read` and require semantic/output equality plus the + retained-object bounds; do not accept the change from elapsed time alone. +- Run `pnpm check`. + +## Boundaries + +- Do not add a migration, change the public document schema or digest scheme, + trust stored digests or metrics, or weaken complete canonical validation. +- Do not add Merkle roots, per-entry digests, inclusion proofs, historical + revisions, batch hydration, a package API, or provider reads. Those change the + evidence contract and require a separate measured architecture plan. +- Do not stream partial JSON to stdout or change bounded-output fail-before-write + behavior. Full export remains on the complete-document path. +- Stop and split the work if exact streamed RFC 8785 equality would require a + second independently maintained projection; the field-by-field projection and + digest owner must remain singular. diff --git a/dev/plans/README.md b/dev/plans/README.md index 4b82403..08c9071 100644 --- a/dev/plans/README.md +++ b/dev/plans/README.md @@ -2,7 +2,7 @@ This directory holds the repository's active implementation plans. -- One plan per independently reviewable change. +- One executor plan per independently reviewable change. - Plans describe outcomes, boundaries, exact change areas, and verification. - Keep plans aligned with the current codebase; update them when scope changes. - Remove completed plans. Git history remains the archive. @@ -10,11 +10,22 @@ This directory holds the repository's active implementation plans. The durable product roadmap lives in the [architecture memo](../../docs/architecture-memo.md#post-v1-roadmap). -## Active executor plans +## Active program plans -- [Digest-guarded coordinate reads](260722-digest-guarded-coordinate-reads.md) - add a caller-supplied document-digest precondition to existing `show` and - `export` reads without adding revision storage or changing successful output. +Each program is ordered internally. Promote one numbered delivery slice at a +time into its own executor plan and independently reviewable change; +evidence-gated slices receive an executor plan only after their gate passes. -Later product milestones still wait for this contract and its acceptance -evidence. +1. [Everyday query hot paths](260723-everyday-query-hot-paths.md) — fix broad + entry selection first, then reduce repeated search hydration and summary + work, add compatible keyset continuation, and measure later planner/startup + candidates. +2. [Indexing hot paths](260723-indexing-hot-paths.md) — measure discovery and + replacement work, remove avoidable provider lookup/serialization and + statement-per-content costs, then gate deeper transaction or WAL changes. +3. [Doctor and maintenance hot paths](260723-doctor-maintenance-hot-paths.md) — + make exact FTS verification memory-bounded, partition oversized terms, page + only orphan candidates, and gate any compact-proof optimization separately. +4. [Verified bounded session reads](260723-verified-bounded-session-reads.md) — + stream complete validation and the existing public-document digest while + retaining only the requested bounded `show` or `export` selection.