From 85df879eadecfd6cdb61c3f19b2b73977d05df42 Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 11:38:05 -0700 Subject: [PATCH 1/5] feat: optimize everyday query paths --- docs/contributing/architecture.md | 33 +- docs/contributing/entries.md | 48 +- docs/contributing/search.md | 29 +- docs/contributing/testing.md | 28 + package.json | 2 + scripts/measure-cli-startup.ts | 182 ++++ scripts/measure-entry-query.ts | 507 ++++++++--- scripts/measure-query-planner-statistics.ts | 824 ++++++++++++++++++ scripts/measure-search-query.ts | 163 +++- src/bin/sessions.ts | 363 +++++--- src/infrastructure/sqlite/query-cursor.ts | 164 +++- .../sqlite/sqlite-query-context.ts | 346 ++++++-- .../sqlite/sqlite-session-entry-query.ts | 350 +++++++- .../sqlite/sqlite-session-query.ts | 413 +++++++-- .../sqlite/sqlite-session-state.ts | 143 ++- test/contracts/session-query.contract.ts | 93 +- .../sqlite-query-primitives.test.ts | 132 +++ .../sqlite-session-entry-query.test.ts | 435 ++++++++- .../sqlite-session-query.test.ts | 571 +++++++++++- .../sqlite-session-state.test.ts | 250 ++++++ ...ery-planner-statistics-measurement.test.ts | 96 ++ 21 files changed, 4657 insertions(+), 515 deletions(-) create mode 100644 scripts/measure-cli-startup.ts create mode 100644 scripts/measure-query-planner-statistics.ts create mode 100644 test/infrastructure/sqlite-session-state.test.ts create mode 100644 test/query-planner-statistics-measurement.test.ts diff --git a/docs/contributing/architecture.md b/docs/contributing/architecture.md index 61cf135..45d1b31 100644 --- a/docs/contributing/architecture.md +++ b/docs/contributing/architecture.md @@ -53,13 +53,15 @@ forget / data repair-orphans / data compact / data clear -> src/infrastructure/sqlite/index-maintenance.ts ``` -The composition root is the only production module that imports both a concrete -adapter and infrastructure. It resolves Cursor and Codex lazily: help, version, -list, search, entries, manifest, show, export, forget, data repair-orphans, data -compact, and data clear do not resolve provider configuration. -Index, paths, and doctor resolve registered sources. Implicit indexing skips -only valid unavailable sources; explicit selection and all other probe failures -remain strict. +The composition root is the only production module that selects both concrete +adapters and infrastructure. CLI grammar stays static, while providers, +lifecycle, maintenance, diagnostics, indexing, and command services load through +memoized callback-local imports. Help and version therefore load none of those +capabilities. List, search, entries, manifest, show, export, forget, data +repair-orphans, data compact, and data clear load their provider-free services +without resolving provider configuration. Index, paths, and doctor resolve +registered sources. Implicit indexing skips only valid unavailable sources; +explicit selection and all other probe failures remain strict. ## Ownership @@ -308,7 +310,11 @@ FTS damage. List/search/manifest read the stored digest directly and do not reconstruct every document. None resolve or reopen a provider, so retained content remains usable after provider disappearance. The guard changes neither the SQLite schema nor physical read cost. -Query cursors bind the query plus library identity/writer generation. An explicit +Query cursors bind the query plus library identity/writer generation. List and +entry continuations use v2 provider-neutral numeric anchors for indexed keyset +continuation while retaining the cumulative offset and accepting existing v1 +offset tokens; search remains on the compatible v1 offset format. No cursor +contains a raw provider identity. An explicit leased index writer can rebuild FTS-only damage from canonical content. Doctor stays read-only and reports canonical integrity, content reachability, projection health, and the global capture aggregate separately. Canonical health @@ -370,10 +376,15 @@ fallback, bounded context assembly, query-wide counts, and cursor encoding. The shared capture-scope reader uses registered-source and tracking columns only, reports applicable source coverage even for a no-hit page, and names canonical metadata, entry, and text filters it cannot assess for unindexed sessions. -Search ranks compact coordinates first, keeps the extra rank-only row used for -pagination, and hydrates text, digest, and snippets only for the selected page. +List, search, and entries carry compact canonical session IDs through page +selection and hydrate the selected retained summaries in one bounded, +identity-checked batch. Search ranks compact coordinates first, keeps the extra +rank-only row used for pagination, and hydrates text, digest, and snippets in one +selected-page statement. Context link discovery is page-wide; physical context +coordinates are deduplicated and their bodies are read in fixed-size chunks. Snippet markers are checked against those selected canonical texts and retried -on collision, so search does not scan or hydrate the rest of the library. +for the complete selected page on collision, so search does not scan or hydrate +unselected text. `any` mode derives exact per-hit terms with candidate-local probes after page selection. One query-scoped root resolver serves support plus list/search/entries attribution. Support remains exact and query-wide rather than being inferred from the page. diff --git a/docs/contributing/entries.md b/docs/contributing/entries.md index c2e1778..e5c2380 100644 --- a/docs/contributing/entries.md +++ b/docs/contributing/entries.md @@ -8,23 +8,35 @@ The public behavior is defined by the 1. The domain validates and freezes shared session filters, exact entry filters, selection, page limit, and cursor. Defaults are `all` and 50; the maximum page is 200 entries. -2. SQLite selects compact entry coordinates through source, tracking, canonical - session, and entry rows. It does not use FTS or reconstruct session documents. +2. SQLite selects compact entry coordinates through an explicit + source→tracking→canonical→entry traversal. `CROSS JOIN` boundaries keep that + loop order stable so SQLite can stop after the requested page instead of + scanning the entry table and sorting the complete result. It does not use FTS + or reconstruct session documents. 3. `all` keeps every qualifying entry. `first` or `last` keeps the lowest or highest canonical ordinal per session after all filters are applied. 4. Origin qualifies through retained segments, including omitted content. An empty entry cannot match an origin. Tool name or namespace selects observed `tool-call` entries only. 5. After page selection, SQLite loads exact segment counts and at most one text - preview per entry. An origin-filtered query previews only text with that - origin; an omitted-only match has no preview. + preview per entry. It deduplicates selected sessions and loads their retained + summaries in one identity-checked batch of at most 200 requests. An + origin-filtered query previews only text with that origin; an omitted-only + match has no preview. 6. The same immutable snapshot supplies one page-level capture scope from registered sources and tracking state. Entry and canonical metadata filters are named as unassessed for tracking-only sessions. 7. One query-scoped lineage resolver returns the known retained root or `unknown` for each result session. -8. A next cursor binds the full query, library identity, writer generation, and - offset. It is a continuation token, not a durable bookmark. +8. A next cursor binds the full query, library identity, writer generation, + cumulative offset, and the last selected numeric session/entry coordinate. + Continued pages strictly resolve that coordinate under the same filters and + selection, recover its binary identity order, and continue after it without + `OFFSET`. The token contains no raw provider identity. +9. Existing v1 offset cursors remain accepted. They use the offset path once and + emit a v2 anchored continuation. A missing or nonqualifying matching-revision + anchor is invalid; a different library or writer generation is stale. Cursors + remain opaque continuation tokens, not durable bookmarks. ## Order and evidence @@ -44,13 +56,25 @@ match or non-match. ## Cost -The query scans matching entry coordinates in stable key order and hydrates text -only for the selected page. `first` and `last` use a per-session ordinal lookup -after filtering. Origin filters probe retained segment rows. Root resolution -reads retained lineage once per nonempty query and reuses results within it. +The query walks matching entry coordinates in stable source and identity index +order, stops after `limit + 1`, and hydrates text only for the selected page. +`first` and `last` use a per-session ordinal lookup after filtering. Origin +filters probe retained segment rows. Root resolution reads retained lineage once +per nonempty query and reuses results within it. -The current schema and indexes are sufficient for the measured generic corpus. -Any later durable index needs separate size and query evidence. +The generated measure uses an on-disk skewed corpus with thousands of sessions, +hundreds of thousands of mostly textless entries, and one session with tens of +thousands of entries. It warms and repeats broad and filtered `all`, `first`, +`last`, origin, tool, early-page, and deep-page reads. The report separates +coordinate selection from full hydration, proves repeated result equality, and +normalizes `EXPLAIN QUERY PLAN` into the outer access, indexes used, and +temporary-ordering presence. It also reports whether the production statement +uses keyset continuation or `OFFSET`; new and v2 entry pages must have no +`OFFSET`. Elapsed time is report-only; semantic equality and the +source-first/no-full-sort bounded plan shape are the structural proof. + +The current schema and indexes are sufficient for this traversal. Any later +durable index needs separate size and query evidence. ## Code and proof diff --git a/docs/contributing/search.md b/docs/contributing/search.md index cde7153..acb639a 100644 --- a/docs/contributing/search.md +++ b/docs/contributing/search.md @@ -14,9 +14,12 @@ The public behavior is defined by the [CLI contract](../reference/cli-contract.m 4. SQLite ranks each entry by its best FTS5 BM25 score. Ties use session activity, source identity, and entry ordinal in a fixed order. The query keeps one extra rank row only to decide whether a next page exists. -5. After selecting the page, SQLite loads full text, its digest, and the FTS - snippet only for selected content IDs. `any` mode probes each selected entry - for its exact matched terms; `all` reuses the unique query terms. +5. After selecting the page, one page-bounded SQLite statement loads full text, + its digest, and the FTS snippet for the distinct selected content IDs. Snippet + marker collisions retry that whole selected page deterministically. `any` + mode probes each selected entry for its exact matched terms; `all` reuses the + unique query terms. Distinct selected sessions load their retained summaries + in one identity-checked batch of at most 200 requests. 6. The same immutable snapshot supplies one page-level capture scope from registered sources and tracking state. Source/tracking filters can narrow that aggregate; canonical metadata, entry, and search-text filters are named as @@ -25,7 +28,9 @@ The public behavior is defined by the [CLI contract](../reference/cli-contract.m `unknown`. 8. Search verifies each selected content hash, bounds snippets and context to 512 UTF-8 bytes, and adds requested neighboring entries plus direct tool-call and - tool-result links. + tool-result links. One page-level query discovers direct links, then selected + physical context coordinates are deduplicated and hydrated in fixed-size + chunks before each hit's flags and order are rebuilt. 9. Support is calculated over the complete filtered retained result, before page slicing: matching occurrences, distinct content, known roots, and sessions with unknown lineage. @@ -57,9 +62,23 @@ and lineage counts still inspect the full qualifying result, so broad searches c still take longer on large libraries. The design favors exact evidence over approximate counts or ranking. +Selected-content, summary, and linked-context query counts are page-bounded +instead of growing once per hit. Context body hydration grows with fixed-size +chunks of distinct selected physical entries, so overlapping hit neighborhoods +share the same read. These batching steps do not change ranking or query-wide +support work. + +The exact query-wide support, identity, and rank passes intentionally remain +separate. A generated `MATERIALIZED` one-pass prototype proved one qualifying FTS +scan and exact result parity, but was about 15% slower across broad profiles +instead of meeting the 20% improvement gate. Its selective regression and peak +memory stayed within bounds, so performance—not correctness—closed that +candidate and no prototype runtime path is retained. + `any` mode adds at most one candidate-local FTS probe per unique term for the selected page. Activity bounds reuse canonical timestamps and add no storage -index. The repeatable measurement covers both term modes; elapsed time remains +index. The repeatable measurement covers small and maximum-size pages, with and +without neighboring context, in both term modes; elapsed time remains report-only. ## Retrieve an exact span diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index c5f070f..3fc898d 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -252,6 +252,34 @@ set-based query shape from N+1 drift. It constructs no provider adapter, prints only aggregate counts and timings, has no timed threshold, and stays outside `pnpm check`. +`pnpm measure:query-planner-statistics` is an opt-in, provider-free experiment +for SQLite planner statistics. It builds one skewed generic library, closes it, +then makes byte-exact control, `ANALYZE`, and forced `PRAGMA optimize` clones. +Only the two experiment clones receive statistics mutations. The command +compares normalized representative plans and alternating warm elapsed +aggregates for broad and narrow entries, identity and activity list filters, +broad and selective `all`/`any` search, and manifest. Production query results +must remain exactly equal across all clones. The report also includes +statistics-table row counts, statistics payload bytes, mutation time, database +growth, clone verification, and final temporary-root cleanup. The experiment +has no elapsed threshold and does not define or invoke a runtime refresh policy. +The full corpus stays outside `pnpm check`; a small `-- --contract` run in the +test suite checks mutation isolation, semantic equality, report shape, and +cleanup. Current generated evidence leaves runtime statistics disabled: entry +plans stayed unchanged while several narrow, list, and search profiles regressed. +Any future policy proposal must rerun this experiment against the then-current +queries and separately design certified writer mutation and refresh semantics. + +`pnpm measure:cli-startup` is the compiled, provider-free startup measure. Run +`pnpm build` first. It rotates bare Node, version, top-level help, command help, +and one uninitialized local read across warm-up and measured rounds using only a +mode-0700 temporary data root. It checks exact success/output shapes, reports +median and p95 elapsed aggregates plus overhead against bare Node, and removes +the temporary root before emitting its report. Lazy CLI composition is a +candidate only when version, top help, and command help each cost at least 25 ms +over bare Node and at least 2× bare Node. These are implementation decision +thresholds, not CI performance budgets; the command stays outside `pnpm check`. + `pnpm measure:indexing:codex -- --allow-provider-read` is the Codex local provider-read measurement. It is macOS/Linux-only and fails before provider resolution elsewhere. It uses the production Codex adapter, exhausts its full diff --git a/package.json b/package.json index 6f06edf..e828ffb 100644 --- a/package.json +++ b/package.json @@ -46,6 +46,7 @@ "format:docs:check": "oxfmt --check \"**/*.md\"", "lint": "oxlint -c .oxlintrc.json src scripts test vitest.config.ts", "lint:fix": "oxlint -c .oxlintrc.json --fix src scripts test vitest.config.ts", + "measure:cli-startup": "node scripts/measure-cli-startup.ts", "measure:content-storage": "node scripts/measure-content-storage.ts", "measure:entry-query": "node scripts/measure-entry-query.ts", "measure:indexing": "node scripts/measure-indexing.ts", @@ -53,6 +54,7 @@ "measure:indexing:cursor": "node scripts/measure-cursor-indexing.ts", "measure:manifest": "node scripts/measure-manifest.ts", "measure:query-lineage": "node scripts/measure-query-lineage.ts", + "measure:query-planner-statistics": "node scripts/measure-query-planner-statistics.ts", "measure:search-query": "node scripts/measure-search-query.ts", "hooks:install": "simple-git-hooks", "prepack": "pnpm build", diff --git a/scripts/measure-cli-startup.ts b/scripts/measure-cli-startup.ts new file mode 100644 index 0000000..100b2bb --- /dev/null +++ b/scripts/measure-cli-startup.ts @@ -0,0 +1,182 @@ +import assert from "node:assert/strict"; +import { chmod, mkdtemp, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { performance } from "node:perf_hooks"; +import { spawnSync } from "node:child_process"; + +const WARMUP_ROTATIONS = 2; +const MEASURED_ROTATIONS = 15; +const MATERIAL_OVERHEAD_MILLISECONDS = 25; +const MATERIAL_OVERHEAD_RATIO = 2; + +interface Scenario { + readonly name: "bareNode" | "version" | "topHelp" | "commandHelp" | "uninitializedList"; + readonly arguments: readonly string[]; + readonly assertOutput: (stdout: string) => void; +} + +const compiledEntry = resolve("dist/bin/sessions.js"); +const compiled = await stat(compiledEntry).catch(() => undefined); +assert(compiled?.isFile(), "Build dist before measuring CLI startup"); + +const temporaryRoot = await mkdtemp(join(tmpdir(), "sessions-cli-startup-")); +await chmod(temporaryRoot, 0o700); +let report: MeasurementReport | undefined; +try { + const generatedDataDirectory = join(temporaryRoot, "library"); + const scenarios: readonly Scenario[] = [ + { + name: "bareNode", + arguments: ["-e", ""], + assertOutput: (stdout) => assert.equal(stdout, ""), + }, + { + name: "version", + arguments: [compiledEntry, "--version"], + assertOutput: (stdout) => assert.match(stdout, /^\d+\.\d+\.\d+\n$/u), + }, + { + name: "topHelp", + arguments: [compiledEntry, "--help"], + assertOutput: (stdout) => assert.match(stdout, /^Usage: sessions/u), + }, + { + name: "commandHelp", + arguments: [compiledEntry, "list", "--help"], + assertOutput: (stdout) => assert.match(stdout, /^Usage: sessions list/u), + }, + { + name: "uninitializedList", + arguments: [compiledEntry, "list", "--limit", "1", "--format", "json"], + assertOutput: assertUninitializedList, + }, + ]; + const samples = new Map(scenarios.map(({ name }) => [name, []])); + + for (let rotation = 0; rotation < WARMUP_ROTATIONS + MEASURED_ROTATIONS; rotation += 1) { + const ordered = rotate(scenarios, rotation % scenarios.length); + for (const scenario of ordered) { + const elapsed = runScenario(scenario, generatedDataDirectory); + if (rotation >= WARMUP_ROTATIONS) samples.get(scenario.name)?.push(elapsed); + } + } + + const measurements = Object.fromEntries( + scenarios.map(({ name }) => { + const values = samples.get(name); + assert(values !== undefined && values.length === MEASURED_ROTATIONS); + return [name, summarize(values)]; + }), + ) as Record; + const bareMedian = measurements.bareNode.medianMilliseconds; + const candidates = (["version", "topHelp", "commandHelp"] as const).map((name) => { + const median = measurements[name].medianMilliseconds; + return { + name, + overheadMilliseconds: roundMilliseconds(median - bareMedian), + overheadRatio: roundRatio(median / bareMedian), + }; + }); + const lazyCompositionCandidate = candidates.every( + ({ overheadMilliseconds, overheadRatio }) => + overheadMilliseconds >= MATERIAL_OVERHEAD_MILLISECONDS && + overheadRatio >= MATERIAL_OVERHEAD_RATIO, + ); + + report = { + rotations: MEASURED_ROTATIONS, + measurements, + startupCandidates: candidates, + materialGate: { + minimumOverheadMilliseconds: MATERIAL_OVERHEAD_MILLISECONDS, + minimumOverheadRatio: MATERIAL_OVERHEAD_RATIO, + lazyCompositionCandidate, + }, + }; +} finally { + await rm(temporaryRoot, { recursive: true, force: true }); +} +assert.equal(await stat(temporaryRoot).catch(() => undefined), undefined); +assert(report !== undefined, "CLI startup measurement did not produce a report"); +process.stdout.write(`${JSON.stringify({ ...report, temporaryCleanup: true })}\n`); + +function runScenario(scenario: Scenario, generatedDataDirectory: string): number { + const startedAt = performance.now(); + const result = spawnSync(process.execPath, scenario.arguments, { + encoding: "utf8", + env: { + ...process.env, + SESSIONS_DATA_DIR: generatedDataDirectory, + }, + maxBuffer: 4 * 1_024 * 1_024, + stdio: ["ignore", "pipe", "pipe"], + }); + const elapsed = performance.now() - startedAt; + if (result.error) throw result.error; + assert.equal(result.signal, null, `${scenario.name} received a signal`); + assert.equal(result.status, 0, `${scenario.name} failed: ${result.stderr}`); + assert.equal(result.stderr, "", `${scenario.name} wrote stderr`); + scenario.assertOutput(result.stdout); + return elapsed; +} + +function assertUninitializedList(stdout: string): void { + const parsed = JSON.parse(stdout) as unknown; + assert( + typeof parsed === "object" && + parsed !== null && + !Array.isArray(parsed) && + "sessions" in parsed && + Array.isArray(parsed.sessions) && + parsed.sessions.length === 0, + "Uninitialized list output changed", + ); +} + +function rotate(values: readonly T[], offset: number): readonly T[] { + return [...values.slice(offset), ...values.slice(0, offset)]; +} + +interface Summary { + readonly minimumMilliseconds: number; + readonly medianMilliseconds: number; + readonly p95Milliseconds: number; +} + +interface MeasurementReport { + readonly rotations: number; + readonly measurements: Record; + readonly startupCandidates: readonly { + readonly name: "version" | "topHelp" | "commandHelp"; + readonly overheadMilliseconds: number; + readonly overheadRatio: number; + }[]; + readonly materialGate: { + readonly minimumOverheadMilliseconds: number; + readonly minimumOverheadRatio: number; + readonly lazyCompositionCandidate: boolean; + }; +} + +function summarize(values: readonly number[]): Summary { + const sorted = values.toSorted((left, right) => left - right); + return { + minimumMilliseconds: roundMilliseconds(sorted[0] ?? 0), + medianMilliseconds: roundMilliseconds(percentile(sorted, 0.5)), + p95Milliseconds: roundMilliseconds(percentile(sorted, 0.95)), + }; +} + +function percentile(sorted: readonly number[], fraction: number): number { + const index = Math.max(0, Math.ceil(sorted.length * fraction) - 1); + return sorted[index] ?? 0; +} + +function roundMilliseconds(value: number): number { + return Number(value.toFixed(3)); +} + +function roundRatio(value: number): number { + return Number(value.toFixed(2)); +} diff --git a/scripts/measure-entry-query.ts b/scripts/measure-entry-query.ts index 5f54ee7..8f303ae 100644 --- a/scripts/measure-entry-query.ts +++ b/scripts/measure-entry-query.ts @@ -1,4 +1,7 @@ import assert from "node:assert/strict"; +import { mkdtemp, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { performance } from "node:perf_hooks"; import { DatabaseSync } from "node:sqlite"; @@ -15,7 +18,6 @@ import { type SessionEntryQuery, } from "../src/domain/session-query.ts"; import type { - ContentOrigin, SessionDocument, SessionEntry, SessionIdentity, @@ -25,16 +27,25 @@ import { applyMigrations, CURRENT_INDEX_SCHEMA_VERSION, } from "../src/infrastructure/sqlite/migrations.ts"; +import { + buildSqliteEntryCoordinateStatement, + listSqliteSessionEntries, + type SqliteEntryCoordinatePosition, + type SqliteEntryQueryWork, +} from "../src/infrastructure/sqlite/sqlite-session-entry-query.ts"; import { createCoordinatedSqliteSessionIndex } from "../src/infrastructure/sqlite/sqlite-session-index.ts"; -import { createSqliteSessionQuery } from "../src/infrastructure/sqlite/sqlite-session-query.ts"; import { acquireWriterLease, interruptOwnedRunsAndReleaseWriterLease, } from "../src/infrastructure/sqlite/writer-lease.ts"; import { initializeWriterRecoveryReceipt } from "../src/infrastructure/sqlite/writer-recovery-receipt.ts"; -const CORPUS_SESSIONS = 2_000; -const ENTRIES_PER_SESSION = 5; +const REGULAR_SESSIONS = 2_000; +const REGULAR_ENTRIES = 96; +const DENSE_SESSION_ENTRIES = 20_000; +const CORPUS_SESSIONS = REGULAR_SESSIONS + 1; +const CORPUS_ENTRIES = REGULAR_SESSIONS * REGULAR_ENTRIES + DENSE_SESSION_ENTRIES; +const SHARED_TEXT = "bounded shared injected evidence"; const SOURCE: SourceInstance = Object.freeze({ kind: "synthetic-measurement", instanceId: "entry-query", @@ -42,69 +53,251 @@ const SOURCE: SourceInstance = Object.freeze({ const CAPTURED_AT = "2026-07-15T12:00:00.000Z"; const RELEASED_AT = "2026-07-15T12:02:00.000Z"; -const database = openDatabase(); +const temporaryDirectory = await mkdtemp(join(tmpdir(), "sessions-entry-measure-")); +const databasePath = join(temporaryDirectory, "library.sqlite"); +const database = openDatabase(databasePath); try { + const seedStartedAt = performance.now(); await seedCorpus(database); - const repository = createSqliteSessionQuery(database); - const measurements = { - all: await measure(repository.entries, createSessionEntryQuery({ limit: 7 }), assertAllPage), - first: await measure( - repository.entries, - createSessionEntryQuery({ filter: { actor: "human" }, selection: "first", limit: 5 }), - (page) => assertSelectedPage(page, 1), + const seedingMilliseconds = roundMilliseconds(performance.now() - seedStartedAt); + const databaseBytes = (await stat(databasePath)).size; + + const broadDeep = cursorAtOffset(database, createSessionEntryQuery({ limit: 200 }), 10_000); + const profiles = { + broadEarly: measureProfile(database, createSessionEntryQuery({ limit: 50 }), 0, (page) => + assertDenseRange(page, 0, 50), + ), + broadDeep: measureProfile(database, broadDeep, 10_000, (page) => + assertDenseRange(page, 10_000, 200), + ), + filteredAll: measureProfile( + database, + createSessionEntryQuery({ + filter: { nativeId: nativeIdAt(0) }, + limit: 50, + }), + 0, + (page) => assertDenseRange(page, 0, 50), ), - last: await measure( - repository.entries, - createSessionEntryQuery({ filter: { actor: "human" }, selection: "last", limit: 5 }), - (page) => assertSelectedPage(page, 4), + first: measureProfile( + database, + createSessionEntryQuery({ + filter: { actor: "human" }, + selection: "first", + limit: 50, + }), + 0, + (page) => assertSelectedPage(page, "first"), + ), + last: measureProfile( + database, + createSessionEntryQuery({ + filter: { actor: "human" }, + selection: "last", + limit: 50, + }), + 0, + (page) => assertSelectedPage(page, "last"), ), - tool: await measure( - repository.entries, + origin: measureProfile( + database, + createSessionEntryQuery({ + filter: { origin: "injected" }, + limit: 50, + }), + 0, + (page) => assertOrdinalPage(page, 0), + ), + tool: measureProfile( + database, createSessionEntryQuery({ filter: { toolName: "exec_command", toolNamespace: "functions" }, - limit: 5, + limit: 50, }), - (page) => assertSelectedPage(page, 2), + 0, + (page) => assertOrdinalPage(page, 2), ), - activity: await measure( - repository.entries, + activity: measureProfile( + database, createSessionEntryQuery({ filter: { activityAfter: "2026-07-15T11:59:59.999Z", activityBefore: "2026-07-15T12:00:00.001Z", }, - limit: 7, + limit: 50, }), - assertAllPage, + 0, + (page) => assertDenseRange(page, 0, 50), ), }; process.stdout.write( `${JSON.stringify({ - corpusSessions: CORPUS_SESSIONS, - entriesPerSession: ENTRIES_PER_SESSION, - corpusEntries: CORPUS_SESSIONS * ENTRIES_PER_SESSION, - elapsedMs: measurements, + corpus: { + sessions: CORPUS_SESSIONS, + entries: CORPUS_ENTRIES, + regularSessionEntries: REGULAR_ENTRIES, + denseSessionEntries: DENSE_SESSION_ENTRIES, + retainedTextOccurrences: CORPUS_SESSIONS, + uniqueRetainedTexts: 1, + databaseBytes, + }, + seedingMilliseconds, + profiles, })}\n`, ); } finally { database.close(); + await rm(temporaryDirectory, { recursive: true, force: true }); } -async function measure( - run: (query: SessionEntryQuery) => Promise, +function measureProfile( + database: DatabaseSync, query: SessionEntryQuery, + offset: number, assertPage: (page: SessionEntryPage) => void, -): Promise { - const warm = await run(query); +): Measurement { + const warmWork: SqliteEntryQueryWork[] = []; + const warm = listSqliteSessionEntries(database, query, { + observeWork: (work) => warmWork.push(work), + }); assertPage(warm); + assertWorkShape(warmWork); + const repeatedWork: SqliteEntryQueryWork[] = []; const startedAt = performance.now(); - const repeated = await run(query); - const elapsedMilliseconds = performance.now() - startedAt; + const repeated = listSqliteSessionEntries(database, query, { + observeWork: (work) => repeatedWork.push(work), + }); + const totalMilliseconds = performance.now() - startedAt; assertPage(repeated); + assertWorkShape(repeatedWork); assert.deepStrictEqual(repeated, warm, "repeated entry query changed its result"); - return roundMilliseconds(elapsedMilliseconds); + + return { + totalMilliseconds: roundMilliseconds(totalMilliseconds), + coordinateSelection: workResult(repeatedWork, "coordinate-selection"), + hydration: workResult(repeatedWork, "hydration"), + selectedEntries: repeated.entries.length, + hasContinuation: repeated.nextCursor !== undefined, + repeatedResultEqual: true, + plan: readPlanFacts(database, query, offset), + }; +} + +function cursorAtOffset( + database: DatabaseSync, + initialQuery: SessionEntryQuery, + targetOffset: number, +): SessionEntryQuery { + assert.equal(targetOffset % initialQuery.limit, 0, "deep offset must align to page size"); + let query = initialQuery; + for (let offset = 0; offset < targetOffset; offset += initialQuery.limit) { + const page = listSqliteSessionEntries(database, query); + assert(page.nextCursor !== undefined, "deep measurement offset exceeds the corpus"); + query = createSessionEntryQuery({ + filter: initialQuery.filter, + selection: initialQuery.selection, + limit: initialQuery.limit, + cursor: page.nextCursor, + }); + } + return query; +} + +function readPlanFacts( + database: DatabaseSync, + query: SessionEntryQuery, + offset: number, +): PlanFacts { + const statement = buildSqliteEntryCoordinateStatement( + query, + measurementPosition(database, offset), + ); + const rows = database + .prepare(`EXPLAIN QUERY PLAN ${statement.sql}`) + .all(...statement.parameters) as unknown as readonly QueryPlanRow[]; + const accesses = rows.flatMap(({ detail }) => { + const match = /^(SCAN|SEARCH) ([A-Za-z0-9_]+)/u.exec(detail); + return match === null + ? [] + : [{ operation: match[1]?.toLowerCase(), relation: match[2] } as PlanAccess]; + }); + const indexes = new Set(); + for (const { detail } of rows) { + const named = /USING (?:COVERING )?INDEX ([^ ]+)/u.exec(detail); + if (named?.[1] !== undefined) indexes.add(named[1]); + if (detail.includes("USING INTEGER PRIMARY KEY")) indexes.add("integer-primary-key"); + } + return { + outerAccess: accesses[0] ?? { operation: "unknown", relation: "unknown" }, + indexes: [...indexes].sort(), + usesOffset: statement.sql.includes("OFFSET"), + usesKeyset: statement.sql.includes("entry.ordinal > CASE"), + usesEntryOrdinalRange: rows.some( + ({ detail }) => detail.startsWith("SEARCH entry ") && detail.includes("ordinal>?"), + ), + usesTemporaryOrderBy: rows.some(({ detail }) => + detail.includes("USE TEMP B-TREE FOR ORDER BY"), + ), + }; +} + +function measurementPosition( + database: DatabaseSync, + offset: number, +): SqliteEntryCoordinatePosition { + if (offset === 0) return { kind: "first" }; + const row = database + .prepare( + `SELECT canonical.session_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE source.kind = ? + AND source.instance_id = ? + AND tracking.native_id = ?`, + ) + .get(SOURCE.kind, SOURCE.instanceId, nativeIdAt(0)) as + | { readonly session_id?: unknown } + | undefined; + const sessionId = row?.session_id; + assert( + typeof sessionId === "number" && Number.isSafeInteger(sessionId), + "dense measurement session is missing", + ); + return { + kind: "keyset", + anchor: { + sessionId, + entryOrdinal: offset - 1, + sourceKind: SOURCE.kind, + instanceId: SOURCE.instanceId, + nativeId: nativeIdAt(0), + }, + }; +} + +function workResult( + work: readonly SqliteEntryQueryWork[], + phase: SqliteEntryQueryWork["phase"], +): WorkResult { + const selected = work.find((candidate) => candidate.phase === phase); + assert(selected !== undefined, `missing ${phase} measurement`); + return { + milliseconds: roundMilliseconds(selected.elapsedMilliseconds), + rows: selected.rowCount, + }; +} + +function assertWorkShape(work: readonly SqliteEntryQueryWork[]): void { + assert.deepStrictEqual( + work.map(({ phase }) => phase), + ["coordinate-selection", "hydration"], + "entry query emitted an unexpected work phase", + ); } async function seedCorpus(database: DatabaseSync): Promise { @@ -124,8 +317,9 @@ async function seedCorpus(database: DatabaseSync): Promise { schemaVersion: CURRENT_INDEX_SCHEMA_VERSION, }); const run = await index.startRun({ source: SOURCE, startedAt: CAPTURED_AT }); - for (let ordinal = 0; ordinal < CORPUS_SESSIONS; ordinal += 1) { - await index.replaceSession(run, replacement(ordinal)); + await index.replaceSession(run, replacement(0, DENSE_SESSION_ENTRIES)); + for (let ordinal = 1; ordinal <= REGULAR_SESSIONS; ordinal += 1) { + await index.replaceSession(run, replacement(ordinal, REGULAR_ENTRIES)); } const result = await index.finishRun(run, { status: "completed", @@ -146,7 +340,7 @@ async function seedCorpus(database: DatabaseSync): Promise { } } -function replacement(ordinal: number): ValidatedSessionReplacement { +function replacement(ordinal: number, entryCount: number): ValidatedSessionReplacement { const identity = identityAt(ordinal); const candidate = createDiscoveredSession({ identity, @@ -161,12 +355,15 @@ function replacement(ordinal: number): ValidatedSessionReplacement { }); const observation = admitSessionObservation(candidate); assert(observation.ok, "measurement observation was rejected"); - const admitted = admitSessionReplacement(observation.observation, documentAt(ordinal)); + const admitted = admitSessionReplacement( + observation.observation, + documentAt(ordinal, entryCount), + ); assert(admitted.ok, "measurement document was rejected"); return admitted.replacement; } -function documentAt(ordinal: number): SessionDocument { +function documentAt(ordinal: number, entryCount: number): SessionDocument { const identity = identityAt(ordinal); return { identity, @@ -175,138 +372,133 @@ function documentAt(ordinal: number): SessionDocument { updatedAt: CAPTURED_AT, lineageCoverage: "complete", relations: [], - entries: [ - textEntry(0, "system", "injected", `Injected context ${nativeIdAt(ordinal)}`), - textEntry(1, "human", "human", `Initial request ${nativeIdAt(ordinal)}`), - { - ...textEntry(2, "model", "model", `Observed tool call ${nativeIdAt(ordinal)}`), - kind: "tool-call", - toolCallId: `call-${ordinal}`, - toolName: "exec_command", - toolNamespace: "functions", - }, - { - ordinal: 3, - kind: "tool-result", - actor: "tool", - timestamp: timestampAt(3), - relatedEntryOrdinal: 2, - toolCallId: `call-${ordinal}`, - sourceLocator: { uri: `memory://entry-measurement/${nativeIdAt(ordinal)}/entry/3` }, - content: [ - { - kind: "omitted", - ordinal: 0, - contentClass: "structured", - sourceType: "tool-result", - origin: "tool", - originConfidence: "high", - sourceMetadata: {}, - }, - ], - }, - textEntry(4, "human", "human", `Final correction ${nativeIdAt(ordinal)}`), - ], + entries: Array.from({ length: entryCount }, (_, entryOrdinal) => + entryAt(ordinal, entryOrdinal), + ), }; } -function textEntry( - ordinal: number, - actor: SessionEntry["actor"], - origin: ContentOrigin, - text: string, -): SessionEntry { +function entryAt(sessionOrdinal: number, entryOrdinal: number): SessionEntry { + const sourceLocator = { + uri: `memory://entry-measurement/${nativeIdAt(sessionOrdinal)}/${String(entryOrdinal)}`, + }; + if (entryOrdinal === 0) { + return { + ordinal: entryOrdinal, + kind: "message", + actor: "system", + timestamp: CAPTURED_AT, + sourceLocator, + content: [ + { + kind: "text", + ordinal: 0, + text: SHARED_TEXT, + contentHash: hashContent(SHARED_TEXT), + origin: "injected", + originConfidence: "high", + sourceMetadata: {}, + }, + ], + }; + } + if (entryOrdinal === 2) { + return { + ordinal: entryOrdinal, + kind: "tool-call", + actor: "model", + timestamp: CAPTURED_AT, + toolCallId: `call-${String(sessionOrdinal)}`, + toolName: "exec_command", + toolNamespace: "functions", + sourceLocator, + content: [], + }; + } + if (entryOrdinal === 3) { + return { + ordinal: entryOrdinal, + kind: "tool-result", + actor: "tool", + timestamp: CAPTURED_AT, + relatedEntryOrdinal: 2, + toolCallId: `call-${String(sessionOrdinal)}`, + sourceLocator, + content: [], + }; + } return { - ordinal, + ordinal: entryOrdinal, kind: "message", - actor, - timestamp: timestampAt(ordinal), - sourceLocator: { - uri: `memory://entry-measurement/entry/${String(ordinal)}`, - }, - content: [ - { - kind: "text", - ordinal: 0, - text, - contentHash: hashContent(text), - origin, - originConfidence: "high", - sourceMetadata: {}, - }, - ], + actor: entryOrdinal % 2 === 1 ? "human" : "model", + timestamp: CAPTURED_AT, + sourceLocator, + content: [], }; } -function assertAllPage(page: SessionEntryPage): void { +function assertDenseRange(page: SessionEntryPage, start: number, count: number): void { assert.deepStrictEqual( page.entries.map(({ session, entry }) => [session.identity.nativeId, entry.ordinal]), - [ - [nativeIdAt(0), 0], - [nativeIdAt(0), 1], - [nativeIdAt(0), 2], - [nativeIdAt(0), 3], - [nativeIdAt(0), 4], - [nativeIdAt(1), 0], - [nativeIdAt(1), 1], - ], + Array.from({ length: count }, (_, index) => [nativeIdAt(0), start + index]), ); - assert(page.nextCursor !== undefined, "broad entry page must have a continuation cursor"); + assert(page.nextCursor !== undefined, "dense page must have a continuation"); page.entries.forEach(assertEntryFacts); } -function assertSelectedPage(page: SessionEntryPage, expectedEntryOrdinal: number): void { +function assertSelectedPage(page: SessionEntryPage, selection: "first" | "last"): void { + assert.deepStrictEqual( + page.entries.map(({ session, entry }) => [ + session.identity.nativeId, + entry.ordinal, + entry.actor, + ]), + Array.from({ length: 50 }, (_, index) => [ + nativeIdAt(index), + selection === "first" ? 1 : index === 0 ? DENSE_SESSION_ENTRIES - 1 : REGULAR_ENTRIES - 1, + "human", + ]), + ); + assert(page.nextCursor !== undefined, "selected page must have a continuation"); + page.entries.forEach(assertEntryFacts); +} + +function assertOrdinalPage(page: SessionEntryPage, expectedEntryOrdinal: number): void { assert.deepStrictEqual( page.entries.map(({ session, entry }) => [session.identity.nativeId, entry.ordinal]), - Array.from({ length: 5 }, (_, ordinal) => [nativeIdAt(ordinal), expectedEntryOrdinal]), + Array.from({ length: 50 }, (_, index) => [nativeIdAt(index), expectedEntryOrdinal]), ); - assert(page.nextCursor !== undefined, "selected entry page must have a continuation cursor"); + assert(page.nextCursor !== undefined, "filtered page must have a continuation"); page.entries.forEach(assertEntryFacts); } function assertEntryFacts(item: SessionEntryPage["entries"][number]): void { assert.deepStrictEqual(item.root, { kind: "known", root: item.session.identity }); - if (item.entry.ordinal === 3) { + if (item.entry.ordinal === 0) { assert.deepStrictEqual(item.content, { - textSegmentCount: 0, - omittedSegmentCount: 1, + textSegmentCount: 1, + omittedSegmentCount: 0, unpreviewedTextSegmentCount: 0, + preview: { + segmentOrdinal: 0, + origin: "injected", + originConfidence: "high", + contentHash: hashContent(SHARED_TEXT), + text: SHARED_TEXT, + truncated: false, + }, }); return; } - const text = expectedText(item.session.identity.nativeId, item.entry.ordinal); assert.deepStrictEqual(item.content, { - textSegmentCount: 1, + textSegmentCount: 0, omittedSegmentCount: 0, unpreviewedTextSegmentCount: 0, - preview: { - segmentOrdinal: 0, - origin: expectedOrigin(item.entry.ordinal), - originConfidence: "high", - contentHash: hashContent(text), - text, - truncated: false, - }, }); } -function expectedText(nativeId: string, entryOrdinal: number): string { - if (entryOrdinal === 0) return `Injected context ${nativeId}`; - if (entryOrdinal === 1) return `Initial request ${nativeId}`; - if (entryOrdinal === 2) return `Observed tool call ${nativeId}`; - if (entryOrdinal === 4) return `Final correction ${nativeId}`; - throw new Error("unexpected text entry ordinal"); -} - -function expectedOrigin(entryOrdinal: number): ContentOrigin { - if (entryOrdinal === 0) return "injected"; - if (entryOrdinal === 1 || entryOrdinal === 4) return "human"; - if (entryOrdinal === 2) return "model"; - throw new Error("unexpected text entry ordinal"); -} - -function openDatabase(): DatabaseSync { - const database = new DatabaseSync(":memory:", { +function openDatabase(path: string): DatabaseSync { + const database = new DatabaseSync(path, { allowExtension: false, defensive: true, enableDoubleQuotedStringLiterals: false, @@ -325,10 +517,39 @@ function nativeIdAt(ordinal: number): string { return `session-${String(ordinal).padStart(4, "0")}`; } -function timestampAt(ordinal: number): string { - return `2026-07-15T12:00:0${String(ordinal)}.000Z`; -} - function roundMilliseconds(value: number): number { return Number(value.toFixed(3)); } + +interface WorkResult { + readonly milliseconds: number; + readonly rows: number; +} + +interface PlanAccess { + readonly operation: string; + readonly relation: string; +} + +interface PlanFacts { + readonly outerAccess: PlanAccess; + readonly indexes: readonly string[]; + readonly usesOffset: boolean; + readonly usesKeyset: boolean; + readonly usesEntryOrdinalRange: boolean; + readonly usesTemporaryOrderBy: boolean; +} + +interface Measurement { + readonly totalMilliseconds: number; + readonly coordinateSelection: WorkResult; + readonly hydration: WorkResult; + readonly selectedEntries: number; + readonly hasContinuation: boolean; + readonly repeatedResultEqual: true; + readonly plan: PlanFacts; +} + +interface QueryPlanRow { + readonly detail: string; +} diff --git a/scripts/measure-query-planner-statistics.ts b/scripts/measure-query-planner-statistics.ts new file mode 100644 index 0000000..0855a76 --- /dev/null +++ b/scripts/measure-query-planner-statistics.ts @@ -0,0 +1,824 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { chmod, copyFile, lstat, mkdtemp, readFile, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { performance } from "node:perf_hooks"; +import { DatabaseSync } from "node:sqlite"; + +import { createDiscoveredSession } from "../src/application/source-input-fingerprint.ts"; +import type { SessionQueryRepository } from "../src/application/ports/session-query.ts"; +import { + admitSessionObservation, + admitSessionReplacement, + type ValidatedSessionReplacement, +} from "../src/application/validate-session.ts"; +import { hashContent } from "../src/domain/content-hash.ts"; +import { createSessionManifestQuery } from "../src/domain/session-manifest.ts"; +import { + createSessionEntryQuery, + createSessionListQuery, + createSessionSearchQuery, +} from "../src/domain/session-query.ts"; +import type { + ContentOrigin, + SessionDocument, + SessionEntry, + SessionIdentity, + SourceInstance, +} from "../src/domain/session.ts"; +import { + applyMigrations, + CURRENT_INDEX_SCHEMA_VERSION, +} from "../src/infrastructure/sqlite/migrations.ts"; +import { createCoordinatedSqliteSessionIndex } from "../src/infrastructure/sqlite/sqlite-session-index.ts"; +import { buildSqliteEntryCoordinateStatement } from "../src/infrastructure/sqlite/sqlite-session-entry-query.ts"; +import { createSqliteSessionQuery } from "../src/infrastructure/sqlite/sqlite-session-query.ts"; +import { + acquireWriterLease, + interruptOwnedRunsAndReleaseWriterLease, +} from "../src/infrastructure/sqlite/writer-lease.ts"; +import { initializeWriterRecoveryReceipt } from "../src/infrastructure/sqlite/writer-recovery-receipt.ts"; + +const CONTRACT_MODE = process.argv.includes("--contract"); +const CORPUS_SESSIONS = CONTRACT_MODE ? 24 : 600; +const HEAVY_SESSION_ENTRIES = CONTRACT_MODE ? 50 : 2_000; +const REGULAR_SESSION_ENTRIES = 3; +const TIMING_ROUNDS = CONTRACT_MODE ? 2 : 5; +const PAGE_LIMIT = 10; +const SOURCE: SourceInstance = Object.freeze({ + kind: "synthetic-measurement", + instanceId: "planner-statistics", +}); +const CAPTURED_AT = "2026-07-15T12:00:00.000Z"; +const RELEASED_AT = "2026-07-15T12:02:00.000Z"; +const OLD_ACTIVITY = "2026-07-14T12:00:00.000Z"; +const RECENT_ACTIVITY = "2026-07-15T12:00:00.000Z"; +const VARIANT_NAMES = ["control", "analyze", "optimize"] as const; +const PLAN_CASES = createPlanCases(); +const tempRoot = await mkdtemp(path.join(tmpdir(), "sessions-query-planner-statistics-")); +await chmod(tempRoot, 0o700); + +let report: MeasurementReport | undefined; +try { + report = await runMeasurement(tempRoot); +} finally { + await rm(tempRoot, { force: true, recursive: true }); +} +await assertRemoved(tempRoot); +assert(report !== undefined, "planner-statistics measurement did not produce a report"); +process.stdout.write(`${JSON.stringify({ ...report, temporaryCleanup: true })}\n`); + +async function runMeasurement(root: string): Promise { + const basePath = path.join(root, "base.sqlite"); + await createBaseDatabase(basePath); + await assertNoSidecars(basePath); + + const variantPaths = { + control: path.join(root, "control.sqlite"), + analyze: path.join(root, "analyze.sqlite"), + optimize: path.join(root, "optimize.sqlite"), + } as const; + const baseDigest = await fileDigest(basePath); + for (const variantPath of Object.values(variantPaths)) { + await copyFile(basePath, variantPath); + await chmod(variantPath, 0o600); + assert.equal(await fileDigest(variantPath), baseDigest, "database clone was not exact"); + } + + const beforeBytes = await fileBytes(basePath); + const mutationReports = { + control: await inspectVariant(variantPaths.control, "none", 0, beforeBytes), + analyze: await mutateAndInspect( + variantPaths.analyze, + "ANALYZE", + (database) => database.exec("ANALYZE"), + beforeBytes, + ), + optimize: await mutateAndInspect( + variantPaths.optimize, + "PRAGMA optimize = 0x10002", + (database) => { + database.prepare("PRAGMA optimize = 0x10002").all(); + }, + beforeBytes, + ), + } satisfies Record; + + const handles = Object.fromEntries( + VARIANT_NAMES.map((name) => [name, openReadDatabase(variantPaths[name])]), + ) as unknown as Record; + try { + const repositories = Object.fromEntries( + VARIANT_NAMES.map((name) => [name, createSqliteSessionQuery(handles[name])]), + ) as unknown as Record; + const cases = await measureCases(handles, repositories); + return { + schemaVersion: 1, + mode: CONTRACT_MODE ? "contract" : "full", + corpus: { + sessions: CORPUS_SESSIONS, + entries: corpusEntryCount(), + heavySessionEntries: HEAVY_SESSION_ENTRIES, + regularSessionEntries: REGULAR_SESSION_ENTRIES, + }, + clonesVerifiedExact: true, + explicitEntryTraversal: await hasExplicitEntryTraversal(), + timingRounds: TIMING_ROUNDS, + variants: mutationReports, + cases, + semanticEquality: Object.values(cases).every((measurement) => measurement.semanticEqual), + }; + } finally { + for (const database of Object.values(handles)) database.close(); + } +} + +async function createBaseDatabase(file: string): Promise { + const database = openWriteDatabase(file); + try { + applyMigrations(database); + await seedCorpus(database); + database.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get(); + } finally { + database.close(); + } + await chmod(file, 0o600); +} + +async function seedCorpus(database: DatabaseSync): Promise { + const now = () => new Date(CAPTURED_AT); + const lease = acquireWriterLease(database, "index", { + now, + token: () => "planner-statistics-writer", + }); + initializeWriterRecoveryReceipt(database, lease, { + now, + schemaVersion: CURRENT_INDEX_SCHEMA_VERSION, + }); + try { + const index = createCoordinatedSqliteSessionIndex(database, { + lease, + now, + schemaVersion: CURRENT_INDEX_SCHEMA_VERSION, + }); + const run = await index.startRun({ source: SOURCE, startedAt: CAPTURED_AT }); + for (let ordinal = 0; ordinal < CORPUS_SESSIONS; ordinal += 1) { + await index.replaceSession(run, replacement(ordinal)); + } + const result = await index.finishRun(run, { + status: "completed", + finishedAt: "2026-07-15T12:01:00.000Z", + }); + assert.deepStrictEqual(result.counts, { + discovered: CORPUS_SESSIONS, + unchanged: 0, + updated: CORPUS_SESSIONS, + failed: 0, + missing: 0, + stale: 0, + }); + } finally { + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date(RELEASED_AT), + }); + } +} + +function replacement(ordinal: number): ValidatedSessionReplacement { + const identity = identityAt(ordinal); + const nativeId = identity.nativeId; + const candidate = createDiscoveredSession({ + identity, + inputs: [ + { + role: "transcript", + locator: { uri: `memory://planner-statistics/${nativeId}` }, + fingerprint: `revision-${nativeId}`, + }, + ], + adapterVersion: "synthetic-v1", + }); + const observation = admitSessionObservation(candidate); + assert(observation.ok, "measurement observation was rejected"); + const admitted = admitSessionReplacement(observation.observation, documentAt(ordinal)); + assert(admitted.ok, "measurement document was rejected"); + return admitted.replacement; +} + +function documentAt(sessionOrdinal: number): SessionDocument { + const identity = identityAt(sessionOrdinal); + const entryCount = sessionOrdinal === 0 ? HEAVY_SESSION_ENTRIES : REGULAR_SESSION_ENTRIES; + return { + identity, + title: `Planner measurement ${identity.nativeId}`, + workspace: + sessionOrdinal % 10 === 0 ? "/generic/workspace/recent" : "/generic/workspace/common", + createdAt: OLD_ACTIVITY, + updatedAt: activityAt(sessionOrdinal), + lineageCoverage: "complete", + relations: [], + entries: Array.from({ length: entryCount }, (_, entryOrdinal) => + entryAt(sessionOrdinal, entryOrdinal, entryCount), + ), + }; +} + +function entryAt(sessionOrdinal: number, entryOrdinal: number, entryCount: number): SessionEntry { + const actor = actorAt(entryOrdinal); + const origin = originAt(actor); + const text = contentAt(sessionOrdinal, entryOrdinal, entryCount); + return { + ordinal: entryOrdinal, + kind: "message", + actor, + timestamp: activityAt(sessionOrdinal), + sourceLocator: { + uri: `memory://planner-statistics/${nativeIdAt(sessionOrdinal)}/entry/${String(entryOrdinal)}`, + }, + content: [ + { + kind: "text", + ordinal: 0, + text, + contentHash: hashContent(text), + origin, + originConfidence: "high", + sourceMetadata: {}, + }, + ], + }; +} + +function contentAt(sessionOrdinal: number, entryOrdinal: number, entryCount: number): string { + const alternate = (sessionOrdinal + entryOrdinal) % 2 === 0 ? "alternateeven" : "alternateodd"; + const rare = sessionOrdinal === 0 && entryOrdinal === entryCount - 1 ? " rare marker" : ""; + return `common evidence ${alternate} generic session ${String(sessionOrdinal)} entry ${String(entryOrdinal)}${rare}`; +} + +function actorAt(entryOrdinal: number): SessionEntry["actor"] { + if (entryOrdinal % 3 === 0) return "human"; + if (entryOrdinal % 3 === 1) return "model"; + return "tool"; +} + +function originAt(actor: SessionEntry["actor"]): ContentOrigin { + if (actor === "human" || actor === "model" || actor === "tool") return actor; + return "unknown"; +} + +function activityAt(sessionOrdinal: number): string { + return sessionOrdinal >= Math.floor(CORPUS_SESSIONS * 0.9) ? RECENT_ACTIVITY : OLD_ACTIVITY; +} + +function identityAt(ordinal: number): SessionIdentity { + return { source: SOURCE, nativeId: nativeIdAt(ordinal) }; +} + +function nativeIdAt(ordinal: number): string { + return `session-${String(ordinal).padStart(5, "0")}`; +} + +function corpusEntryCount(): number { + return HEAVY_SESSION_ENTRIES + (CORPUS_SESSIONS - 1) * REGULAR_SESSION_ENTRIES; +} + +async function mutateAndInspect( + file: string, + command: string, + mutate: (database: DatabaseSync) => void, + beforeBytes: number, +): Promise { + const database = openWriteDatabase(file); + const startedAt = performance.now(); + try { + mutate(database); + database.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get(); + } finally { + database.close(); + } + const applyElapsedMs = roundMilliseconds(performance.now() - startedAt); + await assertNoSidecars(file); + return inspectVariant(file, command, applyElapsedMs, beforeBytes); +} + +async function inspectVariant( + file: string, + command: string, + applyElapsedMs: number, + beforeBytes: number, +): Promise { + const database = openReadDatabase(file); + try { + const databaseBytes = await fileBytes(file); + return { + statisticsCommand: command, + applyElapsedMs, + databaseBytes, + databaseByteGrowth: databaseBytes - beforeBytes, + statistics: readStatistics(database), + }; + } finally { + database.close(); + } +} + +function readStatistics(database: DatabaseSync): StatisticsReport { + const tables = database + .prepare( + `SELECT name + FROM sqlite_schema + WHERE type = 'table' AND name GLOB 'sqlite_stat*' + ORDER BY name`, + ) + .all() as unknown as readonly { readonly name: unknown }[]; + return { + tables: tables.map((row) => { + if (typeof row.name !== "string") { + throw new TypeError("statistics table name was not text"); + } + const name = row.name; + assert(/^sqlite_stat[1-9][0-9]*$/u.test(name), "unexpected statistics table name"); + const rows = database.prepare(`SELECT * FROM ${name}`).all() as unknown as readonly Record< + string, + unknown + >[]; + return { + name, + rows: rows.length, + payloadBytes: rows.reduce( + (total, statisticsRow) => + total + + Object.values(statisticsRow).reduce( + (rowTotal, value) => rowTotal + sqliteValueBytes(value), + 0, + ), + 0, + ), + }; + }), + }; +} + +function sqliteValueBytes(value: unknown): number { + if (value === null) return 0; + if (typeof value === "string") return Buffer.byteLength(value); + if (typeof value === "number" || typeof value === "bigint") { + return Buffer.byteLength(String(value)); + } + if (value instanceof Uint8Array) return value.byteLength; + throw new TypeError("statistics table contained an unsupported SQLite value"); +} + +async function measureCases( + databases: Record, + repositories: Record, +): Promise> { + const result: Record = {}; + for (const measurementCase of createQueryCases()) { + const expected = await measurementCase.run(repositories.control); + const resultCount = countResult(expected); + for (const name of VARIANT_NAMES) { + const actual = await measurementCase.run(repositories[name]); + assert.deepStrictEqual( + actual, + expected, + `${measurementCase.name} changed under ${name} statistics`, + ); + } + + const samples = Object.fromEntries( + VARIANT_NAMES.map((name) => [name, [] as number[]]), + ) as unknown as Record; + for (let round = 0; round < TIMING_ROUNDS; round += 1) { + for (const name of rotateVariants(round)) { + const startedAt = performance.now(); + const actual = await measurementCase.run(repositories[name]); + samples[name].push(roundMilliseconds(performance.now() - startedAt)); + assert.deepStrictEqual( + actual, + expected, + `${measurementCase.name} changed during ${name} timing`, + ); + } + } + + const plans = Object.fromEntries( + VARIANT_NAMES.map((name) => [ + name, + normalizedPlan(databases[name], PLAN_CASES[measurementCase.name]), + ]), + ) as unknown as Record; + result[measurementCase.name] = { + semanticEqual: true, + resultCount, + elapsedMs: Object.fromEntries( + VARIANT_NAMES.map((name) => [name, summarize(samples[name])]), + ) as unknown as Record, + plans, + planChanged: { + analyze: !samePlan(plans.control, plans.analyze), + optimize: !samePlan(plans.control, plans.optimize), + }, + }; + } + return result; +} + +function createQueryCases(): readonly QueryCase[] { + const heavySession = identityAt(0); + return [ + { + name: "entries-broad", + run: (repository) => + repository.entries(createSessionEntryQuery({ selection: "all", limit: PAGE_LIMIT })), + }, + { + name: "entries-narrow", + run: (repository) => + repository.entries( + createSessionEntryQuery({ + filter: { session: heavySession, actor: "model" }, + selection: "all", + limit: PAGE_LIMIT, + }), + ), + }, + { + name: "list-identity", + run: (repository) => + repository.list( + createSessionListQuery({ filter: { session: heavySession }, limit: PAGE_LIMIT }), + ), + }, + { + name: "list-activity", + run: (repository) => + repository.list( + createSessionListQuery({ + filter: { activityAfter: "2026-07-14T23:59:59.999Z" }, + limit: PAGE_LIMIT, + }), + ), + }, + { + name: "search-broad-all", + run: (repository) => + repository.search( + createSessionSearchQuery({ + text: "common evidence", + termMode: "all", + limit: PAGE_LIMIT, + context: 0, + }), + ), + }, + { + name: "search-selective-all", + run: (repository) => + repository.search( + createSessionSearchQuery({ + text: "rare marker", + termMode: "all", + limit: PAGE_LIMIT, + context: 0, + }), + ), + }, + { + name: "search-broad-any", + run: (repository) => + repository.search( + createSessionSearchQuery({ + text: "alternateeven alternateodd", + termMode: "any", + limit: PAGE_LIMIT, + context: 0, + }), + ), + }, + { + name: "search-selective-any", + run: (repository) => + repository.search( + createSessionSearchQuery({ + text: "rare unavailable", + termMode: "any", + limit: PAGE_LIMIT, + context: 0, + }), + ), + }, + { + name: "manifest", + run: (repository) => repository.manifest(createSessionManifestQuery()), + }, + ]; +} + +function countResult(result: unknown): number { + if (typeof result !== "object" || result === null) { + throw new TypeError("query measurement result was not an object"); + } + for (const key of ["entries", "sessions", "hits", "revisions"] as const) { + if (key in result) { + const value = (result as Record)[key]; + if (Array.isArray(value)) return value.length; + } + } + throw new TypeError("query measurement result had no counted collection"); +} + +function createPlanCases(): Readonly> { + const identityParameters = [SOURCE.kind, SOURCE.instanceId, nativeIdAt(0)] as const; + const broadEntries = buildSqliteEntryCoordinateStatement( + createSessionEntryQuery({ selection: "all", limit: PAGE_LIMIT }), + { kind: "first" }, + ); + const narrowEntries = buildSqliteEntryCoordinateStatement( + createSessionEntryQuery({ + filter: { session: identityAt(0), actor: "model" }, + selection: "all", + limit: PAGE_LIMIT, + }), + { kind: "first" }, + ); + const listSql = `SELECT source.kind, source.instance_id, tracking.native_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE 1 = 1%s + ORDER BY CASE WHEN COALESCE(canonical.updated_at, canonical.created_at) IS NULL THEN 1 ELSE 0 END, + COALESCE(canonical.updated_at, canonical.created_at) DESC, + source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY + LIMIT ?`; + const searchSql = `SELECT canonical.session_id, entry.ordinal + FROM sessions_content_fts + JOIN sessions_content_values AS content + ON content.content_id = sessions_content_fts.rowid + JOIN sessions_content_occurrences AS occurrence + ON occurrence.content_id = content.content_id + JOIN sessions_entries AS entry + ON entry.session_id = occurrence.session_id + AND entry.ordinal = occurrence.entry_ordinal + JOIN sessions_canonical_sessions AS canonical + ON canonical.session_id = entry.session_id + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE sessions_content_fts MATCH ? + ORDER BY canonical.session_id, entry.ordinal + LIMIT ?`; + const manifestSql = `WITH cohort AS ( + SELECT canonical.session_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + ORDER BY source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY + LIMIT ? + ) + SELECT canonical.session_id + FROM cohort + JOIN sessions_canonical_sessions AS canonical + ON canonical.session_id = cohort.session_id + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + LEFT JOIN sessions_canonical_document_metrics AS metrics + ON metrics.session_id = canonical.session_id + ORDER BY canonical.session_id`; + + return { + "entries-broad": broadEntries, + "entries-narrow": { + sql: narrowEntries.sql, + parameters: narrowEntries.parameters, + }, + "list-identity": { + sql: listSql.replace( + "%s", + " AND source.kind = ? AND source.instance_id = ? AND tracking.native_id = ?", + ), + parameters: [...identityParameters, PAGE_LIMIT], + }, + "list-activity": { + sql: listSql.replace("%s", " AND COALESCE(canonical.updated_at, canonical.created_at) > ?"), + parameters: ["2026-07-14T23:59:59.999Z", PAGE_LIMIT], + }, + "search-broad-all": { + sql: searchSql, + parameters: ['"common" "evidence"', PAGE_LIMIT], + }, + "search-selective-all": { + sql: searchSql, + parameters: ['"rare" "marker"', PAGE_LIMIT], + }, + "search-broad-any": { + sql: searchSql, + parameters: ['"alternateeven" OR "alternateodd"', PAGE_LIMIT], + }, + "search-selective-any": { + sql: searchSql, + parameters: ['"rare" OR "unavailable"', PAGE_LIMIT], + }, + manifest: { + sql: manifestSql, + parameters: [10_001], + }, + }; +} + +function normalizedPlan(database: DatabaseSync, planCase: PlanCase | undefined): readonly string[] { + assert(planCase !== undefined, "query case has no representative plan"); + const rows = database + .prepare(`EXPLAIN QUERY PLAN ${planCase.sql}`) + .all(...planCase.parameters) as unknown as readonly { readonly detail: unknown }[]; + return rows.map((row) => { + if (typeof row.detail !== "string") throw new TypeError("query-plan detail was not text"); + return row.detail + .replace(/\b[0-9]+\b/gu, "?") + .replace(/\s+/gu, " ") + .trim(); + }); +} + +function rotateVariants(round: number): readonly VariantName[] { + const offset = round % VARIANT_NAMES.length; + return [...VARIANT_NAMES.slice(offset), ...VARIANT_NAMES.slice(0, offset)]; +} + +function summarize(values: readonly number[]): ElapsedAggregate { + assert(values.length > 0, "elapsed aggregate was empty"); + const sorted = [...values].sort((left, right) => left - right); + return { + samples: values, + median: percentile(sorted, 0.5), + p95: percentile(sorted, 0.95), + }; +} + +function percentile(sorted: readonly number[], fraction: number): number { + const index = Math.min(sorted.length - 1, Math.ceil(sorted.length * fraction) - 1); + return sorted[index]!; +} + +function samePlan(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((detail, index) => detail === right[index]); +} + +function openWriteDatabase(file: string): DatabaseSync { + const database = new DatabaseSync(file, { + allowExtension: false, + defensive: true, + enableDoubleQuotedStringLiterals: false, + enableForeignKeyConstraints: true, + timeout: 1_000, + }); + database.exec("PRAGMA trusted_schema = OFF"); + return database; +} + +function openReadDatabase(file: string): DatabaseSync { + const database = new DatabaseSync(file, { + allowExtension: false, + defensive: true, + enableDoubleQuotedStringLiterals: false, + enableForeignKeyConstraints: true, + readOnly: true, + timeout: 1_000, + }); + database.exec("PRAGMA trusted_schema = OFF"); + return database; +} + +async function hasExplicitEntryTraversal(): Promise { + const source = await readFile( + path.resolve(import.meta.dirname, "../src/infrastructure/sqlite/sqlite-session-entry-query.ts"), + "utf8", + ); + return /CROSS JOIN sessions_session_tracking/u.test(source); +} + +async function assertNoSidecars(file: string): Promise { + for (const suffix of ["-wal", "-shm"] as const) { + try { + await lstat(`${file}${suffix}`); + assert.fail(`closed measurement database retained a ${suffix} sidecar`); + } catch (error) { + if (isMissing(error)) continue; + throw error; + } + } +} + +async function assertRemoved(file: string): Promise { + try { + await lstat(file); + assert.fail("measurement temporary root was not removed"); + } catch (error) { + if (isMissing(error)) return; + throw error; + } +} + +function isMissing(error: unknown): boolean { + return ( + typeof error === "object" && + error !== null && + "code" in error && + (error as { readonly code?: unknown }).code === "ENOENT" + ); +} + +async function fileDigest(file: string): Promise { + return createHash("sha256") + .update(await readFile(file)) + .digest("hex"); +} + +async function fileBytes(file: string): Promise { + return (await stat(file)).size; +} + +function roundMilliseconds(value: number): number { + return Number(value.toFixed(3)); +} + +type VariantName = (typeof VARIANT_NAMES)[number]; + +interface VariantReport { + readonly statisticsCommand: string; + readonly applyElapsedMs: number; + readonly databaseBytes: number; + readonly databaseByteGrowth: number; + readonly statistics: StatisticsReport; +} + +interface StatisticsReport { + readonly tables: readonly { + readonly name: string; + readonly rows: number; + readonly payloadBytes: number; + }[]; +} + +interface ElapsedAggregate { + readonly samples: readonly number[]; + readonly median: number; + readonly p95: number; +} + +interface CaseReport { + readonly semanticEqual: boolean; + readonly resultCount: number; + readonly elapsedMs: Record; + readonly plans: Record; + readonly planChanged: { + readonly analyze: boolean; + readonly optimize: boolean; + }; +} + +interface MeasurementReport { + readonly schemaVersion: 1; + readonly mode: "contract" | "full"; + readonly corpus: { + readonly sessions: number; + readonly entries: number; + readonly heavySessionEntries: number; + readonly regularSessionEntries: number; + }; + readonly clonesVerifiedExact: boolean; + readonly explicitEntryTraversal: boolean; + readonly timingRounds: number; + readonly variants: Record; + readonly cases: Record; + readonly semanticEquality: boolean; +} + +interface QueryCase { + readonly name: + | "entries-broad" + | "entries-narrow" + | "list-identity" + | "list-activity" + | "search-broad-all" + | "search-selective-all" + | "search-broad-any" + | "search-selective-any" + | "manifest"; + readonly run: (repository: SessionQueryRepository) => Promise; +} + +interface PlanCase { + readonly sql: string; + readonly parameters: readonly (string | number)[]; +} diff --git a/scripts/measure-search-query.ts b/scripts/measure-search-query.ts index d984ffb..38cb2b0 100644 --- a/scripts/measure-search-query.ts +++ b/scripts/measure-search-query.ts @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import { performance } from "node:perf_hooks"; -import { DatabaseSync } from "node:sqlite"; +import { constants, DatabaseSync } from "node:sqlite"; import { createDiscoveredSession } from "../src/application/source-input-fingerprint.ts"; import { @@ -16,6 +16,7 @@ import { CURRENT_INDEX_SCHEMA_VERSION, } from "../src/infrastructure/sqlite/migrations.ts"; import { createCoordinatedSqliteSessionIndex } from "../src/infrastructure/sqlite/sqlite-session-index.ts"; +import { SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE } from "../src/infrastructure/sqlite/sqlite-query-context.ts"; import { createSqliteSessionQuery } from "../src/infrastructure/sqlite/sqlite-session-query.ts"; import { acquireWriterLease, @@ -24,7 +25,9 @@ import { import { initializeWriterRecoveryReceipt } from "../src/infrastructure/sqlite/writer-recovery-receipt.ts"; const CORPUS_SESSIONS = 2_000; -const PAGE_LIMIT = 5; +const SMALL_PAGE_LIMIT = 5; +const LARGE_PAGE_LIMIT = 200; +const CONTEXT_LIMIT = 2; const SEARCH_TERM = "searchmeasurement"; const ANY_TERMS = ["alternateeven", "alternateodd"] as const; const SOURCE: SourceInstance = Object.freeze({ @@ -38,26 +41,19 @@ const database = openDatabase(); try { await seedCorpus(database); const repository = createSqliteSessionQuery(database); - const allQuery = createSessionSearchQuery({ - text: SEARCH_TERM, - limit: PAGE_LIMIT, - context: 0, - }); - const anyQuery = createSessionSearchQuery({ - text: ANY_TERMS.join(" "), - termMode: "any", - limit: PAGE_LIMIT, - context: 0, - }); - const elapsedMilliseconds = { - all: await measure(repository.search, allQuery, (page) => assertExpectedPage(page, "all")), - any: await measure(repository.search, anyQuery, (page) => assertExpectedPage(page, "any")), + all: await measureProfiles(database, repository.search, "all"), + any: await measureProfiles(database, repository.search, "any"), }; process.stdout.write( `${JSON.stringify({ corpusSessions: CORPUS_SESSIONS, + profiles: { + small: { limit: SMALL_PAGE_LIMIT, context: 0 }, + large: { limit: LARGE_PAGE_LIMIT, context: 0 }, + largeContext: { limit: LARGE_PAGE_LIMIT, context: CONTEXT_LIMIT }, + }, elapsedMs: elapsedMilliseconds, })}\n`, ); @@ -65,11 +61,38 @@ try { database.close(); } +async function measureProfiles( + database: DatabaseSync, + run: (query: ReturnType) => Promise, + mode: "all" | "any", +): Promise<{ + readonly small: ProfileMeasurement; + readonly large: ProfileMeasurement; + readonly largeContext: ProfileMeasurement; +}> { + const text = mode === "all" ? SEARCH_TERM : ANY_TERMS.join(" "); + const profile = async (limit: number, context: number): Promise => { + const query = createSessionSearchQuery({ + text, + ...(mode === "all" ? {} : { termMode: "any" }), + limit, + context, + }); + return measure(database, run, query, (page) => assertExpectedPage(page, mode, limit, context)); + }; + return { + small: await profile(SMALL_PAGE_LIMIT, 0), + large: await profile(LARGE_PAGE_LIMIT, 0), + largeContext: await profile(LARGE_PAGE_LIMIT, CONTEXT_LIMIT), + }; +} + async function measure( + database: DatabaseSync, run: (query: ReturnType) => Promise, query: ReturnType, assertPage: (page: SessionSearchPage) => void, -): Promise { +): Promise { const warm = await run(query); assertPage(warm); const startedAt = performance.now(); @@ -77,7 +100,53 @@ async function measure( const elapsedMilliseconds = performance.now() - startedAt; assertPage(repeated); assert.deepStrictEqual(repeated, warm, "repeated broad search changed its result"); - return roundMilliseconds(elapsedMilliseconds); + let selectAuthorizations = 0; + database.setAuthorizer((actionCode) => { + if (actionCode === constants.SQLITE_SELECT) selectAuthorizations += 1; + return constants.SQLITE_OK; + }); + let observed: SessionSearchPage; + try { + observed = await run(query); + } finally { + database.setAuthorizer(null); + } + assert.deepStrictEqual(observed, warm, "observed broad search changed its result"); + const selectedContent = new Set( + observed.hits.map( + ({ snippet }) => `${snippet.contentHash.scheme}:${snippet.contentHash.digest}`, + ), + ).size; + const distinctSessions = new Set(observed.hits.map(({ session }) => session.identity.nativeId)) + .size; + const contextCoordinates = new Set( + observed.hits.flatMap((hit) => + hit.context.map(({ ordinal }) => `${hit.session.identity.nativeId}:${String(ordinal)}`), + ), + ).size; + return { + elapsedMilliseconds: roundMilliseconds(elapsedMilliseconds), + work: { + selectedContent, + distinctSessions, + selectedContextCoordinates: contextCoordinates, + contextHydrationChunks: Math.ceil( + contextCoordinates / SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE, + ), + selectAuthorizations, + }, + }; +} + +interface ProfileMeasurement { + readonly elapsedMilliseconds: number; + readonly work: { + readonly selectedContent: number; + readonly distinctSessions: number; + readonly selectedContextCoordinates: number; + readonly contextHydrationChunks: number; + readonly selectAuthorizations: number; + }; } async function seedCorpus(database: DatabaseSync): Promise { @@ -168,12 +237,46 @@ function documentAt(ordinal: number): SessionDocument { }, ], }, + contextEntryAt(ordinal, 1), + contextEntryAt(ordinal, 2), + ], + }; +} + +function contextEntryAt( + sessionOrdinal: number, + entryOrdinal: number, +): SessionDocument["entries"][number] { + const text = contextTextAt(sessionOrdinal, entryOrdinal); + return { + ordinal: entryOrdinal, + kind: "message", + actor: "model", + timestamp: CAPTURED_AT, + sourceLocator: { + uri: `memory://search-measurement/${nativeIdAt(sessionOrdinal)}/entry/${String(entryOrdinal)}`, + }, + content: [ + { + kind: "text", + ordinal: 0, + text, + contentHash: hashContent(text), + origin: "model", + originConfidence: "high", + sourceMetadata: {}, + }, ], }; } -function assertExpectedPage(page: SessionSearchPage, mode: "all" | "any"): void { - const expectedNativeIds = Array.from({ length: PAGE_LIMIT }, (_, ordinal) => nativeIdAt(ordinal)); +function assertExpectedPage( + page: SessionSearchPage, + mode: "all" | "any", + pageLimit: number, + contextLimit: number, +): void { + const expectedNativeIds = Array.from({ length: pageLimit }, (_, ordinal) => nativeIdAt(ordinal)); assert.deepStrictEqual( page.hits.map(({ session }) => session.identity.nativeId), expectedNativeIds, @@ -203,7 +306,21 @@ function assertExpectedPage(page: SessionSearchPage, mode: "all" | "any"): void truncated: false, additionalMatchingSegments: 0, }); - assert.deepStrictEqual(hit.context, []); + assert.deepStrictEqual( + hit.context, + contextLimit === 0 + ? [] + : [1, 2].map((entryOrdinal) => ({ + ordinal: entryOrdinal, + kind: "message", + actor: "model", + timestamp: CAPTURED_AT, + body: contextTextAt(ordinal, entryOrdinal), + bodyTruncated: false, + adjacent: true, + linked: false, + })), + ); assert.equal(hit.linkedContextTruncated, false); } } @@ -232,6 +349,10 @@ function textAt(ordinal: number): string { return `${SEARCH_TERM} ${ANY_TERMS[ordinal % ANY_TERMS.length]} generic evidence ${nativeIdAt(ordinal)}`; } +function contextTextAt(sessionOrdinal: number, entryOrdinal: number): string { + return `context ${nativeIdAt(sessionOrdinal)} entry ${String(entryOrdinal)}`; +} + function roundMilliseconds(value: number): number { return Number(value.toFixed(3)); } diff --git a/src/bin/sessions.ts b/src/bin/sessions.ts index 065dd79..ca3dada 100644 --- a/src/bin/sessions.ts +++ b/src/bin/sessions.ts @@ -3,55 +3,25 @@ import { createRequire } from "node:module"; import { homedir } from "node:os"; -import { createCodexSource } from "../adapters/codex/source.ts"; -import { createCursorSource } from "../adapters/cursor/source.ts"; -import { clearData } from "../application/clear-index.ts"; -import { compactIndex } from "../application/compact-index.ts"; -import { createSessionManifest } from "../application/create-session-manifest.ts"; import type { DoctorProgressObserver } from "../application/doctor-progress.ts"; -import { timeDoctorOperation } from "../application/doctor-timing.ts"; -import { exportSession } from "../application/export-session.ts"; -import { repairOrphanedContent } from "../application/repair-orphaned-content.ts"; -import { forgetSession } from "../application/forget-session.ts"; -import { getPaths } from "../application/get-paths.ts"; -import { timeIndexOperation } from "../application/index-timing.ts"; -import { listSessions } from "../application/list-sessions.ts"; -import { listSessionEntries } from "../application/list-session-entries.ts"; -import { runDoctor } from "../application/run-doctor.ts"; -import { runIndex } from "../application/run-index.ts"; -import { isIndexInterruptedError } from "../application/index-interruption.ts"; import type { IndexProgressObserver } from "../application/index-progress.ts"; -import { searchSessions } from "../application/search-sessions.ts"; -import { showSession } from "../application/show-session.ts"; -import { createSourceDiagnostic } from "../application/source-diagnostic.ts"; +import type { ProgramOptions } from "../cli/program.ts"; import { CliSignalExit, runCli } from "../cli/run.ts"; -import { createNodeDiagnostic } from "../infrastructure/runtime/node-diagnostic.ts"; -import { createIndexStateDiagnostic } from "../infrastructure/state/index-state-diagnostic.ts"; -import { resolveIndexPaths } from "../infrastructure/state/paths.ts"; -import { createSqliteIndexLifecycle } from "../infrastructure/sqlite/database.ts"; -import { createSqliteDiagnostic } from "../infrastructure/sqlite/sqlite-diagnostic.ts"; -import { createSqliteIndexMaintenance } from "../infrastructure/sqlite/index-maintenance.ts"; -import { - createIndexTimingCollector, - encodeIndexTimingDiagnostic, -} from "../infrastructure/runtime/index-timings.ts"; -import { - createDoctorTimingCollector, - encodeDoctorTimingDiagnostic, -} from "../infrastructure/runtime/doctor-timings.ts"; -import { installIndexInterrupt } from "../infrastructure/runtime/index-interrupt.ts"; const require = createRequire(import.meta.url); const manifest = require("../../package.json") as { version?: unknown }; const version = typeof manifest.version === "string" ? manifest.version : "0.0.0"; -const indexLifecycle = createSqliteIndexLifecycle(); -const maintenance = createSqliteIndexMaintenance(); const indexTimingsEnabled = process.env.SESSIONS_INDEX_TIMINGS === "1"; const doctorTimingsEnabled = process.env.SESSIONS_DOCTOR_TIMINGS === "1"; -let codexSource: ReturnType | undefined; -let cursorSource: ReturnType | undefined; -const resolveCodexSource = () => (codexSource ??= createCodexSource()); -const resolveCursorSource = () => (cursorSource ??= createCursorSource()); + +const resolveCodexSource = memoizeAsync(async () => { + const { createCodexSource } = await import("../adapters/codex/source.ts"); + return createCodexSource(); +}); +const resolveCursorSource = memoizeAsync(async () => { + const { createCursorSource } = await import("../adapters/cursor/source.ts"); + return createCursorSource(); +}); const registeredSources = Object.freeze([ Object.freeze({ kind: "codex", resolve: resolveCodexSource }), Object.freeze({ kind: "cursor", resolve: resolveCursorSource }), @@ -65,32 +35,73 @@ const resolveIndexSources = async (kind: string | undefined) => { if (selected.length === 0) throw new TypeError("Unknown session source"); return Promise.all(selected.map((source) => source.resolve())); }; -const resolvePaths = () => - resolveIndexPaths({ + +const loadPathResolver = memoizeAsync(() => import("../infrastructure/state/paths.ts")); +const resolvePaths = async () => { + const { resolveIndexPaths } = await loadPathResolver(); + return resolveIndexPaths({ platform: process.platform, env: process.env, homeDirectory: homedir(), }); +}; +const resolveIndexLifecycle = memoizeAsync(async () => { + const { createSqliteIndexLifecycle } = await import("../infrastructure/sqlite/database.ts"); + return createSqliteIndexLifecycle(); +}); +const resolveMaintenance = memoizeAsync(async () => { + const { createSqliteIndexMaintenance } = + await import("../infrastructure/sqlite/index-maintenance.ts"); + return createSqliteIndexMaintenance(); +}); + +const loadIndexComposition = memoizeAsync(async () => { + const [ + { isIndexInterruptedError }, + { runIndex }, + { timeIndexOperation }, + { createIndexTimingCollector, encodeIndexTimingDiagnostic }, + { installIndexInterrupt }, + ] = await Promise.all([ + import("../application/index-interruption.ts"), + import("../application/run-index.ts"), + import("../application/index-timing.ts"), + import("../infrastructure/runtime/index-timings.ts"), + import("../infrastructure/runtime/index-interrupt.ts"), + ]); + return { + isIndexInterruptedError, + runIndex, + timeIndexOperation, + createIndexTimingCollector, + encodeIndexTimingDiagnostic, + installIndexInterrupt, + }; +}); const executeIndex = async ( source: string | undefined, signal: AbortSignal, progress: IndexProgressObserver | undefined, ) => { - const collector = indexTimingsEnabled ? createIndexTimingCollector() : undefined; + const [composition, lifecycle] = await Promise.all([ + loadIndexComposition(), + resolveIndexLifecycle(), + ]); + const collector = indexTimingsEnabled ? composition.createIndexTimingCollector() : undefined; try { const execute = async () => { - const paths = resolvePaths(); + const paths = await resolvePaths(); const sources = collector - ? await timeIndexOperation(collector.recorder, "sourceResolution", () => + ? await composition.timeIndexOperation(collector.recorder, "sourceResolution", async () => resolveIndexSources(source), ) : await resolveIndexSources(source); - return runIndex({ + return composition.runIndex({ paths, sources, sourceSelection: source === undefined ? "optional" : "required", - lifecycle: indexLifecycle, + lifecycle, clock: { now: () => new Date() }, signal, ...(progress === undefined ? {} : { progress }), @@ -99,11 +110,11 @@ const executeIndex = async ( }; return collector === undefined ? await execute() - : await timeIndexOperation(collector.recorder, "total", execute); + : await composition.timeIndexOperation(collector.recorder, "total", execute); } finally { if (collector !== undefined) { try { - process.stderr.write(encodeIndexTimingDiagnostic(collector.snapshot())); + process.stderr.write(composition.encodeIndexTimingDiagnostic(collector.snapshot())); } catch { // Opt-in timing output is best-effort and must not change indexing. } @@ -115,11 +126,12 @@ const indexSessions = async ( source: string | undefined, options: { readonly progress?: IndexProgressObserver } = {}, ) => { - const interrupt = installIndexInterrupt(); + const composition = await loadIndexComposition(); + const interrupt = composition.installIndexInterrupt(); try { return await executeIndex(source, interrupt.signal, options.progress); } catch (error) { - if (isIndexInterruptedError(error) && interrupt.interruption !== undefined) { + if (composition.isIndexInterruptedError(error) && interrupt.interruption !== undefined) { throw new CliSignalExit(interrupt.interruption.exitCode); } throw error; @@ -128,31 +140,76 @@ const indexSessions = async ( } }; +const loadDoctorComposition = memoizeAsync(async () => { + const [ + { runDoctor }, + { timeDoctorOperation }, + { createSourceDiagnostic }, + { createDoctorTimingCollector, encodeDoctorTimingDiagnostic }, + { createNodeDiagnostic }, + { createIndexStateDiagnostic }, + { createSqliteDiagnostic }, + ] = await Promise.all([ + import("../application/run-doctor.ts"), + import("../application/doctor-timing.ts"), + import("../application/source-diagnostic.ts"), + import("../infrastructure/runtime/doctor-timings.ts"), + import("../infrastructure/runtime/node-diagnostic.ts"), + import("../infrastructure/state/index-state-diagnostic.ts"), + import("../infrastructure/sqlite/sqlite-diagnostic.ts"), + ]); + return { + runDoctor, + timeDoctorOperation, + createSourceDiagnostic, + createDoctorTimingCollector, + encodeDoctorTimingDiagnostic, + createNodeDiagnostic, + createIndexStateDiagnostic, + createSqliteDiagnostic, + }; +}); + const doctorSessions = async (options: { readonly progress?: DoctorProgressObserver } = {}) => { - const collector = doctorTimingsEnabled ? createDoctorTimingCollector() : undefined; + const [composition, lifecycle, { resolveIndexPaths }] = await Promise.all([ + loadDoctorComposition(), + resolveIndexLifecycle(), + loadPathResolver(), + ]); + const collector = doctorTimingsEnabled ? composition.createDoctorTimingCollector() : undefined; try { const execute = async () => { const sources = collector === undefined ? await resolveAllSources() - : await timeDoctorOperation(collector.recorder, "sourceResolution", resolveAllSources); - return runDoctor([ - createNodeDiagnostic(), - createSqliteDiagnostic(), - createIndexStateDiagnostic(resolvePaths, indexLifecycle, { + : await composition.timeDoctorOperation( + collector.recorder, + "sourceResolution", + resolveAllSources, + ); + const resolveDoctorPaths = () => + resolveIndexPaths({ + platform: process.platform, + env: process.env, + homeDirectory: homedir(), + }); + return composition.runDoctor([ + composition.createNodeDiagnostic(), + composition.createSqliteDiagnostic(), + composition.createIndexStateDiagnostic(resolveDoctorPaths, lifecycle, { ...(options.progress === undefined ? {} : { progress: options.progress }), ...(collector === undefined ? {} : { timing: collector.recorder }), }), - ...sources.map(createSourceDiagnostic), + ...sources.map(composition.createSourceDiagnostic), ]); }; return collector === undefined ? await execute() - : await timeDoctorOperation(collector.recorder, "total", execute); + : await composition.timeDoctorOperation(collector.recorder, "total", execute); } finally { if (collector !== undefined) { try { - process.stderr.write(encodeDoctorTimingDiagnostic(collector.snapshot())); + process.stderr.write(composition.encodeDoctorTimingDiagnostic(collector.snapshot())); } catch { // Opt-in timing output is best-effort and must not change doctor. } @@ -160,7 +217,41 @@ const doctorSessions = async (options: { readonly progress?: DoctorProgressObser } }; -const exitCode = await runCli(process.argv.slice(2), { +const loadGetPaths = memoizeAsync( + async () => (await import("../application/get-paths.ts")).getPaths, +); +const loadListSessions = memoizeAsync( + async () => (await import("../application/list-sessions.ts")).listSessions, +); +const loadCreateSessionManifest = memoizeAsync( + async () => (await import("../application/create-session-manifest.ts")).createSessionManifest, +); +const loadListSessionEntries = memoizeAsync( + async () => (await import("../application/list-session-entries.ts")).listSessionEntries, +); +const loadSearchSessions = memoizeAsync( + async () => (await import("../application/search-sessions.ts")).searchSessions, +); +const loadShowSession = memoizeAsync( + async () => (await import("../application/show-session.ts")).showSession, +); +const loadExportSession = memoizeAsync( + async () => (await import("../application/export-session.ts")).exportSession, +); +const loadForgetSession = memoizeAsync( + async () => (await import("../application/forget-session.ts")).forgetSession, +); +const loadClearData = memoizeAsync( + async () => (await import("../application/clear-index.ts")).clearData, +); +const loadCompactIndex = memoizeAsync( + async () => (await import("../application/compact-index.ts")).compactIndex, +); +const loadRepairOrphanedContent = memoizeAsync( + async () => (await import("../application/repair-orphaned-content.ts")).repairOrphanedContent, +); + +const programOptions: ProgramOptions = { version, output: { stderrIsInteractive: process.stderr.isTTY === true && process.env.TERM !== "dumb", @@ -168,68 +259,146 @@ const exitCode = await runCli(process.argv.slice(2), { writeErr: (text) => process.stderr.write(text), }, doctor: doctorSessions, - paths: async () => getPaths(resolvePaths(), indexLifecycle, await resolveAllSources()), + paths: async () => { + const [getPaths, paths, lifecycle, sources] = await Promise.all([ + loadGetPaths(), + resolvePaths(), + resolveIndexLifecycle(), + resolveAllSources(), + ]); + return getPaths(paths, lifecycle, sources); + }, indexSources: registeredSources.map(({ kind }) => kind), index: indexSessions, - list: ({ filter, limit, cursor }) => - listSessions({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + list: async ({ filter, limit, cursor }) => { + const [listSessions, paths, lifecycle] = await Promise.all([ + loadListSessions(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return listSessions({ + paths, + lifecycle, ...(filter === undefined ? {} : { filter }), ...(limit === undefined ? {} : { limit }), ...(cursor === undefined ? {} : { cursor }), - }), - manifest: ({ filter }) => - createSessionManifest({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + }); + }, + manifest: async ({ filter }) => { + const [createSessionManifest, paths, lifecycle] = await Promise.all([ + loadCreateSessionManifest(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return createSessionManifest({ + paths, + lifecycle, ...(filter === undefined ? {} : { filter }), - }), - entries: ({ filter, selection, limit, cursor }) => - listSessionEntries({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + }); + }, + entries: async ({ filter, selection, limit, cursor }) => { + const [listSessionEntries, paths, lifecycle] = await Promise.all([ + loadListSessionEntries(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return listSessionEntries({ + paths, + lifecycle, ...(filter === undefined ? {} : { filter }), ...(selection === undefined ? {} : { selection }), ...(limit === undefined ? {} : { limit }), ...(cursor === undefined ? {} : { cursor }), - }), - search: ({ text, termMode, filter, limit, context, cursor }) => - searchSessions({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + }); + }, + search: async ({ text, termMode, filter, limit, context, cursor }) => { + const [searchSessions, paths, lifecycle] = await Promise.all([ + loadSearchSessions(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return searchSessions({ + paths, + lifecycle, text, ...(termMode === undefined ? {} : { termMode }), ...(filter === undefined ? {} : { filter }), ...(limit === undefined ? {} : { limit }), ...(context === undefined ? {} : { context }), ...(cursor === undefined ? {} : { cursor }), - }), - show: ({ identity, expectedDocumentDigest, entry, context, fromEntry, toEntry }) => - showSession({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + }); + }, + show: async ({ identity, expectedDocumentDigest, entry, context, fromEntry, toEntry }) => { + const [showSession, paths, lifecycle] = await Promise.all([ + loadShowSession(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return showSession({ + paths, + lifecycle, identity, ...(expectedDocumentDigest === undefined ? {} : { expectedDocumentDigest }), ...(entry === undefined ? {} : { entry }), ...(context === undefined ? {} : { context }), ...(fromEntry === undefined ? {} : { fromEntry }), ...(toEntry === undefined ? {} : { toEntry }), - }), - export: ({ identity, expectedDocumentDigest, full, fromEntry, toEntry }) => - exportSession({ - paths: resolvePaths(), - lifecycle: indexLifecycle, + }); + }, + export: async ({ identity, expectedDocumentDigest, full, fromEntry, toEntry }) => { + const [exportSession, paths, lifecycle] = await Promise.all([ + loadExportSession(), + resolvePaths(), + resolveIndexLifecycle(), + ]); + return exportSession({ + paths, + lifecycle, identity, ...(expectedDocumentDigest === undefined ? {} : { expectedDocumentDigest }), ...(full === undefined ? {} : { full }), ...(fromEntry === undefined ? {} : { fromEntry }), ...(toEntry === undefined ? {} : { toEntry }), - }), - forget: (identity) => forgetSession(resolvePaths(), maintenance, identity), - clearData: () => clearData(resolvePaths(), maintenance), - compactData: () => compactIndex(resolvePaths(), maintenance), - repairOrphanedData: () => repairOrphanedContent(resolvePaths(), maintenance), -}); + }); + }, + forget: async (identity) => { + const [forgetSession, paths, maintenance] = await Promise.all([ + loadForgetSession(), + resolvePaths(), + resolveMaintenance(), + ]); + return forgetSession(paths, maintenance, identity); + }, + clearData: async () => { + const [clearData, paths, maintenance] = await Promise.all([ + loadClearData(), + resolvePaths(), + resolveMaintenance(), + ]); + return clearData(paths, maintenance); + }, + compactData: async () => { + const [compactIndex, paths, maintenance] = await Promise.all([ + loadCompactIndex(), + resolvePaths(), + resolveMaintenance(), + ]); + return compactIndex(paths, maintenance); + }, + repairOrphanedData: async () => { + const [repairOrphanedContent, paths, maintenance] = await Promise.all([ + loadRepairOrphanedContent(), + resolvePaths(), + resolveMaintenance(), + ]); + return repairOrphanedContent(paths, maintenance); + }, +}; +const exitCode = await runCli(process.argv.slice(2), programOptions); process.exitCode = exitCode; + +function memoizeAsync(load: () => Promise): () => Promise { + let pending: Promise | undefined; + return () => (pending ??= load()); +} diff --git a/src/infrastructure/sqlite/query-cursor.ts b/src/infrastructure/sqlite/query-cursor.ts index 6e0dbdf..226c464 100644 --- a/src/infrastructure/sqlite/query-cursor.ts +++ b/src/infrastructure/sqlite/query-cursor.ts @@ -1,7 +1,8 @@ import { createHash } from "node:crypto"; import type { DatabaseSync } from "node:sqlite"; -const CURSOR_VERSION = 1; +const CURSOR_VERSION_V1 = 1; +const CURSOR_VERSION_V2 = 2; const MAX_CURSOR_BYTES = 2_048; const INSTANCE_ID_PATTERN = /^[a-f0-9]{32}$/u; const FINGERPRINT_PATTERN = /^[a-f0-9]{64}$/u; @@ -13,8 +14,19 @@ export interface QueryRevision { readonly writerGeneration: number; } +export type QueryCursorAnchor = + | { + readonly kind: "list"; + readonly sessionId: number; + } + | { + readonly kind: "entries"; + readonly sessionId: number; + readonly entryOrdinal: number; + }; + export type CursorDecodeResult = - | { readonly ok: true; readonly offset: number } + | { readonly ok: true; readonly offset: number; readonly anchor?: QueryCursorAnchor } | { readonly ok: false; readonly reason: "invalid" | "mismatch" | "stale" }; export function readQueryRevision(database: DatabaseSync): QueryRevision { @@ -59,7 +71,7 @@ export function encodeQueryCursor(input: { assertOffset(input.offset); return Buffer.from( JSON.stringify({ - v: CURSOR_VERSION, + v: CURSOR_VERSION_V1, c: input.command, q: input.fingerprint, l: input.revision.libraryInstanceId, @@ -70,6 +82,61 @@ export function encodeQueryCursor(input: { ).toString("base64url"); } +export function encodeAnchoredQueryCursor( + input: + | { + readonly command: "list"; + readonly fingerprint: string; + readonly revision: QueryRevision; + readonly offset: number; + readonly anchor: Extract; + } + | { + readonly command: "entries"; + readonly fingerprint: string; + readonly revision: QueryRevision; + readonly offset: number; + readonly anchor: Extract; + }, +): string { + assertAnchoredCommand(input.command); + assertAnchorKind(input.anchor.kind); + if (!FINGERPRINT_PATTERN.test(input.fingerprint)) + throw new TypeError("Invalid query fingerprint"); + assertRevision(input.revision); + assertOffset(input.offset); + assertOffset(input.anchor.sessionId); + if (input.command !== input.anchor.kind) { + throw new TypeError("Query cursor anchor does not match command"); + } + if (input.command === "entries") { + assertOffset(input.anchor.entryOrdinal); + } + + const payload = + input.command === "list" + ? { + v: CURSOR_VERSION_V2, + c: input.command, + q: input.fingerprint, + l: input.revision.libraryInstanceId, + g: input.revision.writerGeneration, + o: input.offset, + s: input.anchor.sessionId, + } + : { + v: CURSOR_VERSION_V2, + c: input.command, + q: input.fingerprint, + l: input.revision.libraryInstanceId, + g: input.revision.writerGeneration, + o: input.offset, + s: input.anchor.sessionId, + e: input.anchor.entryOrdinal, + }; + return Buffer.from(JSON.stringify(payload), "utf8").toString("base64url"); +} + export function decodeQueryCursor( cursor: string, expected: { @@ -96,12 +163,8 @@ export function decodeQueryCursor( } catch { return invalid(); } - if (!isCursorPayload(parsed)) return invalid(); - if ( - parsed.v !== CURSOR_VERSION || - parsed.c !== expected.command || - parsed.q !== expected.fingerprint - ) { + if (!isCursorPayloadV1(parsed) && !isCursorPayloadV2(parsed)) return invalid(); + if (parsed.c !== expected.command || parsed.q !== expected.fingerprint) { return { ok: false, reason: "mismatch" }; } if ( @@ -110,10 +173,24 @@ export function decodeQueryCursor( ) { return { ok: false, reason: "stale" }; } - return { ok: true, offset: parsed.o }; + if (parsed.v === CURSOR_VERSION_V1) { + return { ok: true, offset: parsed.o }; + } + return { + ok: true, + offset: parsed.o, + anchor: + parsed.c === "list" + ? { kind: "list", sessionId: parsed.s } + : { + kind: "entries", + sessionId: parsed.s, + entryOrdinal: parsed.e, + }, + }; } -function isCursorPayload(value: unknown): value is CursorPayload { +function isCursorPayloadV1(value: unknown): value is CursorPayloadV1 { if (typeof value !== "object" || value === null || Array.isArray(value)) return false; const record = value as Record; if ( @@ -128,7 +205,7 @@ function isCursorPayload(value: unknown): value is CursorPayload { return false; } return ( - record.v === CURSOR_VERSION && + record.v === CURSOR_VERSION_V1 && (record.c === "entries" || record.c === "list" || record.c === "search") && typeof record.q === "string" && FINGERPRINT_PATTERN.test(record.q) && @@ -139,6 +216,34 @@ function isCursorPayload(value: unknown): value is CursorPayload { ); } +function isCursorPayloadV2(value: unknown): value is CursorPayloadV2 { + if (typeof value !== "object" || value === null || Array.isArray(value)) return false; + const record = value as Record; + if (record.v !== CURSOR_VERSION_V2 || (record.c !== "entries" && record.c !== "list")) { + return false; + } + const expectedKeys = + record.c === "list" + ? ["v", "c", "q", "l", "g", "o", "s"] + : ["v", "c", "q", "l", "g", "o", "s", "e"]; + if ( + Object.keys(record).length !== expectedKeys.length || + expectedKeys.some((key) => !Object.hasOwn(record, key)) + ) { + return false; + } + return ( + typeof record.q === "string" && + FINGERPRINT_PATTERN.test(record.q) && + typeof record.l === "string" && + INSTANCE_ID_PATTERN.test(record.l) && + isSafeNonNegativeInteger(record.g) && + isSafeNonNegativeInteger(record.o) && + isSafeNonNegativeInteger(record.s) && + (record.c === "list" || isSafeNonNegativeInteger(record.e)) + ); +} + function assertRevision(revision: QueryRevision): void { if ( !INSTANCE_ID_PATTERN.test(revision.libraryInstanceId) || @@ -154,6 +259,18 @@ function assertCommand(command: string): asserts command is QueryCommand { } } +function assertAnchoredCommand(command: string): asserts command is "entries" | "list" { + if (command !== "entries" && command !== "list") { + throw new TypeError("Invalid anchored query command"); + } +} + +function assertAnchorKind(kind: string): asserts kind is QueryCursorAnchor["kind"] { + if (kind !== "entries" && kind !== "list") { + throw new TypeError("Invalid query cursor anchor kind"); + } +} + function assertOffset(value: number): void { if (!isSafeNonNegativeInteger(value)) throw new TypeError("Invalid query offset"); } @@ -172,7 +289,7 @@ function invalid(): CursorDecodeResult { return { ok: false, reason: "invalid" }; } -interface CursorPayload { +interface CursorPayloadV1 { readonly v: 1; readonly c: QueryCommand; readonly q: string; @@ -180,3 +297,24 @@ interface CursorPayload { readonly g: number; readonly o: number; } + +type CursorPayloadV2 = + | { + readonly v: 2; + readonly c: "list"; + readonly q: string; + readonly l: string; + readonly g: number; + readonly o: number; + readonly s: number; + } + | { + readonly v: 2; + readonly c: "entries"; + readonly q: string; + readonly l: string; + readonly g: number; + readonly o: number; + readonly s: number; + readonly e: number; + }; diff --git a/src/infrastructure/sqlite/sqlite-query-context.ts b/src/infrastructure/sqlite/sqlite-query-context.ts index 467921c..40299c9 100644 --- a/src/infrastructure/sqlite/sqlite-query-context.ts +++ b/src/infrastructure/sqlite/sqlite-query-context.ts @@ -10,35 +10,73 @@ import type { Actor } from "../../domain/session.ts"; import { SqliteSessionIndexError } from "./sqlite-session-transaction.ts"; const ACTORS = new Set(["human", "model", "tool", "system", "unknown"]); +// Three parameters per coordinate keeps a chunk below SQLite's common 999-variable limit. +export const SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE = 200; +// At most twenty adjacent entries can precede the twenty-one extras needed +// to prove linked-context truncation. +const LINKED_CANDIDATE_LIMIT = MAX_SESSION_SEARCH_LINKED_CONTEXT * 2 + 1; export interface SqliteSearchContext { readonly entries: readonly SessionSearchContextEntry[]; readonly linkedContextTruncated: boolean; } -export function readSearchContext( +export interface SqliteSearchContextCoordinate { + readonly sessionId: number; + readonly entryOrdinal: number; +} + +export function readSearchContexts( database: DatabaseSync, - sessionId: number, - primaryOrdinal: number, + primaries: readonly SqliteSearchContextCoordinate[], adjacentLimit: number, -): SqliteSearchContext { - const adjacent = adjacentOrdinals(primaryOrdinal, adjacentLimit); - const linked = readLinkedOrdinals(database, sessionId, primaryOrdinal); - const linkedAdditions = linked.filter((ordinal) => !adjacent.has(ordinal)); - const selectedLinked = linkedAdditions.slice(0, MAX_SESSION_SEARCH_LINKED_CONTEXT); - const selected = new Set([...adjacent, ...selectedLinked]); - if (selected.size === 0) { - return Object.freeze({ - entries: Object.freeze([]), - linkedContextTruncated: linkedAdditions.length > MAX_SESSION_SEARCH_LINKED_CONTEXT, - }); +): readonly SqliteSearchContext[] { + if (primaries.length === 0) return Object.freeze([]); + const uniquePrimaries = new Set(primaries.map(coordinateKey)); + if (uniquePrimaries.size !== primaries.length) { + throw new SqliteSessionIndexError("corrupt-data"); } - const linkedSet = new Set(linked); - const entries = readContextEntries(database, sessionId, selected, adjacent, linkedSet); - return Object.freeze({ - entries: Object.freeze(entries), - linkedContextTruncated: linkedAdditions.length > MAX_SESSION_SEARCH_LINKED_CONTEXT, + const linkedByPrimary = readLinkedOrdinals(database, primaries); + const plans = primaries.map((primary, primaryIndex) => { + const adjacent = adjacentOrdinals(primary.entryOrdinal, adjacentLimit); + const linked = linkedByPrimary[primaryIndex]; + if (linked === undefined) throw new SqliteSessionIndexError("corrupt-data"); + const linkedAdditions = linked.filter((ordinal) => !adjacent.has(ordinal)); + return { + primary, + adjacent, + linked: new Set(linked), + selected: new Set([ + ...adjacent, + ...linkedAdditions.slice(0, MAX_SESSION_SEARCH_LINKED_CONTEXT), + ]), + linkedContextTruncated: linkedAdditions.length > MAX_SESSION_SEARCH_LINKED_CONTEXT, + }; }); + const hydrated = readContextEntries(database, selectedContextCoordinates(plans)); + return Object.freeze( + plans.map((plan) => + Object.freeze({ + entries: Object.freeze( + [...plan.selected] + .toSorted((left, right) => left - right) + .flatMap((ordinal) => { + const entry = hydrated.get(coordinateKeyOf(plan.primary.sessionId, ordinal)); + return entry === undefined + ? [] + : [ + Object.freeze({ + ...entry, + adjacent: plan.adjacent.has(ordinal), + linked: plan.linked.has(ordinal), + }), + ]; + }), + ), + linkedContextTruncated: plan.linkedContextTruncated, + }), + ), + ); } export function truncateUtf8( @@ -112,96 +150,197 @@ function adjacentOrdinals(primaryOrdinal: number, limit: number): Set { function readLinkedOrdinals( database: DatabaseSync, - sessionId: number, - primaryOrdinal: number, -): readonly number[] { + primaries: readonly SqliteSearchContextCoordinate[], +): readonly (readonly number[])[] { + const selected = selectedPrimaryCte(primaries); const rows = database .prepare( - `SELECT candidate.ordinal - FROM sessions_entries AS primary_entry - JOIN sessions_entries AS candidate - ON candidate.session_id = primary_entry.session_id - AND candidate.ordinal <> primary_entry.ordinal - WHERE primary_entry.session_id = ? - AND primary_entry.ordinal = ? - AND ( + `${selected.sql}, + ranked_links AS ( + SELECT selected.primary_index, + selected.session_id, + selected.primary_ordinal, + candidate.ordinal AS candidate_ordinal, + ROW_NUMBER() OVER ( + PARTITION BY selected.primary_index + ORDER BY candidate.ordinal + ) AS candidate_rank + FROM selected_primaries AS selected + JOIN sessions_entries AS primary_entry + ON primary_entry.session_id = selected.session_id + AND primary_entry.ordinal = selected.primary_ordinal + JOIN sessions_entries AS candidate + ON candidate.session_id = primary_entry.session_id + AND candidate.ordinal <> primary_entry.ordinal + WHERE ( primary_entry.related_entry_ordinal = candidate.ordinal OR candidate.related_entry_ordinal = primary_entry.ordinal ) - AND ( - (primary_entry.kind = 'tool-call' AND candidate.kind = 'tool-result') - OR - (primary_entry.kind = 'tool-result' AND candidate.kind = 'tool-call') - ) - ORDER BY candidate.ordinal - LIMIT ?`, + AND ( + (primary_entry.kind = 'tool-call' AND candidate.kind = 'tool-result') + OR + (primary_entry.kind = 'tool-result' AND candidate.kind = 'tool-call') + ) + ) + SELECT primary_index, session_id, primary_ordinal, candidate_ordinal + FROM ranked_links + WHERE candidate_rank <= ? + ORDER BY primary_index, candidate_ordinal`, ) - // At most twenty adjacent entries can precede the twenty-one extras needed - // to prove linked-context truncation. - .all( - sessionId, - primaryOrdinal, - MAX_SESSION_SEARCH_LINKED_CONTEXT * 2 + 1, - ) as unknown as readonly { - readonly ordinal: unknown; - }[]; - return rows.map((row) => integerAt(row.ordinal)); + .all(...selected.parameters, LINKED_CANDIDATE_LIMIT) as unknown as readonly LinkedRow[]; + const result = primaries.map(() => [] as number[]); + for (const row of rows) { + const primaryIndex = integerAt(row.primary_index); + const primary = primaries[primaryIndex]; + const linked = result[primaryIndex]; + if ( + primary === undefined || + linked === undefined || + integerAt(row.session_id) !== primary.sessionId || + integerAt(row.primary_ordinal) !== primary.entryOrdinal + ) { + throw new SqliteSessionIndexError("corrupt-data"); + } + const ordinal = integerAt(row.candidate_ordinal); + const previous = linked.at(-1); + if (previous !== undefined && ordinal <= previous) { + throw new SqliteSessionIndexError("corrupt-data"); + } + linked.push(ordinal); + } + return result; } function readContextEntries( database: DatabaseSync, - sessionId: number, - selected: ReadonlySet, - adjacent: ReadonlySet, - linked: ReadonlySet, -): readonly SessionSearchContextEntry[] { - const ordinals = [...selected].toSorted((left, right) => left - right); - const placeholders = ordinals.map(() => "?").join(", "); - const rows = database - .prepare( - `SELECT entry.ordinal, - entry.kind, - entry.actor, - entry.timestamp, - entry.related_entry_ordinal, - entry.tool_call_id, - entry.tool_name, - entry.tool_namespace, - occurrence.segment_ordinal, - content.text - FROM sessions_entries AS entry - LEFT JOIN sessions_content_occurrences AS occurrence - ON occurrence.session_id = entry.session_id - AND occurrence.entry_ordinal = entry.ordinal - LEFT JOIN sessions_content_values AS content - ON content.content_id = occurrence.content_id - WHERE entry.session_id = ? - AND entry.ordinal IN (${placeholders}) - ORDER BY entry.ordinal, occurrence.segment_ordinal`, - ) - .all(sessionId, ...ordinals) as unknown as readonly ContextRow[]; - const grouped = new Map(); - for (const row of rows) { - const ordinal = integerAt(row.ordinal); - const current = grouped.get(ordinal) ?? { entry: entryAt(row), body: [] }; - if (row.text !== null) { - if (typeof row.text !== "string" || !row.text.isWellFormed()) { + coordinates: readonly SqliteSearchContextCoordinate[], +): ReadonlyMap { + const result = new Map(); + for ( + let start = 0; + start < coordinates.length; + start += SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE + ) { + const chunk = coordinates.slice(start, start + SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE); + const selected = selectedContextCte(chunk); + const rows = database + .prepare( + `${selected.sql} + SELECT selected.coordinate_index, + selected.session_id AS selected_session_id, + selected.entry_ordinal AS selected_entry_ordinal, + entry.ordinal, + entry.kind, + entry.actor, + entry.timestamp, + entry.related_entry_ordinal, + entry.tool_call_id, + entry.tool_name, + entry.tool_namespace, + occurrence.segment_ordinal, + content.text + FROM selected_context AS selected + JOIN sessions_entries AS entry + ON entry.session_id = selected.session_id + AND entry.ordinal = selected.entry_ordinal + LEFT JOIN sessions_content_occurrences AS occurrence + ON occurrence.session_id = entry.session_id + AND occurrence.entry_ordinal = entry.ordinal + LEFT JOIN sessions_content_values AS content + ON content.content_id = occurrence.content_id + ORDER BY selected.coordinate_index, occurrence.segment_ordinal`, + ) + .all(...selected.parameters) as unknown as readonly ContextRow[]; + const grouped = new Map(); + for (const row of rows) { + const coordinateIndex = integerAt(row.coordinate_index); + const coordinate = chunk[coordinateIndex]; + if ( + coordinate === undefined || + integerAt(row.selected_session_id) !== coordinate.sessionId || + integerAt(row.selected_entry_ordinal) !== coordinate.entryOrdinal || + integerAt(row.ordinal) !== coordinate.entryOrdinal + ) { throw new SqliteSessionIndexError("corrupt-data"); } - current.body.push(row.text); + const current = grouped.get(coordinateIndex) ?? { entry: entryAt(row), body: [] }; + if (row.text !== null) { + if (typeof row.text !== "string" || !row.text.isWellFormed()) { + throw new SqliteSessionIndexError("corrupt-data"); + } + current.body.push(row.text); + } + grouped.set(coordinateIndex, current); + } + for (const [coordinateIndex, value] of grouped) { + const coordinate = chunk[coordinateIndex]; + if (coordinate === undefined) throw new SqliteSessionIndexError("corrupt-data"); + const body = truncateUtf8(value.body.join("\n")); + const key = coordinateKey(coordinate); + if (result.has(key)) throw new SqliteSessionIndexError("corrupt-data"); + result.set( + key, + Object.freeze({ + ...value.entry, + body: body.text, + bodyTruncated: body.truncated, + }), + ); } - grouped.set(ordinal, current); } - return [...grouped.entries()].map(([ordinal, value]) => { - const body = truncateUtf8(value.body.join("\n")); - return Object.freeze({ - ...value.entry, - body: body.text, - bodyTruncated: body.truncated, - adjacent: adjacent.has(ordinal), - linked: linked.has(ordinal), - }); - }); + return result; +} + +function selectedContextCoordinates( + plans: readonly { + readonly primary: SqliteSearchContextCoordinate; + readonly selected: ReadonlySet; + }[], +): readonly SqliteSearchContextCoordinate[] { + const byKey = new Map(); + for (const plan of plans) { + for (const entryOrdinal of plan.selected) { + const coordinate = { sessionId: plan.primary.sessionId, entryOrdinal }; + byKey.set(coordinateKey(coordinate), coordinate); + } + } + return [...byKey.values()].toSorted( + (left, right) => left.sessionId - right.sessionId || left.entryOrdinal - right.entryOrdinal, + ); +} + +function selectedPrimaryCte(primaries: readonly SqliteSearchContextCoordinate[]): SelectedCte { + return { + sql: `WITH selected_primaries(primary_index, session_id, primary_ordinal) AS ( + VALUES ${primaries.map(() => "(?, ?, ?)").join(", ")} + )`, + parameters: primaries.flatMap((primary, primaryIndex) => [ + primaryIndex, + primary.sessionId, + primary.entryOrdinal, + ]), + }; +} + +function selectedContextCte(coordinates: readonly SqliteSearchContextCoordinate[]): SelectedCte { + return { + sql: `WITH selected_context(coordinate_index, session_id, entry_ordinal) AS ( + VALUES ${coordinates.map(() => "(?, ?, ?)").join(", ")} + )`, + parameters: coordinates.flatMap((coordinate, coordinateIndex) => [ + coordinateIndex, + coordinate.sessionId, + coordinate.entryOrdinal, + ]), + }; +} + +function coordinateKey(coordinate: SqliteSearchContextCoordinate): string { + return coordinateKeyOf(coordinate.sessionId, coordinate.entryOrdinal); +} + +function coordinateKeyOf(sessionId: number, entryOrdinal: number): string { + return `${String(sessionId)}:${String(entryOrdinal)}`; } export function entryAt(row: EntryRow): SessionSearchEntry { @@ -276,6 +415,23 @@ interface EntryRow { } interface ContextRow extends EntryRow { + readonly coordinate_index: unknown; + readonly selected_session_id: unknown; + readonly selected_entry_ordinal: unknown; readonly segment_ordinal: unknown; readonly text: unknown; } + +interface LinkedRow { + readonly primary_index: unknown; + readonly session_id: unknown; + readonly primary_ordinal: unknown; + readonly candidate_ordinal: unknown; +} + +type ContextEntryBody = Omit; + +interface SelectedCte { + readonly sql: string; + readonly parameters: readonly number[]; +} diff --git a/src/infrastructure/sqlite/sqlite-session-entry-query.ts b/src/infrastructure/sqlite/sqlite-session-entry-query.ts index c849fbd..ffdba09 100644 --- a/src/infrastructure/sqlite/sqlite-session-entry-query.ts +++ b/src/infrastructure/sqlite/sqlite-session-entry-query.ts @@ -1,4 +1,5 @@ import type { DatabaseSync } from "node:sqlite"; +import { performance } from "node:perf_hooks"; import { SessionQueryOperationalError, @@ -19,9 +20,10 @@ import { isSessionIdentity } from "../../domain/session-identity.ts"; import type { ContentOrigin, OriginConfidence, SessionIdentity } from "../../domain/session.ts"; import { decodeQueryCursor, - encodeQueryCursor, + encodeAnchoredQueryCursor, fingerprintQuery, readQueryRevision, + type QueryCursorAnchor, type QueryRevision, } from "./query-cursor.ts"; import { decodeSqliteContentDigest } from "./sqlite-content-digest.ts"; @@ -32,7 +34,10 @@ import { type SqliteQueryWhere, } from "./sqlite-query-filters.ts"; import { createRetainedSessionRootResolver } from "./sqlite-query-lineage.ts"; -import { readSessionSummary } from "./sqlite-session-state.ts"; +import { + readSessionSummariesBatch, + type SessionSummaryBatchRequest, +} from "./sqlite-session-state.ts"; import { SqliteSessionIndexError } from "./sqlite-session-transaction.ts"; import { readSqliteCaptureScope } from "./sqlite-capture-scope.ts"; @@ -48,23 +53,62 @@ const ORIGINS = new Set([ ]); const CONFIDENCES = new Set(["high", "medium", "low", "unknown"]); +export type SqliteEntryQueryPhase = "coordinate-selection" | "hydration"; + +export interface SqliteEntryQueryWork { + readonly phase: SqliteEntryQueryPhase; + readonly elapsedMilliseconds: number; + readonly rowCount: number; +} + +export interface SqliteEntryQueryOptions { + readonly observeWork?: (work: SqliteEntryQueryWork) => void; +} + +export interface SqliteEntryCoordinateStatement { + readonly sql: string; + readonly parameters: readonly (string | number)[]; +} + +export type SqliteEntryCoordinatePosition = + | { readonly kind: "first" } + | { readonly kind: "offset"; readonly offset: number } + | { + readonly kind: "keyset"; + readonly anchor: SqliteEntryResolvedAnchor; + }; + +export interface SqliteEntryResolvedAnchor { + readonly sessionId: number; + readonly entryOrdinal: number; + readonly sourceKind: string; + readonly instanceId: string; + readonly nativeId: string; +} + export function listSqliteSessionEntries( database: DatabaseSync, query: SessionEntryQuery, + options: SqliteEntryQueryOptions = {}, ): SessionEntryPage { const cursor = prepareEntryCursor(database, query); const captureScope = readSqliteCaptureScope(database, query.filter); - const rows = readEntryRows(database, query, cursor.offset); + const rows = observeQueryWork("coordinate-selection", options.observeWork, () => + readEntryRows(database, query, cursor.position), + ); const pageRows = rows.slice(0, query.limit); - const entries = pageRows.length === 0 ? [] : hydrateEntries(database, pageRows, query); + const entries = observeQueryWork("hydration", options.observeWork, () => + pageRows.length === 0 ? [] : hydrateEntries(database, pageRows, query), + ); const nextCursor = rows.length > query.limit ? createSessionQueryCursor( - encodeQueryCursor({ + encodeAnchoredQueryCursor({ command: "entries", fingerprint: cursor.fingerprint, revision: cursor.revision, offset: cursor.offset + query.limit, + anchor: entryCursorAnchor(pageRows), }), ) : undefined; @@ -78,44 +122,111 @@ export function listSqliteSessionEntries( function readEntryRows( database: DatabaseSync, query: SessionEntryQuery, - offset: number, + position: SqliteEntryCoordinatePosition, ): readonly EntryCoordinateRow[] { + const statement = buildSqliteEntryCoordinateStatement(query, position); + return database + .prepare(statement.sql) + .all(...statement.parameters) as unknown as readonly EntryCoordinateRow[]; +} + +export function buildSqliteEntryCoordinateStatement( + query: SessionEntryQuery, + position: SqliteEntryCoordinatePosition, +): SqliteEntryCoordinateStatement { const outer = entryInventoryWhere(query.filter); const selection = selectionClause(query); - return database - .prepare( - `SELECT canonical.session_id, - source.kind AS source_kind, - source.instance_id, - tracking.native_id, - entry.ordinal, - entry.kind, - entry.actor, - entry.timestamp, - entry.related_entry_ordinal, - entry.tool_call_id, - entry.tool_name, - entry.tool_namespace - FROM sessions_source_instances AS source - JOIN sessions_session_tracking AS tracking - ON tracking.source_instance_id = source.source_instance_id - JOIN sessions_canonical_sessions AS canonical - ON canonical.session_id = tracking.session_id - JOIN sessions_entries AS entry - ON entry.session_id = canonical.session_id - WHERE 1 = 1${outer.sql}${selection.sql} - ORDER BY source.kind COLLATE BINARY, - source.instance_id COLLATE BINARY, - tracking.native_id COLLATE BINARY, - entry.ordinal - LIMIT ? OFFSET ?`, - ) - .all( + const continuation = continuationClause(position); + return Object.freeze({ + sql: `SELECT canonical.session_id, + source.kind AS source_kind, + source.instance_id, + tracking.native_id, + entry.ordinal, + entry.kind, + entry.actor, + entry.timestamp, + entry.related_entry_ordinal, + entry.tool_call_id, + entry.tool_name, + entry.tool_namespace + FROM sessions_source_instances AS source + CROSS JOIN sessions_session_tracking AS tracking + CROSS JOIN sessions_canonical_sessions AS canonical + CROSS JOIN sessions_entries AS entry + WHERE tracking.source_instance_id = source.source_instance_id + AND canonical.session_id = tracking.session_id + AND entry.session_id = canonical.session_id${outer.sql}${selection.sql}${continuation.sql} + ORDER BY source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY, + entry.ordinal + LIMIT ?${position.kind === "offset" ? " OFFSET ?" : ""}`, + parameters: Object.freeze([ ...outer.parameters, ...selection.parameters, + ...continuation.parameters, query.limit + 1, - offset, - ) as unknown as readonly EntryCoordinateRow[]; + ...(position.kind === "offset" ? [position.offset] : []), + ]), + }); +} + +function continuationClause(position: SqliteEntryCoordinatePosition): SqliteEntryContinuation { + if (position.kind !== "keyset") return { sql: "", parameters: [] }; + return { + sql: ` AND ( + source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY + ) >= (?, ?, ?) + AND entry.ordinal > CASE + WHEN canonical.session_id = ? THEN ? + ELSE -1 + END`, + parameters: [ + position.anchor.sourceKind, + position.anchor.instanceId, + position.anchor.nativeId, + position.anchor.sessionId, + position.anchor.entryOrdinal, + ], + }; +} + +function entryCursorAnchor( + pageRows: readonly EntryCoordinateRow[], +): Extract { + const row = pageRows.at(-1); + if (row === undefined) throw new SqliteSessionIndexError("corrupt-data"); + return { + kind: "entries", + sessionId: integerAt(row.session_id), + entryOrdinal: integerAt(row.ordinal), + }; +} + +function observeQueryWork( + phase: SqliteEntryQueryPhase, + observer: SqliteEntryQueryOptions["observeWork"], + run: () => T, +): T { + const startedAt = performance.now(); + const result = run(); + if (observer !== undefined) { + try { + observer( + Object.freeze({ + phase, + elapsedMilliseconds: performance.now() - startedAt, + rowCount: result.length, + }), + ); + } catch { + // Diagnostics are private and best-effort; they never affect query results. + } + } + return result; } function selectionClause(query: SessionEntryQuery): SqliteQueryWhere { @@ -142,21 +253,21 @@ function hydrateEntries( sessionId: integerAt(row.session_id), entryOrdinal: integerAt(row.ordinal), })); + const identities = rows.map(identityAt); const counts = readContentCounts(database, coordinates); const previews = readPreviews(database, coordinates, query.filter.origin); const resolveRoot = createRetainedSessionRootResolver(database); - const summaryCache = new Map(); + const summaries = readEntrySummaries(database, coordinates, identities); return rows.map((row, index) => { const coordinate = coordinates[index]; - if (coordinate === undefined) throw new SqliteSessionIndexError("corrupt-data"); - const identity = identityAt(row); - let summary = summaryCache.get(coordinate.sessionId); - if (summary === undefined) { - const stored = readSessionSummary(database, identity); - if (stored === undefined) throw new SqliteSessionIndexError("corrupt-data"); - summary = freezeSummary(stored); - summaryCache.set(coordinate.sessionId, summary); + const identity = identities[index]; + if (coordinate === undefined || identity === undefined) { + throw new SqliteSessionIndexError("corrupt-data"); + } + const summary = summaries.get(coordinate.sessionId); + if (summary === undefined || !sameIdentity(summary.identity, identity)) { + throw new SqliteSessionIndexError("corrupt-data"); } const root = resolveRoot(identity); const coordinateKey = contentKey(coordinate.sessionId, coordinate.entryOrdinal); @@ -173,6 +284,32 @@ function hydrateEntries( }); } +function readEntrySummaries( + database: DatabaseSync, + coordinates: readonly EntryCoordinate[], + identities: readonly SessionIdentity[], +): ReadonlyMap { + const requests = new Map(); + for (const [index, coordinate] of coordinates.entries()) { + const identity = identities[index]; + if (identity === undefined) throw new SqliteSessionIndexError("corrupt-data"); + const previous = requests.get(coordinate.sessionId); + if (previous !== undefined) { + if (!sameIdentity(previous.identity, identity)) { + throw new SqliteSessionIndexError("corrupt-data"); + } + continue; + } + requests.set(coordinate.sessionId, { sessionId: coordinate.sessionId, identity }); + } + const stored = readSessionSummariesBatch(database, [...requests.values()]); + const summaries = new Map(); + for (const [sessionId, summary] of stored) { + summaries.set(sessionId, freezeSummary(summary)); + } + return summaries; +} + function readContentCounts( database: DatabaseSync, coordinates: readonly EntryCoordinate[], @@ -305,7 +442,9 @@ function contentSummary( function prepareEntryCursor(database: DatabaseSync, query: SessionEntryQuery): PreparedCursor { const revision = readQueryRevision(database); const fingerprint = fingerprintQuery(sessionQueryFingerprintMaterial(query)); - if (query.cursor === undefined) return { revision, fingerprint, offset: 0 }; + if (query.cursor === undefined) { + return { revision, fingerprint, offset: 0, position: { kind: "first" } }; + } const decoded = decodeQueryCursor(query.cursor, { command: "entries", fingerprint, @@ -317,7 +456,104 @@ function prepareEntryCursor(database: DatabaseSync, query: SessionEntryQuery): P decoded.reason === "mismatch" ? "cursor-query-mismatch" : "invalid-cursor", ); } - return { revision, fingerprint, offset: decoded.offset }; + if (decoded.anchor === undefined) { + return { + revision, + fingerprint, + offset: decoded.offset, + position: { kind: "offset", offset: decoded.offset }, + }; + } + if (decoded.anchor.kind !== "entries") { + throw new SessionQueryUsageError("invalid-cursor"); + } + return { + revision, + fingerprint, + offset: decoded.offset, + position: { + kind: "keyset", + anchor: resolveEntryAnchor(database, query, decoded.anchor), + }, + }; +} + +function resolveEntryAnchor( + database: DatabaseSync, + query: SessionEntryQuery, + anchor: Extract, +): SqliteEntryResolvedAnchor { + const outer = entryInventoryWhere(query.filter); + const selection = selectionClause(query); + const rows = database + .prepare( + `WITH physical_anchor AS MATERIALIZED ( + SELECT canonical.session_id, + source.kind AS source_kind, + source.instance_id, + tracking.native_id, + entry.ordinal, + entry.kind, + entry.actor, + entry.timestamp, + entry.related_entry_ordinal, + entry.tool_call_id, + entry.tool_name, + entry.tool_namespace + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + JOIN sessions_entries AS entry + ON entry.session_id = canonical.session_id + WHERE canonical.session_id = ? + AND entry.ordinal = ? + ), + qualifying_anchor AS MATERIALIZED ( + SELECT canonical.session_id, entry.ordinal + FROM sessions_source_instances AS source + CROSS JOIN sessions_session_tracking AS tracking + CROSS JOIN sessions_canonical_sessions AS canonical + CROSS JOIN sessions_entries AS entry + WHERE tracking.source_instance_id = source.source_instance_id + AND canonical.session_id = tracking.session_id + AND entry.session_id = canonical.session_id + AND canonical.session_id = ? + AND entry.ordinal = ?${outer.sql}${selection.sql} + ) + SELECT physical_anchor.*, + EXISTS (SELECT 1 FROM qualifying_anchor) AS qualifies + FROM physical_anchor`, + ) + .all( + anchor.sessionId, + anchor.entryOrdinal, + anchor.sessionId, + anchor.entryOrdinal, + ...outer.parameters, + ...selection.parameters, + ) as unknown as readonly EntryAnchorRow[]; + if (rows.length === 0) throw new SessionQueryUsageError("invalid-cursor"); + if (rows.length !== 1) throw new SqliteSessionIndexError("corrupt-data"); + const row = rows[0]; + if (row === undefined) throw new SqliteSessionIndexError("corrupt-data"); + const sessionId = integerAt(row.session_id); + const entryOrdinal = integerAt(row.ordinal); + const identity = identityAt(row); + entryAt(row); + const qualifies = integerAt(row.qualifies); + if (sessionId !== anchor.sessionId || entryOrdinal !== anchor.entryOrdinal || qualifies > 1) { + throw new SqliteSessionIndexError("corrupt-data"); + } + if (qualifies === 0) throw new SessionQueryUsageError("invalid-cursor"); + return { + sessionId, + entryOrdinal, + sourceKind: identity.source.kind, + instanceId: identity.source.instanceId, + nativeId: identity.nativeId, + }; } function identityAt(row: EntryCoordinateRow): SessionIdentity { @@ -339,6 +575,14 @@ function freezeSummary(summary: SessionQuerySummary): SessionQuerySummary { }); } +function sameIdentity(left: SessionIdentity, right: SessionIdentity): boolean { + return ( + left.source.kind === right.source.kind && + left.source.instanceId === right.source.instanceId && + left.nativeId === right.nativeId + ); +} + function storedString(value: unknown): string { if (typeof value !== "string" || !value.isWellFormed()) { throw new SqliteSessionIndexError("corrupt-data"); @@ -362,6 +606,7 @@ interface PreparedCursor { readonly revision: QueryRevision; readonly fingerprint: string; readonly offset: number; + readonly position: SqliteEntryCoordinatePosition; } interface EntryCoordinate { @@ -374,6 +619,11 @@ interface SelectedEntriesCte { readonly parameters: readonly number[]; } +interface SqliteEntryContinuation { + readonly sql: string; + readonly parameters: readonly (string | number)[]; +} + interface ContentCount { readonly textSegmentCount: number; readonly omittedSegmentCount: number; @@ -394,6 +644,10 @@ interface EntryCoordinateRow { readonly tool_namespace: unknown; } +interface EntryAnchorRow extends EntryCoordinateRow { + readonly qualifies: unknown; +} + interface ContentCountRow { readonly session_id: unknown; readonly entry_ordinal: unknown; diff --git a/src/infrastructure/sqlite/sqlite-session-query.ts b/src/infrastructure/sqlite/sqlite-session-query.ts index bcd45bb..8cb2beb 100644 --- a/src/infrastructure/sqlite/sqlite-session-query.ts +++ b/src/infrastructure/sqlite/sqlite-session-query.ts @@ -16,6 +16,7 @@ import { type SessionSearchQuery, type SessionSearchSnippet, } from "../../domain/session-query.ts"; +import { isCanonicalTimestamp } from "../../domain/canonical-timestamp.ts"; import { contentHashMatches } from "../../domain/content-hash.ts"; import type { SessionRootResolver } from "../../domain/session-lineage.ts"; import { isSessionIdentity } from "../../domain/session-identity.ts"; @@ -25,13 +26,15 @@ import { readSqliteCaptureScope } from "./sqlite-capture-scope.ts"; import { literalFtsQuery } from "./literal-fts-query.ts"; import { decodeQueryCursor, + encodeAnchoredQueryCursor, encodeQueryCursor, fingerprintQuery, readQueryRevision, + type QueryCursorAnchor, type QueryCommand, type QueryRevision, } from "./query-cursor.ts"; -import { readSearchContext, entryAt, truncateUtf8Around } from "./sqlite-query-context.ts"; +import { entryAt, readSearchContexts, truncateUtf8Around } from "./sqlite-query-context.ts"; import { EFFECTIVE_SESSION_ACTIVITY_SQL, searchWhere, @@ -40,7 +43,10 @@ import { } from "./sqlite-query-filters.ts"; import { countRootSupport, createRetainedSessionRootResolver } from "./sqlite-query-lineage.ts"; import { decodeSqliteContentDigest } from "./sqlite-content-digest.ts"; -import { readSessionSummary } from "./sqlite-session-state.ts"; +import { + readSessionSummariesBatch, + type SessionSummaryBatchRequest, +} from "./sqlite-session-state.ts"; import { SqliteSessionIndexError } from "./sqlite-session-transaction.ts"; import { listSqliteSessionEntries } from "./sqlite-session-entry-query.ts"; import { readSqliteSessionManifest } from "./sqlite-session-manifest.ts"; @@ -77,39 +83,28 @@ export function createSqliteSessionQuery(database: DatabaseSync): SessionQueryRe function listSessions(database: DatabaseSync, query: SessionListQuery): SessionListPage { const cursor = prepareCursor(database, "list", query); const captureScope = readSqliteCaptureScope(database, query.filter); - const where = sessionWhere(query.filter); + const position = listCoordinatePosition(database, query, cursor); + const statement = buildSqliteListCoordinateStatement(query, position); const rows = database - .prepare( - `SELECT source.kind, source.instance_id, tracking.native_id - FROM sessions_canonical_sessions AS canonical - JOIN sessions_session_tracking AS tracking - ON tracking.session_id = canonical.session_id - JOIN sessions_source_instances AS source - ON source.source_instance_id = tracking.source_instance_id - WHERE 1 = 1${where.sql} - ORDER BY - CASE WHEN ${EFFECTIVE_SESSION_ACTIVITY_SQL} IS NULL THEN 1 ELSE 0 END, - ${EFFECTIVE_SESSION_ACTIVITY_SQL} DESC, - source.kind COLLATE BINARY, - source.instance_id COLLATE BINARY, - tracking.native_id COLLATE BINARY - LIMIT ? OFFSET ?`, - ) - .all(...where.parameters, query.limit + 1, cursor.offset) as unknown as readonly IdentityRow[]; + .prepare(statement.sql) + .all(...statement.parameters) as unknown as readonly ListCoordinateRow[]; const pageRows = rows.slice(0, query.limit); const resolveRoot = pageRows.length === 0 ? undefined : createRetainedSessionRootResolver(database); + const summaries = readSelectedSummaries(database, pageRows.map(listSessionCoordinateAt)); const sessions = pageRows.map((row) => { const identity = identityAt(row); - const summary = readSessionSummary(database, identity); + const summary = summaries.get(integerAt(row.session_id)); if (summary === undefined || resolveRoot === undefined) { throw new SqliteSessionIndexError("corrupt-data"); } + if (!sameIdentity(summary.identity, identity)) + throw new SqliteSessionIndexError("corrupt-data"); return Object.freeze({ ...freezeSummary(summary), root: resolveRoot(identity) }); }); const nextCursor = rows.length > query.limit - ? nextQueryCursor("list", cursor, cursor.offset + query.limit) + ? nextListQueryCursor(cursor, cursor.offset + query.limit, pageRows) : undefined; return Object.freeze({ sessions: Object.freeze(sessions), @@ -118,6 +113,220 @@ function listSessions(database: DatabaseSync, query: SessionListQuery): SessionL }); } +export interface SqliteListCoordinateStatement { + readonly sql: string; + readonly parameters: readonly (number | string)[]; +} + +export type SqliteListCoordinatePosition = + | { readonly kind: "first" } + | { readonly kind: "offset"; readonly offset: number } + | { readonly kind: "keyset"; readonly anchor: SqliteListResolvedAnchor }; + +export interface SqliteListResolvedAnchor { + readonly sessionId: number; + readonly activity: string | null; + readonly sourceKind: string; + readonly instanceId: string; + readonly nativeId: string; +} + +export function buildSqliteListCoordinateStatement( + query: SessionListQuery, + position: SqliteListCoordinatePosition, +): SqliteListCoordinateStatement { + const where = sessionWhere(query.filter); + if (position.kind !== "keyset") { + return Object.freeze({ + sql: `${listCoordinateSelect()} + WHERE 1 = 1${where.sql} + ${listCoordinateOrder()} + LIMIT ?${position.kind === "offset" ? " OFFSET ?" : ""}`, + parameters: Object.freeze([ + ...where.parameters, + query.limit + 1, + ...(position.kind === "offset" ? [position.offset] : []), + ]), + }); + } + + return Object.freeze({ + sql: `WITH resolved_anchor AS MATERIALIZED ( + SELECT canonical.session_id, + ${EFFECTIVE_SESSION_ACTIVITY_SQL} AS activity, + source.kind AS source_kind, + source.instance_id, + tracking.native_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE canonical.session_id = ? + AND 1 = 1${where.sql} + ) + ${listCoordinateSelect()} + CROSS JOIN resolved_anchor AS anchor + WHERE 1 = 1${where.sql} + AND ( + CASE WHEN ${EFFECTIVE_SESSION_ACTIVITY_SQL} IS NULL THEN 1 ELSE 0 END + > CASE WHEN anchor.activity IS NULL THEN 1 ELSE 0 END + OR ( + CASE WHEN ${EFFECTIVE_SESSION_ACTIVITY_SQL} IS NULL THEN 1 ELSE 0 END + = CASE WHEN anchor.activity IS NULL THEN 1 ELSE 0 END + AND ( + ( + anchor.activity IS NOT NULL + AND ${EFFECTIVE_SESSION_ACTIVITY_SQL} < anchor.activity + ) + OR ( + ( + ${EFFECTIVE_SESSION_ACTIVITY_SQL} = anchor.activity + OR ( + ${EFFECTIVE_SESSION_ACTIVITY_SQL} IS NULL + AND anchor.activity IS NULL + ) + ) + AND ( + source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY + ) > ( + anchor.source_kind, + anchor.instance_id, + anchor.native_id + ) + ) + ) + ) + ) + ${listCoordinateOrder()} + LIMIT ?`, + parameters: Object.freeze([ + position.anchor.sessionId, + ...where.parameters, + ...where.parameters, + query.limit + 1, + ]), + }); +} + +function listCoordinateSelect(): string { + return `SELECT canonical.session_id, + source.kind, + source.instance_id, + tracking.native_id, + ${EFFECTIVE_SESSION_ACTIVITY_SQL} AS activity + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id`; +} + +function listCoordinateOrder(): string { + return `ORDER BY + CASE WHEN ${EFFECTIVE_SESSION_ACTIVITY_SQL} IS NULL THEN 1 ELSE 0 END, + ${EFFECTIVE_SESSION_ACTIVITY_SQL} DESC, + source.kind COLLATE BINARY, + source.instance_id COLLATE BINARY, + tracking.native_id COLLATE BINARY`; +} + +function listCoordinatePosition( + database: DatabaseSync, + query: SessionListQuery, + cursor: PreparedCursor, +): SqliteListCoordinatePosition { + if (cursor.anchor === undefined) { + return cursor.legacyOffset ? { kind: "offset", offset: cursor.offset } : { kind: "first" }; + } + if (cursor.anchor.kind !== "list") throw new SessionQueryUsageError("invalid-cursor"); + return { + kind: "keyset", + anchor: resolveListAnchor(database, query, cursor.anchor), + }; +} + +function resolveListAnchor( + database: DatabaseSync, + query: SessionListQuery, + anchor: Extract, +): SqliteListResolvedAnchor { + const where = sessionWhere(query.filter); + const rows = database + .prepare( + `WITH physical_anchor AS MATERIALIZED ( + SELECT canonical.session_id, + ${EFFECTIVE_SESSION_ACTIVITY_SQL} AS activity, + source.kind, + source.instance_id, + tracking.native_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE canonical.session_id = ? + ), + qualifying_anchor AS MATERIALIZED ( + SELECT canonical.session_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE canonical.session_id = ? + AND 1 = 1${where.sql} + ) + SELECT physical_anchor.*, + EXISTS (SELECT 1 FROM qualifying_anchor) AS qualifies + FROM physical_anchor`, + ) + .all( + anchor.sessionId, + anchor.sessionId, + ...where.parameters, + ) as unknown as readonly ListAnchorRow[]; + if (rows.length === 0) throw new SessionQueryUsageError("invalid-cursor"); + if (rows.length !== 1) throw new SqliteSessionIndexError("corrupt-data"); + const row = rows[0]; + if (row === undefined) throw new SqliteSessionIndexError("corrupt-data"); + const sessionId = integerAt(row.session_id); + const identity = identityAt(row); + const activity = optionalCanonicalTimestampAt(row.activity); + const qualifies = integerAt(row.qualifies); + if (sessionId !== anchor.sessionId || qualifies > 1) { + throw new SqliteSessionIndexError("corrupt-data"); + } + if (qualifies === 0) throw new SessionQueryUsageError("invalid-cursor"); + return { + sessionId, + activity, + sourceKind: identity.source.kind, + instanceId: identity.source.instanceId, + nativeId: identity.nativeId, + }; +} + +function nextListQueryCursor( + cursor: PreparedCursor, + offset: number, + pageRows: readonly ListCoordinateRow[], +) { + const row = pageRows.at(-1); + if (row === undefined) throw new SqliteSessionIndexError("corrupt-data"); + return createSessionQueryCursor( + encodeAnchoredQueryCursor({ + command: "list", + fingerprint: cursor.fingerprint, + revision: cursor.revision, + offset, + anchor: { kind: "list", sessionId: integerAt(row.session_id) }, + }), + ); +} + function searchSessions(database: DatabaseSync, query: SessionSearchQuery): SessionSearchPage { const cursor = prepareCursor(database, "search", query); const captureScope = readSqliteCaptureScope(database, { @@ -144,7 +353,7 @@ function searchSessions(database: DatabaseSync, query: SessionSearchQuery): Sess ); const nextCursor = searchRows.length > query.limit - ? nextQueryCursor("search", cursor, cursor.offset + query.limit) + ? nextSearchQueryCursor(cursor, cursor.offset + query.limit) : undefined; return Object.freeze({ hits: Object.freeze(hits), @@ -237,33 +446,51 @@ function hydrateSearchContent( libraryInstanceId: string, ): HydratedSearchContent { const contentIds = [...new Set(rows.map((row) => integerAt(row.content_id)))]; + const selectedSql = contentIds.map(() => "(?, ?)").join(", "); + const selectedParameters = contentIds.flatMap((contentId, contentIndex) => [ + contentIndex, + contentId, + ]); // FTS5 can ignore an untyped bound rowid beside MATCH; the explicit cast keeps - // hydration restricted to the one ranked canonical content row. + // hydration restricted to the selected ranked canonical content rows. const statement = database.prepare( - `SELECT content.text, + `WITH selected_content(content_index, content_id) AS ( + VALUES ${selectedSql} + ) + SELECT selected.content_index, + selected.content_id, + content.text, content.digest AS content_digest, snippet(sessions_content_fts, 0, ?, ?, ' … ', 64) AS snippet_text FROM sessions_content_fts JOIN sessions_content_values AS content ON content.content_id = sessions_content_fts.rowid + JOIN selected_content AS selected + ON sessions_content_fts.rowid = CAST(selected.content_id AS INTEGER) WHERE sessions_content_fts MATCH ? - AND sessions_content_fts.rowid = CAST(? AS INTEGER)`, + ORDER BY selected.content_index`, ); for (let candidate = 0; ; candidate += 1) { if (!Number.isSafeInteger(candidate)) throw new SqliteSessionIndexError("corrupt-data"); const markers = snippetMarkers(libraryInstanceId, candidate); + const hydrated = statement.all( + ...selectedParameters, + markers.start, + markers.end, + ftsQuery, + ) as unknown as readonly HydratedContentRow[]; + if (hydrated.length !== contentIds.length) { + throw new SqliteSessionIndexError("corrupt-data"); + } const byContentId = new Map(); let collision = false; - for (const contentId of contentIds) { - const hydrated = statement.all( - markers.start, - markers.end, - ftsQuery, - contentId, - ) as unknown as readonly HydratedContentRow[]; - if (hydrated.length !== 1) throw new SqliteSessionIndexError("corrupt-data"); - const row = hydrated[0]!; + for (const row of hydrated) { + const contentIndex = integerAt(row.content_index); + const contentId = integerAt(row.content_id); + if (contentIds[contentIndex] !== contentId || byContentId.has(contentId)) { + throw new SqliteSessionIndexError("corrupt-data"); + } const text = storedString(row.text); if (text.includes(markers.start) || text.includes(markers.end)) { collision = true; @@ -287,10 +514,14 @@ function hydrateSearchHits( if (resolveRoot === undefined) throw new SqliteSessionIndexError("corrupt-data"); const hydrated = hydrateSearchContent(database, rows, ftsQuery, libraryInstanceId); const matchedTerms = readMatchedTerms(database, rows, query); - const summaryCache = new Map(); - return rows.map((row) => - searchHit(database, row, adjacentContext, summaryCache, hydrated, matchedTerms, resolveRoot), - ); + const contexts = readSearchContexts(database, rows.map(searchCoordinateAt), adjacentContext); + if (contexts.length !== rows.length) throw new SqliteSessionIndexError("corrupt-data"); + const summaries = readSelectedSummaries(database, rows.map(searchSessionCoordinateAt)); + return rows.map((row, rowIndex) => { + const context = contexts[rowIndex]; + if (context === undefined) throw new SqliteSessionIndexError("corrupt-data"); + return searchHit(row, summaries, hydrated, matchedTerms, resolveRoot, context); + }); } function readMatchedTerms( @@ -451,13 +682,12 @@ function searchJoins(): string { } function searchHit( - database: DatabaseSync, row: SearchRankRow, - adjacentContext: number, - summaryCache: Map, + summaries: ReadonlyMap, hydrated: HydratedSearchContent, matchedTerms: ReadonlyMap, resolveRoot: SessionRootResolver, + context: ReturnType[number], ): SessionSearchHit { const sessionId = integerAt(row.session_id); const identity = identityAt({ @@ -465,12 +695,9 @@ function searchHit( instance_id: row.instance_id, native_id: row.native_id, }); - let summary = summaryCache.get(sessionId); - if (summary === undefined) { - const stored = readSessionSummary(database, identity); - if (stored === undefined) throw new SqliteSessionIndexError("corrupt-data"); - summary = freezeSummary(stored); - summaryCache.set(sessionId, summary); + const summary = summaries.get(sessionId); + if (summary === undefined || !sameIdentity(summary.identity, identity)) { + throw new SqliteSessionIndexError("corrupt-data"); } const entry = entryAt({ ordinal: row.entry_ordinal, @@ -486,7 +713,6 @@ function searchHit( const content = hydrated.byContentId.get(contentId); if (content === undefined) throw new SqliteSessionIndexError("corrupt-data"); const snippet = snippetAt(row, content, hydrated.markers); - const context = readSearchContext(database, sessionId, entry.ordinal, adjacentContext); const terms = matchedTerms.get(coordinateKey(sessionId, entry.ordinal)); if (terms === undefined) throw new SqliteSessionIndexError("corrupt-data"); return Object.freeze({ @@ -510,6 +736,45 @@ function searchCoordinateAt(row: { }; } +function listSessionCoordinateAt(row: ListCoordinateRow): SelectedSessionCoordinate { + return { sessionId: integerAt(row.session_id), identity: identityAt(row) }; +} + +function searchSessionCoordinateAt(row: SearchRankRow): SelectedSessionCoordinate { + return { + sessionId: integerAt(row.session_id), + identity: identityAt({ + kind: row.source_kind, + instance_id: row.instance_id, + native_id: row.native_id, + }), + }; +} + +function readSelectedSummaries( + database: DatabaseSync, + coordinates: readonly SelectedSessionCoordinate[], +): ReadonlyMap { + const requests = new Map(); + for (const coordinate of coordinates) { + const previous = requests.get(coordinate.sessionId); + if (previous !== undefined) { + if (!sameIdentity(previous.identity, coordinate.identity)) { + throw new SqliteSessionIndexError("corrupt-data"); + } + continue; + } + requests.set(coordinate.sessionId, { + sessionId: coordinate.sessionId, + identity: coordinate.identity, + }); + } + const stored = readSessionSummariesBatch(database, [...requests.values()]); + return new Map( + [...stored].map(([sessionId, summary]) => [sessionId, freezeSummary(summary)] as const), + ); +} + function coordinateKey(sessionId: number, entryOrdinal: number): string { return `${String(sessionId)}:${String(entryOrdinal)}`; } @@ -562,7 +827,9 @@ function prepareCursor( ): PreparedCursor { const revision = readQueryRevision(database); const fingerprint = fingerprintQuery(sessionQueryFingerprintMaterial(query)); - if (query.cursor === undefined) return { revision, fingerprint, offset: 0 }; + if (query.cursor === undefined) { + return { revision, fingerprint, offset: 0, legacyOffset: false }; + } const decoded = decodeQueryCursor(query.cursor, { command, fingerprint, revision }); if (!decoded.ok) { if (decoded.reason === "stale") throw new SessionQueryOperationalError("stale-cursor"); @@ -570,13 +837,19 @@ function prepareCursor( decoded.reason === "mismatch" ? "cursor-query-mismatch" : "invalid-cursor", ); } - return { revision, fingerprint, offset: decoded.offset }; + return { + revision, + fingerprint, + offset: decoded.offset, + legacyOffset: decoded.anchor === undefined, + ...(decoded.anchor === undefined ? {} : { anchor: decoded.anchor }), + }; } -function nextQueryCursor(command: QueryCommand, cursor: PreparedCursor, offset: number) { +function nextSearchQueryCursor(cursor: PreparedCursor, offset: number) { return createSessionQueryCursor( encodeQueryCursor({ - command, + command: "search", fingerprint: cursor.fingerprint, revision: cursor.revision, offset, @@ -603,6 +876,14 @@ function identityAt(row: IdentityRow): SessionIdentity { return identity; } +function sameIdentity(left: SessionIdentity, right: SessionIdentity): boolean { + return ( + left.source.kind === right.source.kind && + left.source.instanceId === right.source.instanceId && + left.nativeId === right.nativeId + ); +} + function emptySearchPage(captureScope: SessionSearchPage["captureScope"]): SessionSearchPage { return Object.freeze({ hits: Object.freeze([]), @@ -623,6 +904,14 @@ function storedString(value: unknown): string { return value; } +function optionalCanonicalTimestampAt(value: unknown): string | null { + if (value === null) return null; + if (typeof value !== "string" || !isCanonicalTimestamp(value)) { + throw new SqliteSessionIndexError("corrupt-data"); + } + return value; +} + function integerAt(value: unknown): number { const number = typeof value === "bigint" ? Number(value) : value; if (typeof number !== "number" || !Number.isSafeInteger(number) || number < 0) { @@ -635,6 +924,8 @@ interface PreparedCursor { readonly revision: QueryRevision; readonly fingerprint: string; readonly offset: number; + readonly legacyOffset: boolean; + readonly anchor?: QueryCursorAnchor; } interface SnippetMarkers { @@ -657,6 +948,11 @@ interface SearchCoordinate { readonly entryOrdinal: number; } +interface SelectedSessionCoordinate { + readonly sessionId: number; + readonly identity: SessionIdentity; +} + interface SearchCoordinateRow { readonly session_id: unknown; readonly entry_ordinal: unknown; @@ -668,6 +964,15 @@ interface IdentityRow { readonly native_id: unknown; } +interface ListCoordinateRow extends IdentityRow { + readonly session_id: unknown; + readonly activity: unknown; +} + +interface ListAnchorRow extends ListCoordinateRow { + readonly qualifies: unknown; +} + interface AggregateRow { readonly occurrences: unknown; readonly unique_content: unknown; @@ -694,6 +999,8 @@ interface SearchRankRow { } interface HydratedContentRow { + readonly content_index: unknown; + readonly content_id: unknown; readonly text: unknown; readonly content_digest: unknown; readonly snippet_text: unknown; diff --git a/src/infrastructure/sqlite/sqlite-session-state.ts b/src/infrastructure/sqlite/sqlite-session-state.ts index fe28fcd..dbf5737 100644 --- a/src/infrastructure/sqlite/sqlite-session-state.ts +++ b/src/infrastructure/sqlite/sqlite-session-state.ts @@ -6,6 +6,7 @@ import type { SessionIndexFailureCode, } from "../../application/ports/session-index.ts"; import type { SessionRevision } from "../../application/validate-session.ts"; +import { isSessionIdentity } from "../../domain/session-identity.ts"; import type { SessionIdentity } from "../../domain/session.ts"; import { EFFECTIVE_SOURCE_OBSERVED_AT_SQL, @@ -22,6 +23,7 @@ const FAILURE_CODES: ReadonlySet = new Set([ "unsupported-format", "repository-write", ]); +const SESSION_SUMMARY_BATCH_LIMIT = 200; export interface SessionTrackingRecord { readonly session_id: number | bigint; @@ -241,6 +243,106 @@ export function readSessionSummary( return decodeRetainedSessionSummary(identity, row, freshness); } +export interface SessionSummaryBatchRequest { + readonly sessionId: number; + readonly identity: SessionIdentity; +} + +export function readSessionSummariesBatch( + database: DatabaseSync, + requests: readonly SessionSummaryBatchRequest[], +): ReadonlyMap { + if (requests.length === 0) return new Map(); + if (requests.length > SESSION_SUMMARY_BATCH_LIMIT) { + throw new TypeError("SQLite session summary batch exceeds the supported limit"); + } + + const requestedSessionIds = new Set(); + const requestsBySessionId = new Map(); + const parameters: Array = []; + for (const request of requests) { + if ( + !Number.isSafeInteger(request.sessionId) || + request.sessionId < 0 || + !isSessionIdentity(request.identity) || + requestedSessionIds.has(request.sessionId) + ) { + throw new TypeError("Invalid SQLite session summary batch request"); + } + requestedSessionIds.add(request.sessionId); + requestsBySessionId.set(request.sessionId, request); + parameters.push( + request.sessionId, + request.identity.source.kind, + request.identity.source.instanceId, + request.identity.nativeId, + ); + } + + const values = requests.map(() => "(?, ?, ?, ?)").join(", "); + const statement = database.prepare( + `WITH requested(session_id, source_kind, instance_id, native_id) AS ( + VALUES ${values} + ) + SELECT requested.session_id AS requested_session_id, + source.kind AS source_kind, + source.instance_id, + tracking.native_id, + canonical.title, + canonical.workspace, + canonical.created_at, + canonical.updated_at, + tracking.captured_at, + canonical.document_digest_scheme, + canonical.document_digest, + ${EFFECTIVE_SOURCE_STATE_SQL} AS source_state, + ${EFFECTIVE_SOURCE_OBSERVED_AT_SQL} AS source_observed_at, + tracking.last_good_fingerprint_scheme, + tracking.last_good_fingerprint_digest, + tracking.last_good_adapter_version, + tracking.latest_fingerprint_scheme, + tracking.latest_fingerprint_digest, + tracking.latest_adapter_version, + tracking.latest_outcome, + tracking.latest_failure_code + FROM requested + JOIN sessions_canonical_sessions AS canonical + ON canonical.session_id = requested.session_id + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + AND tracking.native_id = requested.native_id COLLATE BINARY + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + AND source.kind = requested.source_kind COLLATE BINARY + AND source.instance_id = requested.instance_id COLLATE BINARY`, + ); + statement.setReadBigInts(true); + const rows = statement.all(...parameters) as unknown as readonly SessionSummaryBatchRow[]; + if (rows.length !== requests.length) throw new SqliteSessionIndexError("corrupt-data"); + + const summaries = new Map(); + for (const row of rows) { + const sessionId = integerAt(row.requested_session_id); + const request = requestsBySessionId.get(sessionId); + if ( + request === undefined || + row.source_kind !== request.identity.source.kind || + row.instance_id !== request.identity.source.instanceId || + row.native_id !== request.identity.nativeId || + summaries.has(sessionId) + ) { + throw new SqliteSessionIndexError("corrupt-data"); + } + const freshness = decodeTrackedSessionFreshness(request.identity, true, row); + if (freshness.status !== "current" && freshness.status !== "stale") { + throw new SqliteSessionIndexError("corrupt-data"); + } + summaries.set(sessionId, decodeRetainedSessionSummary(request.identity, row, freshness)); + } + if (summaries.size !== requests.length) throw new SqliteSessionIndexError("corrupt-data"); + return summaries; +} + export function decodeRetainedSessionSummary( identity: SessionIdentity, row: SessionSummaryColumns, @@ -251,10 +353,10 @@ export function decodeRetainedSessionSummary( const sourceObservedAt = canonicalTimestampAt(row.source_observed_at); return { identity: copyIdentity(identity), - ...optional("title", row.title), - ...optional("workspace", row.workspace), - ...optional("createdAt", row.created_at), - ...optional("updatedAt", row.updated_at), + ...optionalStoredString("title", row.title), + ...optionalStoredString("workspace", row.workspace), + ...optionalStoredTimestamp("createdAt", row.created_at), + ...optionalStoredTimestamp("updatedAt", row.updated_at), freshness: freshness.status, sourceState, capturedAt, @@ -321,11 +423,23 @@ function copyIdentity(identity: SessionIdentity): SessionIdentity { return { source: { ...identity.source }, nativeId: identity.nativeId }; } -function optional( +function optionalStoredString( key: Key, - value: string | null, + value: unknown, ): { readonly [Property in Key]?: string } { - return value === null ? {} : ({ [key]: value } as { [Property in Key]: string }); + if (value === null) return {}; + if (typeof value !== "string" || !value.isWellFormed()) { + throw new SqliteSessionIndexError("corrupt-data"); + } + return { [key]: value } as { [Property in Key]: string }; +} + +function optionalStoredTimestamp( + key: Key, + value: unknown, +): { readonly [Property in Key]?: string } { + if (value === null) return {}; + return { [key]: canonicalTimestampAt(value) } as { [Property in Key]: string }; } function booleanIntegerAt(value: unknown): boolean { @@ -356,11 +470,18 @@ interface FreshnessBatchRow extends SessionTrackingFreshnessColumns { readonly has_document: unknown; } +interface SessionSummaryBatchRow extends SessionSummaryColumns, SessionTrackingFreshnessColumns { + readonly requested_session_id: unknown; + readonly source_kind: unknown; + readonly instance_id: unknown; + readonly native_id: unknown; +} + export interface SessionSummaryColumns { - readonly title: string | null; - readonly workspace: string | null; - readonly created_at: string | null; - readonly updated_at: string | null; + readonly title: unknown; + readonly workspace: unknown; + readonly created_at: unknown; + readonly updated_at: unknown; readonly captured_at: unknown; readonly source_state: unknown; readonly source_observed_at: unknown; diff --git a/test/contracts/session-query.contract.ts b/test/contracts/session-query.contract.ts index 750bfbd..8a5a26d 100644 --- a/test/contracts/session-query.contract.ts +++ b/test/contracts/session-query.contract.ts @@ -1155,14 +1155,103 @@ export function runSessionQueryContract( } }); + test("keeps every entry filter and selection stable across page-size-one traversal", async () => { + const fixture = await createFixture(); + const corpus = sessionQueryContractCorpus(); + const present = corpus.present; + const observedAt = SESSION_QUERY_CONTRACT_TIMES.present; + const filterCases: readonly [ + string, + Parameters[0]["filter"], + ][] = [ + ["source", { source: present.identity.source.kind }], + [ + "instance", + { + source: present.identity.source.kind, + instance: present.identity.source.instanceId, + }, + ], + [ + "native ID", + { + source: present.identity.source.kind, + instance: present.identity.source.instanceId, + nativeId: present.identity.nativeId, + }, + ], + ["source state", { sourceState: "present" }], + ["workspace", { workspace: "/workspace/Primary" }], + [ + "activity", + { + activityAfter: "2026-07-14T09:29:59.999Z", + activityBefore: "2026-07-14T09:30:00.001Z", + }, + ], + [ + "capture", + { + capturedAfter: before(observedAt), + capturedBefore: after(observedAt), + }, + ], + [ + "observation", + { + observedAfter: before(observedAt), + observedBefore: after(observedAt), + }, + ], + ["session", { session: present.identity }], + [ + "entry time", + { + entryAfter: "2026-07-14T09:19:59.999Z", + entryBefore: "2026-07-14T09:50:00.001Z", + }, + ], + ["actor", { actor: "model" }], + ["origin", { origin: "model" }], + ["entry kind", { entryKind: "entry-order" }], + ["tool name", { toolName: "read_file" }], + ["tool namespace", { toolNamespace: "filesystem" }], + ]; + + try { + for (const selection of ["all", "first", "last"] as const) { + for (const [label, filter] of filterCases) { + const canonical = await fixture.query.entries( + createSessionEntryQuery({ + selection, + limit: 200, + ...(filter === undefined ? {} : { filter }), + }), + ); + expect(canonical.entries.length, `${selection} ${label}`).toBeGreaterThan(0); + expect(await traverseEntries(fixture.query, filter, 1, selection)).toEqual( + canonical.entries.map(entryKey), + ); + } + } + } finally { + await fixture.close(); + } + }); + test("traverses list and default-sized search cursors without duplicates", async () => { const fixture = await createFixture(); const corpus = sessionQueryContractCorpus(); try { const listed = await traverseList(fixture.query, 5); - expect(listed).toHaveLength(corpus.documents.length); + const canonicalList = await fixture.query.list(createSessionListQuery({ limit: 200 })); + expect(listed).toEqual(canonicalList.sessions.map(({ identity }) => key(identity))); expect(new Set(listed).size).toBe(listed.length); + const pageSizeOne = await traverseList(fixture.query, 1); + expect(pageSizeOne).toEqual(canonicalList.sessions.map(({ identity }) => key(identity))); + expect(pageSizeOne).toHaveLength(corpus.documents.length); + const firstList = await fixture.query.list(createSessionListQuery({ limit: 2 })); if (firstList.nextCursor === undefined) throw new Error("Expected list continuation"); const secondList = await fixture.query.list( @@ -1240,6 +1329,7 @@ async function traverseEntries( query: SessionQueryRepository, filter: Parameters[0]["filter"], limit: number, + selection: NonNullable[0]["selection"]> = "all", ): Promise { const found: string[] = []; let cursor: SessionQueryCursor | undefined; @@ -1247,6 +1337,7 @@ async function traverseEntries( const page = await query.entries( createSessionEntryQuery({ limit, + selection, ...(filter === undefined ? {} : { filter }), ...(cursor === undefined ? {} : { cursor }), }), diff --git a/test/infrastructure/sqlite-query-primitives.test.ts b/test/infrastructure/sqlite-query-primitives.test.ts index 9236bb2..76a5b37 100644 --- a/test/infrastructure/sqlite-query-primitives.test.ts +++ b/test/infrastructure/sqlite-query-primitives.test.ts @@ -7,6 +7,7 @@ import { truncateUtf8Around } from "../../src/infrastructure/sqlite/sqlite-query import { applyMigrations } from "../../src/infrastructure/sqlite/migrations.ts"; import { decodeQueryCursor, + encodeAnchoredQueryCursor, encodeQueryCursor, fingerprintQuery, readQueryRevision, @@ -95,6 +96,92 @@ describe("SQLite query primitives", () => { } }); + test("encodes bounded numeric anchors for list and entry continuation", () => { + const database = migratedDatabase(); + try { + const revision = readQueryRevision(database); + const fingerprint = fingerprintQuery('{"limit":200}'); + const listCursor = encodeAnchoredQueryCursor({ + command: "list", + fingerprint, + revision, + offset: 200, + anchor: { kind: "list", sessionId: 9 }, + }); + const entriesCursor = encodeAnchoredQueryCursor({ + command: "entries", + fingerprint, + revision, + offset: 400, + anchor: { kind: "entries", sessionId: 12, entryOrdinal: 34 }, + }); + + expect(decodeQueryCursor(listCursor, { command: "list", fingerprint, revision })).toEqual({ + ok: true, + offset: 200, + anchor: { kind: "list", sessionId: 9 }, + }); + expect( + decodeQueryCursor(entriesCursor, { command: "entries", fingerprint, revision }), + ).toEqual({ + ok: true, + offset: 400, + anchor: { kind: "entries", sessionId: 12, entryOrdinal: 34 }, + }); + expect(decodeQueryCursor(listCursor, { command: "entries", fingerprint, revision })).toEqual({ + ok: false, + reason: "mismatch", + }); + + expect(JSON.parse(Buffer.from(listCursor, "base64url").toString("utf8"))).toEqual({ + v: 2, + c: "list", + q: fingerprint, + l: revision.libraryInstanceId, + g: revision.writerGeneration, + o: 200, + s: 9, + }); + } finally { + database.close(); + } + }); + + test("rejects non-allowlisted anchored commands and anchor kinds at encoding", () => { + const database = migratedDatabase(); + try { + const common = { + fingerprint: fingerprintQuery("{}"), + revision: readQueryRevision(database), + offset: 1, + }; + + expect(() => + encodeAnchoredQueryCursor({ + ...common, + command: "search", + anchor: { kind: "search", sessionId: 1 }, + } as never), + ).toThrow("Invalid anchored query command"); + expect(() => + encodeAnchoredQueryCursor({ + ...common, + command: "list", + anchor: { kind: "search", sessionId: 1 }, + } as never), + ).toThrow("Invalid query cursor anchor kind"); + expect(() => + encodeAnchoredQueryCursor({ + ...common, + command: "list", + anchor: { kind: "entries", sessionId: 1, entryOrdinal: 0 }, + } as never), + ).toThrow("Query cursor anchor does not match command"); + } finally { + database.close(); + } + }); + test("rejects malformed and non-canonical cursor payloads", () => { const database = migratedDatabase(); try { @@ -111,6 +198,51 @@ describe("SQLite query primitives", () => { ok: false, reason: "invalid", }); + for (const payload of [ + { + v: 2, + c: "list", + q: expected.fingerprint, + l: expected.revision.libraryInstanceId, + g: expected.revision.writerGeneration, + o: 20, + }, + { + v: 2, + c: "search", + q: expected.fingerprint, + l: expected.revision.libraryInstanceId, + g: expected.revision.writerGeneration, + o: 20, + s: 1, + }, + { + v: 2, + c: "list", + q: expected.fingerprint, + l: expected.revision.libraryInstanceId, + g: expected.revision.writerGeneration, + o: 20, + s: -1, + }, + { + v: 2, + c: "list", + q: expected.fingerprint, + l: expected.revision.libraryInstanceId, + g: expected.revision.writerGeneration, + o: 20, + s: 1, + extra: true, + }, + ]) { + expect( + decodeQueryCursor( + Buffer.from(JSON.stringify(payload), "utf8").toString("base64url"), + expected, + ), + ).toEqual({ ok: false, reason: "invalid" }); + } } finally { database.close(); } diff --git a/test/infrastructure/sqlite-session-entry-query.test.ts b/test/infrastructure/sqlite-session-entry-query.test.ts index 765eae4..493514d 100644 --- a/test/infrastructure/sqlite-session-entry-query.test.ts +++ b/test/infrastructure/sqlite-session-entry-query.test.ts @@ -1,15 +1,33 @@ -import { DatabaseSync } from "node:sqlite"; +import { constants, DatabaseSync } from "node:sqlite"; import { describe, expect, test } from "vitest"; import { hashContent } from "../../src/domain/content-hash.ts"; -import { createSessionEntryQuery } from "../../src/domain/session-query.ts"; +import { + createSessionEntryQuery, + sessionQueryFingerprintMaterial, + type SessionEntryFilterInput, + type SessionEntryPage, + type SessionEntrySelection, +} from "../../src/domain/session-query.ts"; import type { SessionDocument, SessionIdentity } from "../../src/domain/session.ts"; import { applyMigrations, CURRENT_INDEX_SCHEMA_VERSION, } from "../../src/infrastructure/sqlite/migrations.ts"; import { createCoordinatedSqliteSessionIndex } from "../../src/infrastructure/sqlite/sqlite-session-index.ts"; +import { + decodeQueryCursor, + encodeAnchoredQueryCursor, + encodeQueryCursor, + fingerprintQuery, + readQueryRevision, +} from "../../src/infrastructure/sqlite/query-cursor.ts"; +import { + buildSqliteEntryCoordinateStatement, + listSqliteSessionEntries, + type SqliteEntryQueryWork, +} from "../../src/infrastructure/sqlite/sqlite-session-entry-query.ts"; import { createSqliteSessionQuery } from "../../src/infrastructure/sqlite/sqlite-session-query.ts"; import { acquireWriterLease, @@ -19,6 +37,279 @@ import { initializeWriterRecoveryReceipt } from "../../src/infrastructure/sqlite import { replacement } from "../contracts/session-index.contract.ts"; describe("SQLite session entry query", () => { + test("traverses binary identities without gaps for all, first, last, and filtered pages", async () => { + const database = migratedDatabase(); + try { + await seed(database, [filterDocument("é"), filterDocument("a"), filterDocument("Z")]); + const repository = createSqliteSessionQuery(database); + const cases: readonly { + readonly selection: SessionEntrySelection; + readonly filter?: SessionEntryFilterInput; + }[] = [ + { selection: "all" }, + { selection: "first", filter: { actor: "human" } }, + { selection: "last", filter: { actor: "human" } }, + { selection: "all", filter: { origin: "model" } }, + { + selection: "all", + filter: { toolName: "exec_command", toolNamespace: "functions" }, + }, + ]; + + for (const candidate of cases) { + const expected = await repository.entries( + createSessionEntryQuery({ ...candidate, limit: 200 }), + ); + const traversed = await traverseEntries(database, candidate); + expect(traversed).toEqual(entryKeys(expected)); + expect(new Set(traversed).size).toBe(traversed.length); + } + } finally { + database.close(); + } + }); + + test("accepts v1 once, emits v2 anchors, and rejects stale or invalid anchors", async () => { + const database = migratedDatabase(); + try { + await seed(database, [filterDocument("anchor")]); + const repository = createSqliteSessionQuery(database); + const query = createSessionEntryQuery({ limit: 1 }); + const revision = readQueryRevision(database); + const fingerprint = fingerprintQuery(sessionQueryFingerprintMaterial(query)); + const v1 = encodeQueryCursor({ + command: "entries", + fingerprint, + revision, + offset: 1, + }); + const fromV1 = await repository.entries(createSessionEntryQuery({ limit: 1, cursor: v1 })); + expect(entryKeys(fromV1)).toEqual(["anchor#1"]); + expect(fromV1.nextCursor).toBeDefined(); + if (fromV1.nextCursor === undefined) throw new Error("Expected v2 continuation"); + const decoded = decodeQueryCursor(fromV1.nextCursor, { + command: "entries", + fingerprint, + revision, + }); + expect(decoded).toEqual({ + ok: true, + offset: 2, + anchor: { + kind: "entries", + sessionId: sessionId(database, "anchor"), + entryOrdinal: 1, + }, + }); + await expect( + repository.entries(createSessionEntryQuery({ limit: 1, cursor: fromV1.nextCursor })), + ).resolves.toMatchObject({ + entries: [{ session: { identity: identity("anchor") }, entry: { ordinal: 2 } }], + }); + + const missingAnchor = encodeAnchoredQueryCursor({ + command: "entries", + fingerprint, + revision, + offset: 1, + anchor: { kind: "entries", sessionId: 999_999, entryOrdinal: 0 }, + }); + await expect( + repository.entries(createSessionEntryQuery({ limit: 1, cursor: missingAnchor })), + ).rejects.toMatchObject({ code: "invalid-cursor" }); + + const humanQuery = createSessionEntryQuery({ filter: { actor: "human" }, limit: 1 }); + const nonqualifyingAnchor = encodeAnchoredQueryCursor({ + command: "entries", + fingerprint: fingerprintQuery(sessionQueryFingerprintMaterial(humanQuery)), + revision, + offset: 1, + anchor: { + kind: "entries", + sessionId: sessionId(database, "anchor"), + entryOrdinal: 2, + }, + }); + await expect( + repository.entries( + createSessionEntryQuery({ + filter: { actor: "human" }, + limit: 1, + cursor: nonqualifyingAnchor, + }), + ), + ).rejects.toMatchObject({ code: "invalid-cursor" }); + + const first = await repository.entries(query); + expect(first.nextCursor).toBeDefined(); + if (first.nextCursor === undefined) throw new Error("Expected stale continuation"); + const lease = acquireWriterLease(database, "index", { + now: () => new Date("2026-07-15T13:00:00.000Z"), + token: () => "entry-query-stale-writer", + }); + try { + await expect( + repository.entries(createSessionEntryQuery({ limit: 1, cursor: first.nextCursor })), + ).rejects.toMatchObject({ code: "stale-cursor" }); + } finally { + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date("2026-07-15T13:01:00.000Z"), + }); + } + } finally { + database.close(); + } + }); + + test("batches summary hydration once at the public page maximum", async () => { + const database = migratedDatabase(); + try { + await seed( + database, + Array.from({ length: 200 }, (_, ordinal) => + document(`batch-${String(ordinal).padStart(3, "0")}`, `evidence ${String(ordinal)}`), + ), + ); + const one = await entriesWithSelectCount(database, 1); + const many = await entriesWithSelectCount(database, 200); + + expect(one.page.entries).toHaveLength(1); + expect(many.page.entries).toHaveLength(200); + expect(many.selectCount).toBe(one.selectCount); + } finally { + database.close(); + } + }); + + test("preserves every filter family and applies entry filters before first or last selection", async () => { + const database = migratedDatabase(); + try { + await seed(database, [filterDocument("filter-target"), filterDocument("other")]); + const repository = createSqliteSessionQuery(database); + const fullyFiltered = await repository.entries( + createSessionEntryQuery({ + filter: { + source: SOURCE.kind, + instance: SOURCE.instanceId, + nativeId: "filter-target", + sourceState: "present", + workspace: "/workspace/filter-target", + activityAfter: "2026-07-15T11:59:59.999Z", + activityBefore: "2026-07-15T12:00:00.001Z", + capturedAfter: "2026-07-15T11:59:59.999Z", + capturedBefore: "2026-07-15T12:00:00.001Z", + observedAfter: "2026-07-15T11:59:59.999Z", + observedBefore: "2026-07-15T12:01:00.001Z", + session: identity("filter-target"), + entryAfter: "2026-07-15T12:00:01.999Z", + entryBefore: "2026-07-15T12:00:02.001Z", + actor: "model", + origin: "model", + entryKind: "tool-call", + toolName: "exec_command", + toolNamespace: "functions", + }, + selection: "first", + limit: 10, + }), + ); + + expect( + fullyFiltered.entries.map(({ session, entry }) => [ + session.identity.nativeId, + entry.ordinal, + ]), + ).toEqual([["filter-target", 2]]); + + for (const [selection, expectedOrdinal] of [ + ["first", 1], + ["last", 4], + ] as const) { + const page = await repository.entries( + createSessionEntryQuery({ + filter: { nativeId: "filter-target", actor: "human" }, + selection, + limit: 10, + }), + ); + expect(page.entries.map(({ entry }) => entry.ordinal)).toEqual([expectedOrdinal]); + } + } finally { + database.close(); + } + }); + + test("keeps binary identity ordering under the explicit traversal", async () => { + const database = migratedDatabase(); + try { + await seed(database, [ + document("é", "accent"), + document("a", "lowercase"), + document("Z", "uppercase"), + ]); + const page = await createSqliteSessionQuery(database).entries( + createSessionEntryQuery({ limit: 10 }), + ); + + expect(page.entries.map(({ session }) => session.identity.nativeId)).toEqual(["Z", "a", "é"]); + } finally { + database.close(); + } + }); + + test("owns the explicit traversal SQL and ignores observer failures", async () => { + const database = migratedDatabase(); + try { + await seed(database, [document("observed", "observed evidence")]); + const query = createSessionEntryQuery({ limit: 10 }); + const firstStatement = buildSqliteEntryCoordinateStatement(query, { kind: "first" }); + const keysetStatement = buildSqliteEntryCoordinateStatement(query, { + kind: "keyset", + anchor: { + sessionId: 1, + entryOrdinal: 2, + sourceKind: SOURCE.kind, + instanceId: SOURCE.instanceId, + nativeId: "observed", + }, + }); + const offsetStatement = buildSqliteEntryCoordinateStatement(query, { + kind: "offset", + offset: 10, + }); + expect(firstStatement.sql).toContain( + "FROM sessions_source_instances AS source\n CROSS JOIN sessions_session_tracking AS tracking", + ); + expect(firstStatement.sql).toContain( + "WHERE tracking.source_instance_id = source.source_instance_id", + ); + expect(firstStatement.sql).toContain("AND entry.session_id = canonical.session_id"); + expect(firstStatement.sql).not.toContain("OFFSET"); + expect(keysetStatement.sql).not.toContain("OFFSET"); + expect(keysetStatement.sql).toContain("entry.ordinal > CASE"); + expect(offsetStatement.sql).toContain("OFFSET ?"); + + const work: SqliteEntryQueryWork[] = []; + const expected = listSqliteSessionEntries(database, query, { + observeWork: (event) => work.push(event), + }); + const withFailingObserver = listSqliteSessionEntries(database, query, { + observeWork: () => { + throw new Error("private observer failure"); + }, + }); + + expect(withFailingObserver).toEqual(expected); + expect(work.map(({ phase, rowCount }) => [phase, rowCount])).toEqual([ + ["coordinate-selection", 1], + ["hydration", 1], + ]); + expect(work.every(({ elapsedMilliseconds }) => elapsedMilliseconds >= 0)).toBe(true); + } finally { + database.close(); + } + }); + test("hydrates only the selected page and excludes storage-only entry fields", async () => { const database = migratedDatabase(); const safeText = "safe retained entry evidence"; @@ -124,6 +415,76 @@ function document(nativeId: string, text: string): SessionDocument { }; } +function filterDocument(nativeId: string): SessionDocument { + const text = `model evidence ${nativeId}`; + return { + identity: identity(nativeId), + title: `Filtered entry ${nativeId}`, + workspace: `/workspace/${nativeId}`, + createdAt: "2026-07-15T12:00:00.000Z", + updatedAt: "2026-07-15T12:00:00.000Z", + lineageCoverage: "complete", + relations: [], + entries: [ + { + ordinal: 0, + kind: "message", + actor: "system", + timestamp: "2026-07-15T12:00:00.000Z", + sourceLocator: { uri: `memory://${nativeId}/0` }, + content: [], + }, + { + ordinal: 1, + kind: "message", + actor: "human", + timestamp: "2026-07-15T12:00:01.000Z", + sourceLocator: { uri: `memory://${nativeId}/1` }, + content: [], + }, + { + ordinal: 2, + kind: "tool-call", + actor: "model", + timestamp: "2026-07-15T12:00:02.000Z", + toolCallId: `call-${nativeId}`, + toolName: "exec_command", + toolNamespace: "functions", + sourceLocator: { uri: `memory://${nativeId}/2` }, + content: [ + { + kind: "text", + ordinal: 0, + text, + contentHash: hashContent(text), + origin: "model", + originConfidence: "high", + sourceMetadata: {}, + }, + ], + }, + { + ordinal: 3, + kind: "tool-result", + actor: "tool", + timestamp: "2026-07-15T12:00:03.000Z", + relatedEntryOrdinal: 2, + toolCallId: `call-${nativeId}`, + sourceLocator: { uri: `memory://${nativeId}/3` }, + content: [], + }, + { + ordinal: 4, + kind: "message", + actor: "human", + timestamp: "2026-07-15T12:00:04.000Z", + sourceLocator: { uri: `memory://${nativeId}/4` }, + content: [], + }, + ], + }; +} + async function seed(database: DatabaseSync, documents: readonly SessionDocument[]): Promise { const now = () => new Date("2026-07-15T12:00:00.000Z"); const lease = acquireWriterLease(database, "index", { @@ -158,6 +519,76 @@ async function seed(database: DatabaseSync, documents: readonly SessionDocument[ } } +async function traverseEntries( + database: DatabaseSync, + candidate: { + readonly selection: SessionEntrySelection; + readonly filter?: SessionEntryFilterInput; + }, +): Promise { + const repository = createSqliteSessionQuery(database); + const result: string[] = []; + let cursor: string | undefined; + do { + const page = await repository.entries( + createSessionEntryQuery({ + ...candidate, + limit: 1, + ...(cursor === undefined ? {} : { cursor }), + }), + ); + result.push(...entryKeys(page)); + cursor = page.nextCursor; + } while (cursor !== undefined); + return result; +} + +function entryKeys(page: SessionEntryPage): readonly string[] { + return page.entries.map( + ({ session, entry }) => `${session.identity.nativeId}#${String(entry.ordinal)}`, + ); +} + +function sessionId(database: DatabaseSync, nativeId: string): number { + const row = database + .prepare( + `SELECT canonical.session_id + FROM sessions_canonical_sessions AS canonical + JOIN sessions_session_tracking AS tracking + ON tracking.session_id = canonical.session_id + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE source.kind = ? + AND source.instance_id = ? + AND tracking.native_id = ?`, + ) + .get(SOURCE.kind, SOURCE.instanceId, nativeId) as { readonly session_id?: unknown } | undefined; + const value = row?.session_id; + if (typeof value !== "number" || !Number.isSafeInteger(value)) { + throw new Error("Missing test session ID"); + } + return value; +} + +async function entriesWithSelectCount( + database: DatabaseSync, + limit: number, +): Promise<{ readonly page: SessionEntryPage; readonly selectCount: number }> { + let selectCount = 0; + database.setAuthorizer((actionCode) => { + if (actionCode === constants.SQLITE_SELECT) selectCount += 1; + return constants.SQLITE_OK; + }); + try { + return { + page: await createSqliteSessionQuery(database).entries(createSessionEntryQuery({ limit })), + selectCount, + }; + } finally { + database.setAuthorizer(null); + } +} + function migratedDatabase(): DatabaseSync { const database = new DatabaseSync(":memory:", { allowExtension: false, diff --git a/test/infrastructure/sqlite-session-query.test.ts b/test/infrastructure/sqlite-session-query.test.ts index 2ab6444..b8cd449 100644 --- a/test/infrastructure/sqlite-session-query.test.ts +++ b/test/infrastructure/sqlite-session-query.test.ts @@ -1,4 +1,4 @@ -import { DatabaseSync } from "node:sqlite"; +import { constants, DatabaseSync } from "node:sqlite"; import { describe, expect, test } from "vitest"; @@ -9,6 +9,7 @@ import { import { createSessionListQuery, createSessionSearchQuery, + sessionQueryFingerprintMaterial, } from "../../src/domain/session-query.ts"; import { hashContent } from "../../src/domain/content-hash.ts"; import type { SessionDocument } from "../../src/domain/session.ts"; @@ -16,12 +17,22 @@ import { applyMigrations, CURRENT_INDEX_SCHEMA_VERSION, } from "../../src/infrastructure/sqlite/migrations.ts"; -import { readQueryRevision } from "../../src/infrastructure/sqlite/query-cursor.ts"; +import { + decodeQueryCursor, + encodeAnchoredQueryCursor, + encodeQueryCursor, + fingerprintQuery, + readQueryRevision, +} from "../../src/infrastructure/sqlite/query-cursor.ts"; +import { readSearchContexts } from "../../src/infrastructure/sqlite/sqlite-query-context.ts"; import { createCoordinatedSqliteSessionIndex, createSqliteSessionIndexReader, } from "../../src/infrastructure/sqlite/sqlite-session-index.ts"; -import { createSqliteSessionQuery } from "../../src/infrastructure/sqlite/sqlite-session-query.ts"; +import { + buildSqliteListCoordinateStatement, + createSqliteSessionQuery, +} from "../../src/infrastructure/sqlite/sqlite-session-query.ts"; import { acquireWriterLease, interruptOwnedRunsAndReleaseWriterLease, @@ -218,6 +229,34 @@ describe("SQLite session query", () => { } }); + test("retries snippet markers for the whole selected page", async () => { + const database = migratedDatabase(); + const marker = firstSnippetMarkerCandidate(database); + const ordinaryText = "pagecollision ordinary selectedevidence"; + const collisionText = `pagecollision before ${marker.start} selectedevidence ${marker.end} after`; + await seedQueryDocuments(database, [ + markerDocument("a-selected", ordinaryText), + markerDocument("b-selected", collisionText), + ]); + try { + const result = await createSqliteSessionQuery(database).search( + createSessionSearchQuery({ text: "pagecollision", limit: 2, context: 0 }), + ); + + expect(result.hits.map((hit) => hit.session.identity.nativeId)).toEqual([ + "a-selected", + "b-selected", + ]); + expect(result.hits.map((hit) => hit.snippet.text)).toEqual([ordinaryText, collisionText]); + expect(result.hits.map((hit) => hit.snippet.contentHash)).toEqual([ + hashContent(ordinaryText), + hashContent(collisionText), + ]); + } finally { + database.close(); + } + }); + test("does not let unselected marker-like text change the selected output", async () => { const withoutCollision = migratedDatabase(); const withCollision = migratedDatabase(); @@ -244,6 +283,44 @@ describe("SQLite session query", () => { } }); + test("does not hydrate corrupt matching content outside the selected page", async () => { + const database = migratedDatabase(); + await seedQueryDocuments(database, [ + markerDocument("a-selected", "pagebounded selected evidence"), + markerDocument("b-unselected", "pagebounded unselected evidence"), + ]); + try { + database.exec("DROP TRIGGER sessions_content_values_bu"); + database + .prepare( + `UPDATE sessions_content_values + SET digest = zeroblob(32) + WHERE text = ?`, + ) + .run("pagebounded unselected evidence"); + const repository = createSqliteSessionQuery(database); + const first = await repository.search( + createSessionSearchQuery({ text: "pagebounded", limit: 1, context: 0 }), + ); + + expect(first.hits.map((hit) => hit.session.identity.nativeId)).toEqual(["a-selected"]); + expect(first.nextCursor).toBeDefined(); + if (first.nextCursor === undefined) throw new Error("Expected search cursor"); + await expect( + repository.search( + createSessionSearchQuery({ + text: "pagebounded", + limit: 1, + context: 0, + cursor: first.nextCursor, + }), + ), + ).rejects.toMatchObject({ code: "corrupt-data" }); + } finally { + database.close(); + } + }); + test("preserves complete shared-content results across bounded pages", async () => { const fixture = await seededQueryFixture(); try { @@ -360,6 +437,136 @@ describe("SQLite session query", () => { } }); + test("rebuilds bounded linked context independently for every selected primary", async () => { + const database = migratedDatabase(); + const firstLinked = Array.from({ length: 21 }, (_, index) => 2 + index); + const secondLinked = Array.from({ length: 21 }, (_, index) => 23 + index); + const document = createTestDocument({ + identity: { source: MARKER_SOURCE, nativeId: "linked-batch" }, + lineageCoverage: "complete", + entries: [ + toolCallEntry(0, "linkedbatch first primary"), + toolCallEntry(1, "linkedbatch second primary"), + ...firstLinked.map((ordinal) => toolResultEntry(ordinal, 0)), + ...secondLinked.map((ordinal) => toolResultEntry(ordinal, 1)), + ], + }); + await seedQueryDocuments(database, [document]); + try { + const result = await createSqliteSessionQuery(database).search( + createSessionSearchQuery({ text: "linkedbatch", limit: 2, context: 0 }), + ); + + expect(result.hits.map((hit) => hit.entry.ordinal)).toEqual([0, 1]); + expect(result.hits[0]?.context.map((entry) => entry.ordinal)).toEqual( + firstLinked.slice(0, 20), + ); + expect(result.hits[1]?.context.map((entry) => entry.ordinal)).toEqual( + secondLinked.slice(0, 20), + ); + for (const hit of result.hits) { + expect(hit.context.every((entry) => !entry.adjacent && entry.linked)).toBe(true); + expect(hit.linkedContextTruncated).toBe(true); + } + } finally { + database.close(); + } + }); + + test("batches hydration work across one page while preserving overlapping context", async () => { + const database = migratedDatabase(); + const document = repeatedEntryDocument("batched", 12, (ordinal) => { + return `batchedhydration evidence ${String(ordinal).padStart(2, "0")}`; + }); + await seedQueryDocuments(database, [document]); + try { + const repository = createSqliteSessionQuery(database); + const small = await searchWithSelectCount(database, () => + repository.search( + createSessionSearchQuery({ + text: "batchedhydration", + limit: 1, + context: 1, + }), + ), + ); + const large = await searchWithSelectCount(database, () => + repository.search( + createSessionSearchQuery({ + text: "batchedhydration", + limit: 10, + context: 1, + }), + ), + ); + + expect(small.page.hits).toHaveLength(1); + expect(large.page.hits).toHaveLength(10); + expect(large.selectCount).toBe(small.selectCount); + expect(large.page.hits[0]?.context.map((entry) => entry.ordinal)).toEqual([1]); + expect(large.page.hits[1]?.context.map((entry) => entry.ordinal)).toEqual([0, 2]); + expect(large.page.hits[1]?.context.map((entry) => entry.body)).toEqual([ + "batchedhydration evidence 00", + "batchedhydration evidence 02", + ]); + expect(large.page.hits[1]?.context.map((entry) => [entry.adjacent, entry.linked])).toEqual([ + [true, false], + [true, false], + ]); + } finally { + database.close(); + } + }); + + test("hydrates deduplicated context coordinates in bounded chunks", async () => { + const database = migratedDatabase(); + const document = repeatedEntryDocument( + "chunked-context", + 231, + (ordinal) => `chunk context ${String(ordinal).padStart(3, "0")}`, + ); + await seedQueryDocuments(database, [document]); + try { + const row = database + .prepare( + `SELECT tracking.session_id + FROM sessions_session_tracking AS tracking + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE source.kind = ? + AND source.instance_id = ? + AND tracking.native_id = ?`, + ) + .get( + document.identity.source.kind, + document.identity.source.instanceId, + document.identity.nativeId, + ) as { readonly session_id: number | bigint } | undefined; + if (row === undefined) throw new Error("Expected seeded session"); + const sessionId = Number(row.session_id); + const primaryOrdinals = Array.from({ length: 11 }, (_, index) => 10 + index * 21); + const contexts = readSearchContexts( + database, + primaryOrdinals.map((entryOrdinal) => ({ sessionId, entryOrdinal })), + 10, + ); + + expect(contexts).toHaveLength(primaryOrdinals.length); + for (const [index, context] of contexts.entries()) { + const primaryOrdinal = primaryOrdinals[index]!; + expect(context.entries).toHaveLength(20); + expect(context.entries.map((entry) => entry.ordinal)).toEqual([ + ...Array.from({ length: 10 }, (_, offset) => primaryOrdinal - 10 + offset), + ...Array.from({ length: 10 }, (_, offset) => primaryOrdinal + 1 + offset), + ]); + expect(context.entries.every((entry) => entry.adjacent && !entry.linked)).toBe(true); + expect(context.linkedContextTruncated).toBe(false); + } + } finally { + database.close(); + } + }); + test("uses coverage observation while effective source state is unknown", async () => { const fixture = await seededQueryFixture(); const now = () => new Date("2026-07-14T13:00:00.000Z"); @@ -476,6 +683,220 @@ describe("SQLite session query", () => { } }); + test("uses compatible v2 list keysets across null activity and binary identity ties", async () => { + const database = migratedDatabase(); + const recent = markerDocument("recent", "list keyset recent"); + const nullActivity = ["é", "a", "Z"].map((nativeId) => + markerDocumentWithoutMetadata(nativeId, `list keyset ${nativeId}`), + ); + await seedQueryDocuments(database, [recent, ...nullActivity]); + try { + const repository = createSqliteSessionQuery(database); + const query = createSessionListQuery({ limit: 1 }); + const revision = readQueryRevision(database); + const fingerprint = fingerprintQuery(sessionQueryFingerprintMaterial(query)); + const canonical = await repository.list(createSessionListQuery({ limit: 200 })); + expect(canonical.sessions.map(({ identity }) => identity.nativeId)).toEqual([ + "recent", + "Z", + "a", + "é", + ]); + + const traversed: string[] = []; + let cursor: string | undefined; + do { + const page = await repository.list( + createSessionListQuery({ + limit: 1, + ...(cursor === undefined ? {} : { cursor }), + }), + ); + traversed.push(...page.sessions.map(({ identity }) => identity.nativeId)); + cursor = page.nextCursor; + } while (cursor !== undefined); + expect(traversed).toEqual(["recent", "Z", "a", "é"]); + + const first = await repository.list(query); + expect(first.nextCursor).toBeDefined(); + if (first.nextCursor === undefined) throw new Error("Expected list continuation"); + expect( + decodeQueryCursor(first.nextCursor, { + command: "list", + fingerprint, + revision, + }), + ).toEqual({ + ok: true, + offset: 1, + anchor: { + kind: "list", + sessionId: retainedSessionId(database, recent.identity), + }, + }); + + const v1 = encodeQueryCursor({ + command: "list", + fingerprint, + revision, + offset: 1, + }); + const fromV1 = await repository.list(createSessionListQuery({ limit: 1, cursor: v1 })); + expect(fromV1.sessions.map(({ identity }) => identity.nativeId)).toEqual(["Z"]); + expect(fromV1.nextCursor).toBeDefined(); + if (fromV1.nextCursor === undefined) throw new Error("Expected v2 list continuation"); + expect( + decodeQueryCursor(fromV1.nextCursor, { + command: "list", + fingerprint, + revision, + }), + ).toMatchObject({ + ok: true, + offset: 2, + anchor: { kind: "list", sessionId: retainedSessionId(database, nullActivity[2]!.identity) }, + }); + + const firstStatement = buildSqliteListCoordinateStatement(query, { kind: "first" }); + const keysetStatement = buildSqliteListCoordinateStatement(query, { + kind: "keyset", + anchor: { + sessionId: retainedSessionId(database, recent.identity), + activity: recent.updatedAt ?? null, + sourceKind: recent.identity.source.kind, + instanceId: recent.identity.source.instanceId, + nativeId: recent.identity.nativeId, + }, + }); + const legacyStatement = buildSqliteListCoordinateStatement(query, { + kind: "offset", + offset: 1, + }); + expect(firstStatement.sql).not.toContain("OFFSET"); + expect(keysetStatement.sql).not.toContain("OFFSET"); + expect(keysetStatement.sql).toContain("WITH resolved_anchor AS MATERIALIZED"); + expect(keysetStatement.sql).toContain( + `${"COALESCE(canonical.updated_at, canonical.created_at)"} < anchor.activity`, + ); + expect(legacyStatement.sql).toContain("OFFSET ?"); + + const missingAnchor = encodeAnchoredQueryCursor({ + command: "list", + fingerprint, + revision, + offset: 1, + anchor: { kind: "list", sessionId: 999_999 }, + }); + await expect( + repository.list(createSessionListQuery({ limit: 1, cursor: missingAnchor })), + ).rejects.toMatchObject({ code: "invalid-cursor" }); + + const filtered = createSessionListQuery({ + filter: { workspace: "/workspace/synthetic" }, + limit: 1, + }); + const nonqualifying = encodeAnchoredQueryCursor({ + command: "list", + fingerprint: fingerprintQuery(sessionQueryFingerprintMaterial(filtered)), + revision, + offset: 1, + anchor: { + kind: "list", + sessionId: retainedSessionId(database, nullActivity[2]!.identity), + }, + }); + await expect( + repository.list( + createSessionListQuery({ + filter: { workspace: "/workspace/synthetic" }, + limit: 1, + cursor: nonqualifying, + }), + ), + ).rejects.toMatchObject({ code: "invalid-cursor" }); + } finally { + database.close(); + } + }); + + test("classifies malformed physical list anchors as corruption", async () => { + const database = migratedDatabase(); + const document = markerDocument("malformed-anchor", "malformed anchor evidence"); + await seedQueryDocuments(database, [ + document, + markerDocumentWithoutMetadata("later", "later anchor evidence"), + ]); + const now = () => new Date("2026-07-14T16:00:00.000Z"); + const lease = acquireWriterLease(database, "index", { + now, + token: () => "malformed-list-anchor-writer", + }); + try { + const repository = createSqliteSessionQuery(database); + const first = await repository.list(createSessionListQuery({ limit: 1 })); + expect(first.nextCursor).toBeDefined(); + if (first.nextCursor === undefined) throw new Error("Expected list continuation"); + + runLeasedImmediateTransaction(database, lease, { now }, () => { + database + .prepare( + `UPDATE sessions_canonical_sessions + SET updated_at = ? + WHERE session_id = ?`, + ) + .run("not-a-canonical-timestamp", retainedSessionId(database, document.identity)); + }); + + await expect( + repository.list(createSessionListQuery({ limit: 1, cursor: first.nextCursor })), + ).rejects.toMatchObject({ code: "corrupt-data" }); + } finally { + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date("2026-07-14T16:01:00.000Z"), + }); + database.close(); + } + }); + + test("batches list and search summaries at the public 200-row maximum", async () => { + const database = migratedDatabase(); + const documents = Array.from({ length: 200 }, (_, ordinal) => + markerDocument( + `summary-${String(ordinal).padStart(3, "0")}`, + `maximumsummary shared evidence ${String(ordinal)}`, + ), + ); + await seedQueryDocuments(database, documents); + try { + const repository = createSqliteSessionQuery(database); + const oneList = await listWithSelectCount(database, () => + repository.list(createSessionListQuery({ limit: 1 })), + ); + const maximumList = await listWithSelectCount(database, () => + repository.list(createSessionListQuery({ limit: 200 })), + ); + expect(oneList.page.sessions).toHaveLength(1); + expect(maximumList.page.sessions).toHaveLength(200); + expect(maximumList.selectCount).toBe(oneList.selectCount); + + const oneSearch = await searchWithSelectCount(database, () => + repository.search( + createSessionSearchQuery({ text: "maximumsummary", limit: 1, context: 0 }), + ), + ); + const maximumSearch = await searchWithSelectCount(database, () => + repository.search( + createSessionSearchQuery({ text: "maximumsummary", limit: 200, context: 0 }), + ), + ); + expect(oneSearch.page.hits).toHaveLength(1); + expect(maximumSearch.page.hits).toHaveLength(200); + expect(maximumSearch.selectCount).toBe(oneSearch.selectCount); + } finally { + database.close(); + } + }); + test("paginates list and search deterministically and classifies cursor failures", async () => { const fixture = await seededQueryFixture(); try { @@ -515,6 +936,9 @@ describe("SQLite session query", () => { now: () => new Date("2026-07-14T13:00:00.000Z"), token: () => "next-query-writer", }); + await expect( + repository.list(createSessionListQuery({ limit: 1, cursor: firstList.nextCursor })), + ).rejects.toBeInstanceOf(SessionQueryOperationalError); await expect( repository.search(createSessionSearchQuery({ ...searchInput, cursor })), ).rejects.toBeInstanceOf(SessionQueryOperationalError); @@ -591,6 +1015,147 @@ function markerDocument(nativeId: string, text: string): SessionDocument { }); } +function markerDocumentWithoutMetadata(nativeId: string, text: string): SessionDocument { + return createTestDocument({ + identity: { source: MARKER_SOURCE, nativeId }, + includeMetadata: false, + lineageCoverage: "complete", + entries: [ + createTestEntry({ + content: [ + createTestSegment({ + text, + origin: "human", + originConfidence: "high", + }), + ], + }), + ], + }); +} + +function repeatedEntryDocument( + nativeId: string, + entryCount: number, + textAt: (ordinal: number) => string, +): SessionDocument { + return createTestDocument({ + identity: { source: MARKER_SOURCE, nativeId }, + lineageCoverage: "complete", + entries: Array.from({ length: entryCount }, (_, ordinal) => + createTestEntry({ + ordinal, + content: [ + createTestSegment({ + text: textAt(ordinal), + origin: "human", + originConfidence: "high", + }), + ], + }), + ), + }); +} + +function toolCallEntry(ordinal: number, text: string): SessionDocument["entries"][number] { + return { + ...createTestEntry({ + ordinal, + content: [ + createTestSegment({ + text, + origin: "model", + originConfidence: "high", + }), + ], + }), + kind: "tool-call", + actor: "model", + toolName: "synthetic_tool", + }; +} + +function toolResultEntry( + ordinal: number, + relatedEntryOrdinal: number, +): SessionDocument["entries"][number] { + return { + ...createTestEntry({ + ordinal, + relatedEntryOrdinal, + content: [ + createTestSegment({ + text: `linked result ${String(ordinal)}`, + origin: "tool", + originConfidence: "high", + }), + ], + }), + kind: "tool-result", + actor: "tool", + }; +} + +async function searchWithSelectCount( + database: DatabaseSync, + search: () => ReturnType["search"]>, +): Promise<{ + readonly page: Awaited["search"]>>; + readonly selectCount: number; +}> { + let selectCount = 0; + database.setAuthorizer((actionCode) => { + if (actionCode === constants.SQLITE_SELECT) selectCount += 1; + return constants.SQLITE_OK; + }); + try { + return { page: await search(), selectCount }; + } finally { + database.setAuthorizer(null); + } +} + +async function listWithSelectCount( + database: DatabaseSync, + list: () => ReturnType["list"]>, +): Promise<{ + readonly page: Awaited["list"]>>; + readonly selectCount: number; +}> { + let selectCount = 0; + database.setAuthorizer((actionCode) => { + if (actionCode === constants.SQLITE_SELECT) selectCount += 1; + return constants.SQLITE_OK; + }); + try { + return { page: await list(), selectCount }; + } finally { + database.setAuthorizer(null); + } +} + +function retainedSessionId(database: DatabaseSync, identity: SessionDocument["identity"]): number { + const row = database + .prepare( + `SELECT tracking.session_id + FROM sessions_session_tracking AS tracking + JOIN sessions_source_instances AS source + ON source.source_instance_id = tracking.source_instance_id + WHERE source.kind = ? + AND source.instance_id = ? + AND tracking.native_id = ?`, + ) + .get(identity.source.kind, identity.source.instanceId, identity.nativeId) as + | { readonly session_id?: unknown } + | undefined; + const value = row?.session_id; + const sessionId = typeof value === "bigint" ? Number(value) : value; + if (typeof sessionId !== "number" || !Number.isSafeInteger(sessionId)) { + throw new Error("Missing retained session ID"); + } + return sessionId; +} + function firstSnippetMarkerCandidate(database: DatabaseSync): { readonly start: string; readonly end: string; diff --git a/test/infrastructure/sqlite-session-state.test.ts b/test/infrastructure/sqlite-session-state.test.ts new file mode 100644 index 0000000..854a214 --- /dev/null +++ b/test/infrastructure/sqlite-session-state.test.ts @@ -0,0 +1,250 @@ +import { DatabaseSync } from "node:sqlite"; + +import { describe, expect, test } from "vitest"; + +import { + CURRENT_INDEX_SCHEMA_VERSION, + applyMigrations, +} from "../../src/infrastructure/sqlite/migrations.ts"; +import { createCoordinatedSqliteSessionIndex } from "../../src/infrastructure/sqlite/sqlite-session-index.ts"; +import { + decodeRetainedSessionSummary, + findSessionTracking, + readSessionFreshness, + readSessionSummariesBatch, + readSessionSummary, +} from "../../src/infrastructure/sqlite/sqlite-session-state.ts"; +import { + acquireWriterLease, + interruptOwnedRunsAndReleaseWriterLease, + runLeasedImmediateTransaction, +} from "../../src/infrastructure/sqlite/writer-lease.ts"; +import { + clearWriterRecoveryReceiptInTransaction, + initializeWriterRecoveryReceiptInTransaction, +} from "../../src/infrastructure/sqlite/writer-recovery-receipt.ts"; +import { replacement } from "../contracts/session-index.contract.ts"; +import { createTestDocument, createTestIdentity } from "../fixtures/session.ts"; + +describe("SQLite retained session summaries", () => { + test("hydrates a bounded identity-checked batch with point-read parity", async () => { + const database = await seededDatabase(); + try { + const first = createTestIdentity("summary-one"); + const second = createTestIdentity("summary-two"); + const firstId = sessionId(database, first); + const secondId = sessionId(database, second); + + const summaries = readSessionSummariesBatch(database, [ + { sessionId: secondId, identity: second }, + { sessionId: firstId, identity: first }, + ]); + + expect([...summaries.keys()]).toEqual([secondId, firstId]); + expect(summaries.get(firstId)).toEqual(readSessionSummary(database, first)); + expect(summaries.get(secondId)).toEqual(readSessionSummary(database, second)); + expect(readSessionSummariesBatch(database, [])).toEqual(new Map()); + } finally { + database.close(); + } + }); + + test("rejects ambiguous, mismatched, and oversized requests", async () => { + const database = await seededDatabase(); + try { + const first = createTestIdentity("summary-one"); + const second = createTestIdentity("summary-two"); + const firstId = sessionId(database, first); + + expect(() => + readSessionSummariesBatch(database, [ + { sessionId: firstId, identity: first }, + { sessionId: firstId, identity: first }, + ]), + ).toThrow(TypeError); + expect(() => + readSessionSummariesBatch(database, [{ sessionId: firstId, identity: second }]), + ).toThrow("SQLite session index operation failed: corrupt-data"); + expect(() => + readSessionSummariesBatch( + database, + Array.from({ length: 201 }, (_, sessionId) => ({ sessionId, identity: first })), + ), + ).toThrow(TypeError); + } finally { + database.close(); + } + }); + + test("executes the exact 200-request batch with 800 bound values", async () => { + const identities = Array.from({ length: 200 }, (_, ordinal) => + createTestIdentity(`summary-${String(ordinal).padStart(3, "0")}`), + ); + const database = await seededDatabase(identities); + try { + const requests = identities.map((identity) => ({ + sessionId: sessionId(database, identity), + identity, + })); + + const summaries = readSessionSummariesBatch(database, requests); + + expect(summaries.size).toBe(200); + for (const request of requests) { + expect(summaries.get(request.sessionId)).toEqual( + readSessionSummary(database, request.identity), + ); + } + } finally { + database.close(); + } + }); + + test.each(["created_at", "updated_at"] as const)( + "classifies malformed stored %s as corrupt in point and batch reads", + async (column) => { + const database = await seededDatabase(); + try { + const identity = createTestIdentity("summary-one"); + const retainedSessionId = sessionId(database, identity); + mutateCanonicalTimestamp(database, retainedSessionId, column); + + expect(() => readSessionSummary(database, identity)).toThrow( + "SQLite session index operation failed: corrupt-data", + ); + expect(() => + readSessionSummariesBatch(database, [{ sessionId: retainedSessionId, identity }]), + ).toThrow("SQLite session index operation failed: corrupt-data"); + } finally { + database.close(); + } + }, + ); + + test.each(["title", "workspace"] as const)( + "rejects a non-well-formed optional stored %s in the shared decoder", + async (field) => { + const database = await seededDatabase(); + try { + const identity = createTestIdentity("summary-one"); + const summary = readSessionSummary(database, identity); + const freshness = readSessionFreshness(database, identity); + if ( + summary === undefined || + (freshness.status !== "current" && freshness.status !== "stale") + ) { + throw new Error("Expected retained summary fixture"); + } + const columns = { + title: null, + workspace: null, + created_at: summary.createdAt ?? null, + updated_at: summary.updatedAt ?? null, + captured_at: summary.capturedAt, + source_state: summary.sourceState, + source_observed_at: summary.sourceObservedAt, + document_digest_scheme: summary.documentDigest.scheme, + document_digest: Buffer.from(summary.documentDigest.digest, "hex"), + [field]: "\ud800", + }; + + expect(() => decodeRetainedSessionSummary(identity, columns, freshness)).toThrow( + "SQLite session index operation failed: corrupt-data", + ); + } finally { + database.close(); + } + }, + ); +}); + +async function seededDatabase( + identities: readonly ReturnType[] = [ + createTestIdentity("summary-one"), + createTestIdentity("summary-two"), + ], +): Promise { + const database = new DatabaseSync(":memory:", { + allowExtension: false, + defensive: true, + enableDoubleQuotedStringLiterals: false, + enableForeignKeyConstraints: true, + }); + database.exec("PRAGMA trusted_schema = OFF"); + applyMigrations(database); + const now = () => new Date("2026-07-23T12:00:00.000Z"); + const lease = acquireWriterLease(database, "index", { + now, + token: () => "summary-batch-writer", + }); + runLeasedImmediateTransaction(database, lease, { now }, (transactionNow) => { + clearWriterRecoveryReceiptInTransaction(database, lease, { now: transactionNow }); + initializeWriterRecoveryReceiptInTransaction(database, lease, { + now: transactionNow, + schemaVersion: CURRENT_INDEX_SCHEMA_VERSION, + }); + }); + const index = createCoordinatedSqliteSessionIndex(database, { + lease, + now, + schemaVersion: CURRENT_INDEX_SCHEMA_VERSION, + }); + const firstIdentity = identities[0]; + if (firstIdentity === undefined) throw new TypeError("Summary fixture requires one identity"); + const run = await index.startRun({ + source: firstIdentity.source, + startedAt: "2026-07-23T12:01:00.000Z", + }); + for (const identity of identities) { + const document = createTestDocument({ identity }); + await index.replaceSession(run, replacement(identity, identity.nativeId, document)); + } + await index.finishRun(run, { + status: "completed", + finishedAt: "2026-07-23T12:02:00.000Z", + }); + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date("2026-07-23T12:03:00.000Z"), + }); + return database; +} + +function mutateCanonicalTimestamp( + database: DatabaseSync, + sessionId: number, + column: "created_at" | "updated_at", +): void { + const now = () => new Date("2026-07-23T13:00:00.000Z"); + const lease = acquireWriterLease(database, "index", { + now, + token: () => `summary-corruption-${column}`, + }); + try { + runLeasedImmediateTransaction(database, lease, { now }, () => { + database + .prepare( + `UPDATE sessions_canonical_sessions + SET ${column} = ? + WHERE session_id = ?`, + ) + .run("not-a-canonical-timestamp", sessionId); + }); + } finally { + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date("2026-07-23T13:01:00.000Z"), + }); + } +} + +function sessionId( + database: DatabaseSync, + identity: ReturnType, +): number { + const row = findSessionTracking(database, identity); + const value = row?.session_id; + const sessionId = typeof value === "bigint" ? Number(value) : value; + if (typeof sessionId !== "number" || !Number.isSafeInteger(sessionId)) { + throw new Error("Missing test session ID"); + } + return sessionId; +} diff --git a/test/query-planner-statistics-measurement.test.ts b/test/query-planner-statistics-measurement.test.ts new file mode 100644 index 0000000..0905de0 --- /dev/null +++ b/test/query-planner-statistics-measurement.test.ts @@ -0,0 +1,96 @@ +import { spawnSync } from "node:child_process"; +import path from "node:path"; + +import { describe, expect, test } from "vitest"; + +const EXPECTED_CASES = [ + "entries-broad", + "entries-narrow", + "list-activity", + "list-identity", + "manifest", + "search-broad-all", + "search-broad-any", + "search-selective-all", + "search-selective-any", +] as const; + +describe("query planner statistics measurement", () => { + test("isolates statistics mutations and proves semantic equality and cleanup", () => { + const script = path.resolve( + import.meta.dirname, + "../scripts/measure-query-planner-statistics.ts", + ); + const result = spawnSync(process.execPath, [script, "--contract"], { + encoding: "utf8", + timeout: 30_000, + }); + + if (result.status !== 0) throw new Error(result.stderr); + expect(result.status).toBe(0); + expect(result.stderr).toBe(""); + const report = JSON.parse(result.stdout) as MeasurementReport; + expect(report).toMatchObject({ + schemaVersion: 1, + mode: "contract", + clonesVerifiedExact: true, + semanticEquality: true, + temporaryCleanup: true, + variants: { + control: { + statisticsCommand: "none", + databaseByteGrowth: 0, + statistics: { tables: [] }, + }, + analyze: { + statisticsCommand: "ANALYZE", + }, + optimize: { + statisticsCommand: "PRAGMA optimize = 0x10002", + }, + }, + }); + expect(Object.keys(report.cases).sort()).toEqual(EXPECTED_CASES); + expect(report.variants.analyze.statistics.tables.length).toBeGreaterThan(0); + expect(report.variants.optimize.statistics.tables.length).toBeGreaterThan(0); + for (const measurement of Object.values(report.cases)) { + expect(measurement.semanticEqual).toBe(true); + expect(measurement.plans.control.length).toBeGreaterThan(0); + expect(measurement.elapsedMs.control.samples).toHaveLength(report.timingRounds); + expect(measurement.elapsedMs.analyze.samples).toHaveLength(report.timingRounds); + expect(measurement.elapsedMs.optimize.samples).toHaveLength(report.timingRounds); + } + expect(result.stdout).not.toContain("sessions-query-planner-statistics-"); + expect(result.stdout).not.toContain("memory://"); + }); +}); + +interface MeasurementReport { + readonly schemaVersion: number; + readonly mode: string; + readonly clonesVerifiedExact: boolean; + readonly semanticEquality: boolean; + readonly temporaryCleanup: boolean; + readonly timingRounds: number; + readonly variants: Record< + "control" | "analyze" | "optimize", + { + readonly statisticsCommand: string; + readonly databaseByteGrowth: number; + readonly statistics: { + readonly tables: readonly unknown[]; + }; + } + >; + readonly cases: Record< + string, + { + readonly semanticEqual: boolean; + readonly elapsedMs: Record< + "control" | "analyze" | "optimize", + { readonly samples: readonly number[] } + >; + readonly plans: Record<"control" | "analyze" | "optimize", readonly string[]>; + } + >; +} From a77659194556776659d69a46deae70b2403febeb Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 11:50:08 -0700 Subject: [PATCH 2/5] fix: close query review gaps --- .../sqlite/sqlite-query-context.ts | 110 +++++++----- .../sqlite/sqlite-session-query.ts | 2 +- test/contracts/session-query.contract.ts | 31 +++- .../sqlite-query-context.test.ts | 159 ++++++++++++++++++ .../sqlite-session-query.test.ts | 48 ++++++ 5 files changed, 304 insertions(+), 46 deletions(-) create mode 100644 test/infrastructure/sqlite-query-context.test.ts diff --git a/src/infrastructure/sqlite/sqlite-query-context.ts b/src/infrastructure/sqlite/sqlite-query-context.ts index 40299c9..061419a 100644 --- a/src/infrastructure/sqlite/sqlite-query-context.ts +++ b/src/infrastructure/sqlite/sqlite-query-context.ts @@ -6,6 +6,7 @@ import { type SessionSearchContextEntry, type SessionSearchEntry, } from "../../domain/session-query.ts"; +import { isCanonicalTimestamp } from "../../domain/canonical-timestamp.ts"; import type { Actor } from "../../domain/session.ts"; import { SqliteSessionIndexError } from "./sqlite-session-transaction.ts"; @@ -14,7 +15,7 @@ const ACTORS = new Set(["human", "model", "tool", "system", "unknown"]); export const SQLITE_SEARCH_CONTEXT_COORDINATE_CHUNK_SIZE = 200; // At most twenty adjacent entries can precede the twenty-one extras needed // to prove linked-context truncation. -const LINKED_CANDIDATE_LIMIT = MAX_SESSION_SEARCH_LINKED_CONTEXT * 2 + 1; +export const SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT = MAX_SESSION_SEARCH_LINKED_CONTEXT * 2 + 1; export interface SqliteSearchContext { readonly entries: readonly SessionSearchContextEntry[]; @@ -154,40 +155,11 @@ function readLinkedOrdinals( ): readonly (readonly number[])[] { const selected = selectedPrimaryCte(primaries); const rows = database - .prepare( - `${selected.sql}, - ranked_links AS ( - SELECT selected.primary_index, - selected.session_id, - selected.primary_ordinal, - candidate.ordinal AS candidate_ordinal, - ROW_NUMBER() OVER ( - PARTITION BY selected.primary_index - ORDER BY candidate.ordinal - ) AS candidate_rank - FROM selected_primaries AS selected - JOIN sessions_entries AS primary_entry - ON primary_entry.session_id = selected.session_id - AND primary_entry.ordinal = selected.primary_ordinal - JOIN sessions_entries AS candidate - ON candidate.session_id = primary_entry.session_id - AND candidate.ordinal <> primary_entry.ordinal - WHERE ( - primary_entry.related_entry_ordinal = candidate.ordinal - OR candidate.related_entry_ordinal = primary_entry.ordinal - ) - AND ( - (primary_entry.kind = 'tool-call' AND candidate.kind = 'tool-result') - OR - (primary_entry.kind = 'tool-result' AND candidate.kind = 'tool-call') - ) - ) - SELECT primary_index, session_id, primary_ordinal, candidate_ordinal - FROM ranked_links - WHERE candidate_rank <= ? - ORDER BY primary_index, candidate_ordinal`, - ) - .all(...selected.parameters, LINKED_CANDIDATE_LIMIT) as unknown as readonly LinkedRow[]; + .prepare(sqliteLinkedContextDiscoverySql(primaries.length)) + .all( + ...selected.parameters, + SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT, + ) as unknown as readonly LinkedRow[]; const result = primaries.map(() => [] as number[]); for (const row of rows) { const primaryIndex = integerAt(row.primary_index); @@ -211,6 +183,60 @@ function readLinkedOrdinals( return result; } +export function sqliteLinkedContextDiscoverySql(primaryCount: number): string { + if (!Number.isSafeInteger(primaryCount) || primaryCount < 1) { + throw new TypeError("Primary count must be a positive safe integer"); + } + return `${selectedPrimaryCteSql(primaryCount)}, + linked_candidates ( + primary_index, + session_id, + primary_ordinal, + candidate_ordinal, + candidate_rank + ) AS ( + SELECT selected.primary_index, + selected.session_id, + selected.primary_ordinal, + -1, + 0 + FROM selected_primaries AS selected + UNION ALL + SELECT linked.primary_index, + linked.session_id, + linked.primary_ordinal, + ( + SELECT candidate.ordinal + FROM sessions_entries AS primary_entry + JOIN sessions_entries AS candidate + ON candidate.session_id = primary_entry.session_id + AND candidate.ordinal > linked.candidate_ordinal + WHERE primary_entry.session_id = linked.session_id + AND primary_entry.ordinal = linked.primary_ordinal + AND ( + primary_entry.related_entry_ordinal = candidate.ordinal + OR candidate.related_entry_ordinal = primary_entry.ordinal + ) + AND ( + (primary_entry.kind = 'tool-call' AND candidate.kind = 'tool-result') + OR + (primary_entry.kind = 'tool-result' AND candidate.kind = 'tool-call') + ) + ORDER BY candidate.ordinal + LIMIT 1 + ), + linked.candidate_rank + 1 + FROM linked_candidates AS linked + WHERE linked.candidate_rank < ? + AND linked.candidate_ordinal IS NOT NULL + ) + SELECT primary_index, session_id, primary_ordinal, candidate_ordinal + FROM linked_candidates + WHERE candidate_rank > 0 + AND candidate_ordinal IS NOT NULL + ORDER BY primary_index, candidate_ordinal`; +} + function readContextEntries( database: DatabaseSync, coordinates: readonly SqliteSearchContextCoordinate[], @@ -311,9 +337,7 @@ function selectedContextCoordinates( function selectedPrimaryCte(primaries: readonly SqliteSearchContextCoordinate[]): SelectedCte { return { - sql: `WITH selected_primaries(primary_index, session_id, primary_ordinal) AS ( - VALUES ${primaries.map(() => "(?, ?, ?)").join(", ")} - )`, + sql: selectedPrimaryCteSql(primaries.length), parameters: primaries.flatMap((primary, primaryIndex) => [ primaryIndex, primary.sessionId, @@ -322,6 +346,12 @@ function selectedPrimaryCte(primaries: readonly SqliteSearchContextCoordinate[]) }; } +function selectedPrimaryCteSql(primaryCount: number): string { + return `WITH RECURSIVE selected_primaries(primary_index, session_id, primary_ordinal) AS ( + VALUES ${Array.from({ length: primaryCount }, () => "(?, ?, ?)").join(", ")} + )`; +} + function selectedContextCte(coordinates: readonly SqliteSearchContextCoordinate[]): SelectedCte { return { sql: `WITH selected_context(coordinate_index, session_id, entry_ordinal) AS ( @@ -348,9 +378,11 @@ export function entryAt(row: EntryRow): SessionSearchEntry { if (typeof row.kind !== "string" || !row.kind.isWellFormed() || !ACTORS.has(row.actor)) { throw new SqliteSessionIndexError("corrupt-data"); } + const timestamp = optionalString(row.timestamp); const toolName = optionalString(row.tool_name); const toolNamespace = optionalString(row.tool_namespace); if ( + (timestamp !== undefined && !isCanonicalTimestamp(timestamp)) || (row.kind !== "tool-call" && (toolName !== undefined || toolNamespace !== undefined)) || (toolNamespace !== undefined && toolName === undefined) ) { @@ -360,7 +392,7 @@ export function entryAt(row: EntryRow): SessionSearchEntry { ordinal, kind: row.kind, actor: row.actor, - ...optionalStringProperty("timestamp", row.timestamp), + ...(timestamp === undefined ? {} : { timestamp }), ...optionalIntegerProperty("relatedEntryOrdinal", row.related_entry_ordinal), ...optionalStringProperty("toolCallId", row.tool_call_id), ...(toolName === undefined ? {} : { toolName }), diff --git a/src/infrastructure/sqlite/sqlite-session-query.ts b/src/infrastructure/sqlite/sqlite-session-query.ts index 8cb2beb..690743c 100644 --- a/src/infrastructure/sqlite/sqlite-session-query.ts +++ b/src/infrastructure/sqlite/sqlite-session-query.ts @@ -100,7 +100,7 @@ function listSessions(database: DatabaseSync, query: SessionListQuery): SessionL } if (!sameIdentity(summary.identity, identity)) throw new SqliteSessionIndexError("corrupt-data"); - return Object.freeze({ ...freezeSummary(summary), root: resolveRoot(identity) }); + return Object.freeze({ ...summary, root: resolveRoot(identity) }); }); const nextCursor = rows.length > query.limit diff --git a/test/contracts/session-query.contract.ts b/test/contracts/session-query.contract.ts index 8a5a26d..25a617c 100644 --- a/test/contracts/session-query.contract.ts +++ b/test/contracts/session-query.contract.ts @@ -13,6 +13,8 @@ import { createSessionEntryQuery, createSessionListQuery, createSessionSearchQuery, + type SessionEntryInventoryItem, + type SessionEntryPage, type SessionQueryCursor, } from "../../src/domain/session-query.ts"; import type { SessionIdentity } from "../../src/domain/session.ts"; @@ -1229,9 +1231,11 @@ export function runSessionQueryContract( }), ); expect(canonical.entries.length, `${selection} ${label}`).toBeGreaterThan(0); - expect(await traverseEntries(fixture.query, filter, 1, selection)).toEqual( - canonical.entries.map(entryKey), - ); + const traversed = await traverseEntryPages(fixture.query, filter, 1, selection); + expect(traversed.entries, `${selection} ${label}`).toEqual(canonical.entries); + for (const scope of traversed.captureScopes) { + expect(scope, `${selection} ${label}`).toEqual(canonical.captureScope); + } } } } finally { @@ -1331,7 +1335,21 @@ async function traverseEntries( limit: number, selection: NonNullable[0]["selection"]> = "all", ): Promise { - const found: string[] = []; + const traversed = await traverseEntryPages(query, filter, limit, selection); + return traversed.entries.map(entryKey); +} + +async function traverseEntryPages( + query: SessionQueryRepository, + filter: Parameters[0]["filter"], + limit: number, + selection: NonNullable[0]["selection"]>, +): Promise<{ + readonly entries: readonly SessionEntryInventoryItem[]; + readonly captureScopes: readonly SessionEntryPage["captureScope"][]; +}> { + const entries: SessionEntryInventoryItem[] = []; + const captureScopes: SessionEntryPage["captureScope"][] = []; let cursor: SessionQueryCursor | undefined; do { const page = await query.entries( @@ -1342,10 +1360,11 @@ async function traverseEntries( ...(cursor === undefined ? {} : { cursor }), }), ); - found.push(...page.entries.map(entryKey)); + entries.push(...page.entries); + captureScopes.push(page.captureScope); cursor = page.nextCursor; } while (cursor !== undefined); - return found; + return { entries, captureScopes }; } async function expectStateAt( diff --git a/test/infrastructure/sqlite-query-context.test.ts b/test/infrastructure/sqlite-query-context.test.ts new file mode 100644 index 0000000..b4dc602 --- /dev/null +++ b/test/infrastructure/sqlite-query-context.test.ts @@ -0,0 +1,159 @@ +import { DatabaseSync } from "node:sqlite"; + +import { describe, expect, test } from "vitest"; + +import { + entryAt, + readSearchContexts, + sqliteLinkedContextDiscoverySql, + SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT, +} from "../../src/infrastructure/sqlite/sqlite-query-context.ts"; + +describe("SQLite query context", () => { + test("bounds high-fan-out linked discovery per primary with indexed next probes", () => { + const database = contextDatabase(); + try { + seedLinkedFanOut(database, 1, 500); + seedLinkedFanOut(database, 2, 500); + const sql = sqliteLinkedContextDiscoverySql(2); + const parameters = [0, 1, 0, 1, 2, 0, SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT]; + const discovered = database.prepare(sql).all(...parameters) as unknown as readonly { + readonly primary_index: number | bigint; + readonly candidate_ordinal: number | bigint; + }[]; + + expect(discovered).toHaveLength(SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT * 2); + for (const primaryIndex of [0, 1]) { + expect( + discovered + .filter((row) => Number(row.primary_index) === primaryIndex) + .map((row) => Number(row.candidate_ordinal)), + ).toEqual( + Array.from({ length: SQLITE_SEARCH_LINKED_CANDIDATE_LIMIT }, (_, index) => index + 1), + ); + } + + const contexts = readSearchContexts( + database, + [ + { sessionId: 1, entryOrdinal: 0 }, + { sessionId: 2, entryOrdinal: 0 }, + ], + 0, + ); + expect(contexts).toHaveLength(2); + for (const context of contexts) { + expect(context.entries.map((entry) => entry.ordinal)).toEqual( + Array.from({ length: 20 }, (_, index) => index + 1), + ); + expect(context.entries.every((entry) => entry.linked && !entry.adjacent)).toBe(true); + expect(context.linkedContextTruncated).toBe(true); + } + + expect(sql).toContain("WITH RECURSIVE"); + expect(sql).not.toContain("ROW_NUMBER"); + const plan = database + .prepare(`EXPLAIN QUERY PLAN ${sql}`) + .all(...parameters) + .map((row) => String(row.detail).replaceAll(/\s+/gu, " ").trim()); + expect( + plan.some( + (detail) => + detail.includes("SEARCH candidate") && + detail.includes("session_id=?") && + detail.includes("ordinal>?"), + ), + ).toBe(true); + expect( + plan.some((detail) => detail.includes("SCAN candidate") && !detail.includes("SEARCH")), + ).toBe(false); + } finally { + database.close(); + } + }); + + test("rejects a schema-valid but non-canonical stored timestamp", () => { + expect(() => + entryAt({ + ordinal: 0, + kind: "message", + actor: "human", + timestamp: "2026-7-22T12:00:00Z", + related_entry_ordinal: null, + tool_call_id: null, + tool_name: null, + tool_namespace: null, + }), + ).toThrow(expect.objectContaining({ code: "corrupt-data" })); + }); +}); + +function contextDatabase(): DatabaseSync { + const database = new DatabaseSync(":memory:", { + allowExtension: false, + defensive: true, + enableDoubleQuotedStringLiterals: false, + enableForeignKeyConstraints: true, + }); + database.exec(` + CREATE TABLE sessions_entries ( + session_id INTEGER NOT NULL, + ordinal INTEGER NOT NULL, + kind TEXT NOT NULL, + actor TEXT NOT NULL, + timestamp TEXT, + related_entry_ordinal INTEGER, + tool_call_id TEXT, + tool_name TEXT, + tool_namespace TEXT, + PRIMARY KEY (session_id, ordinal) + ) STRICT; + CREATE TABLE sessions_content_occurrences ( + session_id INTEGER NOT NULL, + entry_ordinal INTEGER NOT NULL, + segment_ordinal INTEGER NOT NULL, + content_id INTEGER, + PRIMARY KEY (session_id, entry_ordinal, segment_ordinal) + ) STRICT; + CREATE TABLE sessions_content_values ( + content_id INTEGER PRIMARY KEY, + text TEXT NOT NULL + ) STRICT; + `); + return database; +} + +function seedLinkedFanOut(database: DatabaseSync, sessionId: number, candidates: number): void { + const insert = database.prepare( + `INSERT INTO sessions_entries ( + session_id, + ordinal, + kind, + actor, + timestamp, + related_entry_ordinal, + tool_call_id, + tool_name, + tool_namespace + ) VALUES (?, ?, ?, ?, ?, ?, NULL, ?, NULL)`, + ); + database.exec("BEGIN"); + try { + insert.run( + sessionId, + 0, + "tool-call", + "model", + "2026-07-22T12:00:00.000Z", + null, + "synthetic_tool", + ); + for (let ordinal = 1; ordinal <= candidates; ordinal += 1) { + insert.run(sessionId, ordinal, "tool-result", "tool", "2026-07-22T12:00:00.000Z", 0, null); + } + database.exec("COMMIT"); + } catch (error) { + database.exec("ROLLBACK"); + throw error; + } +} diff --git a/test/infrastructure/sqlite-session-query.test.ts b/test/infrastructure/sqlite-session-query.test.ts index b8cd449..77d2825 100644 --- a/test/infrastructure/sqlite-session-query.test.ts +++ b/test/infrastructure/sqlite-session-query.test.ts @@ -567,6 +567,54 @@ describe("SQLite session query", () => { } }); + test("fails the whole search when corruption is reached in a later context chunk", async () => { + const database = migratedDatabase(); + const primaryOrdinals = new Set(Array.from({ length: 11 }, (_, index) => 10 + index * 21)); + const document = repeatedEntryDocument("later-context-corruption", 231, (ordinal) => + primaryOrdinals.has(ordinal) + ? `laterchunkfailure evidence ${String(ordinal)}` + : `context evidence ${String(ordinal)}`, + ); + await seedQueryDocuments(database, [document]); + try { + const repository = createSqliteSessionQuery(database); + const query = createSessionSearchQuery({ + text: "laterchunkfailure", + limit: 11, + context: 10, + }); + const valid = await repository.search(query); + expect(valid.hits).toHaveLength(11); + expect(valid.hits.every((hit) => hit.context.length === 20)).toBe(true); + + const now = () => new Date("2026-07-14T17:00:00.000Z"); + const lease = acquireWriterLease(database, "index", { + now, + token: () => "later-context-corruption-writer", + }); + try { + runLeasedImmediateTransaction(database, lease, { now }, () => { + database + .prepare( + `UPDATE sessions_entries + SET timestamp = ? + WHERE session_id = ? + AND ordinal = ?`, + ) + .run("not-a-canonical-timestamp", retainedSessionId(database, document.identity), 230); + }); + + await expect(repository.search(query)).rejects.toMatchObject({ code: "corrupt-data" }); + } finally { + interruptOwnedRunsAndReleaseWriterLease(database, lease, { + now: () => new Date("2026-07-14T17:01:00.000Z"), + }); + } + } finally { + database.close(); + } + }); + test("uses coverage observation while effective source state is unknown", async () => { const fixture = await seededQueryFixture(); const now = () => new Date("2026-07-14T13:00:00.000Z"); From 376901d0db6b69099545a1118cf66e7743fa64b8 Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 11:54:50 -0700 Subject: [PATCH 3/5] refactor: simplify query verification --- scripts/measure-query-planner-statistics.ts | 10 ---------- .../sqlite/sqlite-session-entry-query.ts | 14 +++----------- .../sqlite/sqlite-session-query.ts | 16 ++++------------ 3 files changed, 7 insertions(+), 33 deletions(-) diff --git a/scripts/measure-query-planner-statistics.ts b/scripts/measure-query-planner-statistics.ts index 0855a76..5df5068 100644 --- a/scripts/measure-query-planner-statistics.ts +++ b/scripts/measure-query-planner-statistics.ts @@ -123,7 +123,6 @@ async function runMeasurement(root: string): Promise { regularSessionEntries: REGULAR_SESSION_ENTRIES, }, clonesVerifiedExact: true, - explicitEntryTraversal: await hasExplicitEntryTraversal(), timingRounds: TIMING_ROUNDS, variants: mutationReports, cases, @@ -699,14 +698,6 @@ function openReadDatabase(file: string): DatabaseSync { return database; } -async function hasExplicitEntryTraversal(): Promise { - const source = await readFile( - path.resolve(import.meta.dirname, "../src/infrastructure/sqlite/sqlite-session-entry-query.ts"), - "utf8", - ); - return /CROSS JOIN sessions_session_tracking/u.test(source); -} - async function assertNoSidecars(file: string): Promise { for (const suffix of ["-wal", "-shm"] as const) { try { @@ -797,7 +788,6 @@ interface MeasurementReport { readonly regularSessionEntries: number; }; readonly clonesVerifiedExact: boolean; - readonly explicitEntryTraversal: boolean; readonly timingRounds: number; readonly variants: Record; readonly cases: Record; diff --git a/src/infrastructure/sqlite/sqlite-session-entry-query.ts b/src/infrastructure/sqlite/sqlite-session-entry-query.ts index ffdba09..f468040 100644 --- a/src/infrastructure/sqlite/sqlite-session-entry-query.ts +++ b/src/infrastructure/sqlite/sqlite-session-entry-query.ts @@ -16,7 +16,7 @@ import { type SessionEntryQuery, type SessionQuerySummary, } from "../../domain/session-query.ts"; -import { isSessionIdentity } from "../../domain/session-identity.ts"; +import { isSessionIdentity, sameSessionIdentity } from "../../domain/session-identity.ts"; import type { ContentOrigin, OriginConfidence, SessionIdentity } from "../../domain/session.ts"; import { decodeQueryCursor, @@ -266,7 +266,7 @@ function hydrateEntries( throw new SqliteSessionIndexError("corrupt-data"); } const summary = summaries.get(coordinate.sessionId); - if (summary === undefined || !sameIdentity(summary.identity, identity)) { + if (summary === undefined || !sameSessionIdentity(summary.identity, identity)) { throw new SqliteSessionIndexError("corrupt-data"); } const root = resolveRoot(identity); @@ -295,7 +295,7 @@ function readEntrySummaries( if (identity === undefined) throw new SqliteSessionIndexError("corrupt-data"); const previous = requests.get(coordinate.sessionId); if (previous !== undefined) { - if (!sameIdentity(previous.identity, identity)) { + if (!sameSessionIdentity(previous.identity, identity)) { throw new SqliteSessionIndexError("corrupt-data"); } continue; @@ -575,14 +575,6 @@ function freezeSummary(summary: SessionQuerySummary): SessionQuerySummary { }); } -function sameIdentity(left: SessionIdentity, right: SessionIdentity): boolean { - return ( - left.source.kind === right.source.kind && - left.source.instanceId === right.source.instanceId && - left.nativeId === right.nativeId - ); -} - function storedString(value: unknown): string { if (typeof value !== "string" || !value.isWellFormed()) { throw new SqliteSessionIndexError("corrupt-data"); diff --git a/src/infrastructure/sqlite/sqlite-session-query.ts b/src/infrastructure/sqlite/sqlite-session-query.ts index 690743c..b5e0309 100644 --- a/src/infrastructure/sqlite/sqlite-session-query.ts +++ b/src/infrastructure/sqlite/sqlite-session-query.ts @@ -19,7 +19,7 @@ import { import { isCanonicalTimestamp } from "../../domain/canonical-timestamp.ts"; import { contentHashMatches } from "../../domain/content-hash.ts"; import type { SessionRootResolver } from "../../domain/session-lineage.ts"; -import { isSessionIdentity } from "../../domain/session-identity.ts"; +import { isSessionIdentity, sameSessionIdentity } from "../../domain/session-identity.ts"; import type { ContentOrigin, OriginConfidence, SessionIdentity } from "../../domain/session.ts"; import { splitUnicodeWhitespaceTerms } from "../../domain/unicode-whitespace.ts"; import { readSqliteCaptureScope } from "./sqlite-capture-scope.ts"; @@ -98,7 +98,7 @@ function listSessions(database: DatabaseSync, query: SessionListQuery): SessionL if (summary === undefined || resolveRoot === undefined) { throw new SqliteSessionIndexError("corrupt-data"); } - if (!sameIdentity(summary.identity, identity)) + if (!sameSessionIdentity(summary.identity, identity)) throw new SqliteSessionIndexError("corrupt-data"); return Object.freeze({ ...summary, root: resolveRoot(identity) }); }); @@ -696,7 +696,7 @@ function searchHit( native_id: row.native_id, }); const summary = summaries.get(sessionId); - if (summary === undefined || !sameIdentity(summary.identity, identity)) { + if (summary === undefined || !sameSessionIdentity(summary.identity, identity)) { throw new SqliteSessionIndexError("corrupt-data"); } const entry = entryAt({ @@ -759,7 +759,7 @@ function readSelectedSummaries( for (const coordinate of coordinates) { const previous = requests.get(coordinate.sessionId); if (previous !== undefined) { - if (!sameIdentity(previous.identity, coordinate.identity)) { + if (!sameSessionIdentity(previous.identity, coordinate.identity)) { throw new SqliteSessionIndexError("corrupt-data"); } continue; @@ -876,14 +876,6 @@ function identityAt(row: IdentityRow): SessionIdentity { return identity; } -function sameIdentity(left: SessionIdentity, right: SessionIdentity): boolean { - return ( - left.source.kind === right.source.kind && - left.source.instanceId === right.source.instanceId && - left.nativeId === right.nativeId - ); -} - function emptySearchPage(captureScope: SessionSearchPage["captureScope"]): SessionSearchPage { return Object.freeze({ hits: Object.freeze([]), From d3ea7a36d2fa7e1a43b270b701f96fa688bdc500 Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 12:00:00 -0700 Subject: [PATCH 4/5] docs: retire completed query plan --- dev/plans/260723-everyday-query-hot-paths.md | 295 ------------------- dev/plans/README.md | 10 +- 2 files changed, 3 insertions(+), 302 deletions(-) delete mode 100644 dev/plans/260723-everyday-query-hot-paths.md diff --git a/dev/plans/260723-everyday-query-hot-paths.md b/dev/plans/260723-everyday-query-hot-paths.md deleted file mode 100644 index 886326f..0000000 --- a/dev/plans/260723-everyday-query-hot-paths.md +++ /dev/null @@ -1,295 +0,0 @@ -# Optimize everyday query hot paths without changing evidence semantics - -## Goal - -Make routine `list`, `entries`, and `search` reads scale with the requested page -wherever their contracts permit it. Query execution should select the smallest -deterministically ordered page of compact coordinates, batch-hydrate and verify -only that page, and calculate exact query-wide evidence once when support or -lineage requires it. - -The current broad entry selector is the first priority. On an aggregate-only -live library with 4,544 sessions and 3.57 million entries, a 50-entry page took -roughly 4.5–6 seconds warm because SQLite scanned `sessions_entries` first and -built a temporary ordering B-tree before applying `LIMIT`. An equivalent -source→tracking→canonical→entry traversal used existing indexes, required no -temporary sort, and selected 51 coordinates in about 0.1 milliseconds warm. -The existing generated measure uses only five entries per session and does not -expose this skew. - -Search is the second priority. It currently scans overlapping FTS results for -aggregate support, distinct matching identities, and ranking, then performs -point reads for selected snippets, contexts, and summaries. Exact support and -ranking inspect the complete qualifying result by design, so broad search -cannot become page-only work; the target is one query-wide match computation -plus bounded page hydration, not approximate evidence. - -Every delivery slice below is independently reviewable and lands in order. -Acceptance requires deep-equal repository results, byte-identical structured -output except for the intentionally versioned opaque cursor token in slice 7, -unchanged cursor field presence and continuation results, unchanged capture -scope, support, ranking, roots, corruption handling, snapshot behavior, and -cursor failures, plus deterministic work-shape or repeatable aggregate -measurements. Machine time is never the sole correctness gate. - -## Changes - -### 1. Bound entry coordinate selection - -1. `src/infrastructure/sqlite/sqlite-session-entry-query.ts:readEntryRows` — - make source→tracking→canonical→entry traversal explicit with order-preserving - `CROSS JOIN` boundaries and the existing equality predicates. Preserve every - `entryInventoryWhere` filter, the filtered `first|last` subquery from - `selectionClause`, binary source/instance/native ordering, ascending entry - ordinal, `LIMIT + 1`, and cursor behavior. Keep hydration after page slicing - and use the existing unique and primary-key indexes; this slice adds no - migration or durable index. -2. `test/contracts/session-query.contract.ts` — traverse every entry page in - `all`, `first`, and `last` modes and compare the concatenated result with the - one-page canonical result. Cover source, instance, native ID, source state, - workspace, activity, entry time, actor, origin, kind, and tool filters; - binary-order identity edges; roots; content counts and previews; capture - scope; cursor mismatch; and cursor staleness. -3. `test/infrastructure/sqlite-session-entry-query.test.ts` — isolate plan-shape - risks that the shared contract does not: selected versus unselected corrupt - rows, each filter family under the explicit traversal, and `first|last` - applying entry filters inside the per-session selection rather than after - ordinal selection. -4. `scripts/measure-entry-query.ts` — replace or supplement the uniform profile - with an owned on-disk corpus containing thousands of sessions, hundreds of - thousands of entries, and at least one session with tens of thousands of - entries. Use mostly textless entries plus bounded shared text so coordinate - work dominates. Measure broad and filtered `all`, `first`, `last`, origin, - tool, early-page, and deep-page cases after a warm-up; report seeding, - coordinate selection, full hydration, aggregate counts, and repeated - semantic equality separately. -5. Add a private, best-effort timing/work observer around `readEntryRows` and - `hydrateEntries`, accepted only by the infrastructure entry-query function - and composed only by the generated measure. Observer failure must not change - query behavior or public output. Factor the exact production coordinate SQL - into one focused internal builder so the measure can run - `EXPLAIN QUERY PLAN` against the statement that ships rather than a copied - query. -6. The measure records normalized plan facts: outer scan, indexes used, and - temporary ordering B-tree presence. These are diagnostics, not Vitest - assertions tied to one SQLite version. Promote the SQL change only when the - generated result is identical and the broad plan avoids the entry-first full - scan and full temporary sort on supported development runtimes. -7. Update `docs/contributing/entries.md` and - `docs/contributing/testing.md` with the bounded-coordinate design, generated - skew profile, structural proof, and report-only elapsed evidence. - -### 2. Measure planner statistics as a separate complement - -1. Add `scripts/measure-query-planner-statistics.ts` and an opt-in package - command. Create equal generated databases from the skewed corpus, apply - explicit `ANALYZE` and `PRAGMA optimize` variants only to experimental - clones, and compare plans, alternating warm elapsed times, statistics rows, - and database byte growth. -2. Cover broad and narrow entries after the explicit traversal, list identity - and activity filters, broad and selective `all|any` search, and manifest - selection. Record whether any benefit remains beyond the local query-shape - fix and how often statistics would need refresh as the corpus changes. -3. Do not invoke planner-statistics commands from readers, migrations, index, - doctor, or maintenance in this slice. If no repeatable benefit remains, if - selective queries regress, or if refresh/storage cost is material, close the - experiment without a runtime policy. If evidence justifies persistence, - create a separate plan covering certified writer mutation, receipt sequence, - recovery, refresh cadence, schema compatibility, and index-time cost. - -### 3. Batch selected search content and snippets - -1. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchContent` — - replace one execution per selected content ID with one page-bounded selected - content CTE. Deduplicate IDs, retain the explicit integer FTS rowid - restriction, and require exactly one canonical text, digest, and snippet per - requested ID. Preserve full-text hash validation and marker-collision - handling; if any selected text contains the candidate markers, retry the - complete selected page with the next deterministic candidate. Unselected - content must not influence marker selection. -2. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — - build the content map once and make snippet projection free of content point - reads. Leave context, summary/root work, and `readMatchedTerms` unchanged in - this slice; `any` mode is already bounded by selected coordinates and at most - 32 admitted terms. -3. Extend `test/infrastructure/sqlite-session-query.test.ts` with limits 1, 20, - and maximum; shared/repeated content IDs; marker collisions; multibyte text; - and empty bodies. Missing or duplicate selected content, FTS disagreement, - malformed text, and digest mismatch must still fail the whole read, while an - unselected corrupt row stays unhydrated. -4. Extend `scripts/measure-search-query.ts` with page-size profiles, - selected-content counts, statement counts, and alternating warm elapsed - time. Update `docs/contributing/search.md` and - `docs/contributing/architecture.md` to claim only batched selected-page - content/snippet hydration. - -### 4. Batch selected search context - -1. `src/infrastructure/sqlite/sqlite-query-context.ts` — replace per-hit - `readSearchContext` calls with a page operation keyed by each selected - `(sessionId, primaryOrdinal)`. One query reads direct tool-call/result - coordinates for all primaries and retains enough ordered candidates to prove - `linkedContextTruncated`. -2. Keep a separate per-primary membership map containing each selected physical - coordinate and its adjacent/linked flags. Deduplicate SQL hydration by - physical `(sessionId,entryOrdinal)`, then read those unique coordinates and - ordered text segments in fixed chunks below SQLite's minimum supported - bind-variable limit. Merge chunk results into the per-primary map - deterministically, preserving direct-link-only expansion, ordinal order, - linked caps, flags, and 512-byte UTF-8 body truncation. Query work is one - primary-coordinate query plus - `ceil(unique physical context coordinates / chunk limit)` hydration queries, - not an infeasible fixed two-query claim. -3. `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — build - the context map once and remove per-hit context point reads, leaving the - content map from slice 3 and separately owned summary/root work unchanged. -4. Extend `test/infrastructure/sqlite-session-query.test.ts` with context 0 and - maximum at the maximum result limit; enough adjacent/linked coordinates to - cross several chunks; links in both directions; linked overflow; - adjacent/linked overlap; multibyte truncation; empty bodies; multiple hits in - one session; and corruption/failure in a later chunk. Require exact output - equality and all-or-nothing failure. -5. Extend the search measure with selected-context counts, expected/actual chunk - counts, and alternating warm elapsed time. Document chunked selected-page - context hydration without claiming query-wide bounded search work. - -### 5. Prototype one query-wide search match set - -1. In an isolated change, prototype one - `WITH matching_segments AS MATERIALIZED` statement derived from the current - literal FTS query, joins, and exact filters. It emits tagged result sections - for aggregate occurrences/distinct content, distinct matching identities - needed for root support, and the ranked `LIMIT + 1` coordinate page using the - current BM25, activity, binary identity, and ordinal order. It creates no - durable or transcript-bearing table and never derives query-wide support from - the visible page. -2. Factor only shared joins and filter parameters needed to compare baseline and - prototype; do not add a general SQL builder. Root resolution continues over - the complete matching identity set and page hydration uses slice 3. -3. Add a dedicated measure or extend `measure:search-query` to alternate - baseline and prototype over selective/broad terms, `all|any`, repeated - occurrences with low unique content, many identities and lineage states, - limits 1/20/maximum, and first/deep pages. Report qualifying FTS traversals, - match-set time, hydration time, total time, and peak RSS. -4. Run both strategies against the same generated fixtures in - `test/infrastructure/sqlite-session-query.test.ts` and - `test/contracts/session-query.contract.ts`; require deep equality of hits, - support, roots, snippets, matched terms, context, capture scope, and cursors. -5. Define “one qualifying FTS traversal” structurally as one normalized - `EXPLAIN QUERY PLAN` virtual-table `MATCH` scan and one execution of the - match-set statement; Node's SQLite API does not expose a reliable FTS row - traversal counter. Promote only if the shape is legal for FTS5/BM25, meets - that structural proof, repeatedly improves broad search by roughly 20% or - more, keeps selective regression within 10%, and keeps peak RSS growth within - 25% in the alternating generated measure. These are prototype decision - thresholds, not CI budgets. If the gate fails, remove the production - prototype and retain the exact multi-scan path plus the independently useful - page batching. - -### 6. Batch selected summaries - -1. `src/infrastructure/sqlite/sqlite-session-state.ts` — add a bounded - `readSessionSummariesBatch` keyed by canonical session ID and expected - identity. Bind only `(sessionId,kind,instanceId,nativeId)` for each distinct - selected session: at the 200-row public maximum this is 800 variables, below - SQLite's minimum supported limit. Join canonical, tracking, and source rows - once, validate one result per requested session, and restore page order in - JavaScript from the validated session-ID map rather than binding an input - ordinal. Reuse the existing freshness and retained summary decoders; reject - missing, duplicate, malformed, or identity-inconsistent rows as - `corrupt-data`. Refactor shared decoding so point and batch reads cannot - drift. -2. `src/infrastructure/sqlite/sqlite-session-query.ts:listSessions` — carry the - compact session ID through the ordered page and batch selected summaries - after slicing. Do not enlarge the pre-limit sort with summary payload. -3. `src/infrastructure/sqlite/sqlite-session-entry-query.ts:hydrateEntries` and - `src/infrastructure/sqlite/sqlite-session-query.ts:hydrateSearchHits` — - deduplicate selected session IDs and replace per-session summary cache misses - with one bounded batch. -4. Add corruption-parity tests and SQLite authorization/work-shape assertions - comparing page size 1 with maximum. The number of summary SELECT - authorizations must remain constant per page rather than grow with distinct - selected sessions. Extend entry/search measures with distinct-session - profiles. - -### 7. Add compatible keyset continuation for list and entries - -1. `src/infrastructure/sqlite/query-cursor.ts` — add cursor v2 while retaining - command, complete query fingerprint, random library instance, writer - generation, and cumulative next offset. Add bounded provider-neutral numeric - anchors: canonical `session_id` for list and - `(session_id,entry_ordinal)` for entries. Do not place raw source instance or - native IDs in the opaque base64url token. Preserve canonical encoding, the - 2,048-byte bound, allowlisted payload shape, usage-versus-stale failures, and - opacity. -2. Decode existing v1 cursors for all commands. New list/entries continuations - emit v2; search continues to emit and consume the existing offset-only v1 - format. A v1 list/entries page uses the current offset path once, then emits - v2 from its last returned row. New v2 list/entry pages use the numeric anchor - while retaining cumulative offset for the existing internal contract. -3. `src/infrastructure/sqlite/sqlite-session-entry-query.ts` — add a strict - anchor-resolution CTE that requires the numeric coordinate to exist and - qualify under the same entry filter and `first|last` selection, recovers its - binary source/instance/native order fields, and applies the lexicographic - after-anchor predicate. `src/infrastructure/sqlite/sqlite-session-query.ts` - does the same for list: coordinate selection carries `canonical.session_id` - and validated effective activity, and anchor resolution recovers null-last - activity plus the binary identity ties inside the same immutable snapshot. - Revision mismatch is `stale-cursor`; a structurally malformed v2 payload or - matching-revision anchor that is absent/non-qualifying is `invalid-cursor`; - an existing anchor row with malformed stored fields is `corrupt-data`. Emit - an anchor only when the extra row proves another page. -4. Extend `test/infrastructure/sqlite-query-primitives.test.ts` for v1/v2 - admission and failure cases. Extend the shared query contract to traverse - page size one across equal and null activity, binary Unicode identities, - every entry selection/filter family, a v1-to-v2 transition, recreated - libraries, and later writer generations. Concatenated keyset traversal must - equal canonical one-page results with no gaps or duplicates. -5. Extend generated measures with early and deep pages. The structural gate is - production v2 SQL with no `OFFSET`, an anchor-resolution CTE, a strict - after-anchor predicate, and normalized `EXPLAIN QUERY PLAN` evidence of - indexed continuation; deep-page elapsed time is supplemental. Update - architecture, entry, and search docs; the public syntax, cursor opacity, - mismatch rules, and stale rules remain unchanged. - -### 8. Measure and, only then, lazy-load CLI composition - -1. After slices 1, 3, 4, 6, and 7 land and the search prototype is closed, add - `scripts/measure-cli-startup.ts`. Against compiled `dist` and generated - temporary state, alternate bare Node, version, top-level help, command help, - and one provider-free read; report median/p95 aggregates without touching - contributor providers or the ordinary library. -2. If the compiled evidence is material, refactor `src/bin/sessions.ts` to use - type-only imports plus memoized dynamic imports inside the command callbacks - that need concrete providers, lifecycle, maintenance, or application - services. Keep provider kinds available for grammar/help and preserve the - composition-root rule that only `src/bin/` selects concrete adapters. -3. `test/cli.test.ts`, `pnpm smoke:dist`, and `pnpm smoke:package` must preserve - exact grammar, help, output, errors, signals, provider laziness, and packaged - module resolution. If callback-level imports do not materially reduce - packaged startup, stop without splitting `program.ts` or adding a custom - dispatcher. - -## Verify - -- Run the focused shared query contract, SQLite entry/search, cursor primitive, - CLI, and applicable measurement-contract tests after their owning slice. -- Run `pnpm measure:entry-query` and `pnpm measure:search-query`; run experimental - planner, search-materialization, and startup measures only in their slices. -- Run `pnpm typecheck`, `pnpm deps:check`, and `pnpm check` before publishing - each independently reviewable slice. - -## Boundaries - -- Do not change human, JSON, or JSONL fields, order, limits, byte bounds, exit - codes, or fail-before-output behavior. -- Do not approximate support, rank, matched terms, snippets, context, capture - scope, or lineage, and do not hydrate transcript text before page selection. -- Do not add a daemon, cross-process cache, package API, batch command, network, - telemetry, or provider-specific query behavior. -- Do not add durable indexes or migrations in the primary query slices. - Planner statistics remain experimental until separately justified. -- Do not fold full-document reads, manifest copying, doctor, or lineage-closure - work into this program. -- Do not place contributor paths, identities, transcripts, or private corpus - details in plans, fixtures, docs, or measurement output. diff --git a/dev/plans/README.md b/dev/plans/README.md index 08c9071..b9c085f 100644 --- a/dev/plans/README.md +++ b/dev/plans/README.md @@ -16,16 +16,12 @@ Each program is ordered internally. Promote one numbered delivery slice at a time into its own executor plan and independently reviewable change; evidence-gated slices receive an executor plan only after their gate passes. -1. [Everyday query hot paths](260723-everyday-query-hot-paths.md) — fix broad - entry selection first, then reduce repeated search hydration and summary - work, add compatible keyset continuation, and measure later planner/startup - candidates. -2. [Indexing hot paths](260723-indexing-hot-paths.md) — measure discovery and +1. [Indexing hot paths](260723-indexing-hot-paths.md) — measure discovery and replacement work, remove avoidable provider lookup/serialization and statement-per-content costs, then gate deeper transaction or WAL changes. -3. [Doctor and maintenance hot paths](260723-doctor-maintenance-hot-paths.md) — +2. [Doctor and maintenance hot paths](260723-doctor-maintenance-hot-paths.md) — make exact FTS verification memory-bounded, partition oversized terms, page only orphan candidates, and gate any compact-proof optimization separately. -4. [Verified bounded session reads](260723-verified-bounded-session-reads.md) — +3. [Verified bounded session reads](260723-verified-bounded-session-reads.md) — stream complete validation and the existing public-document digest while retaining only the requested bounded `show` or `export` selection. From 072deae790bb30f317072eaf82b78461581d7eb3 Mon Sep 17 00:00:00 2001 From: ferueda Date: Thu, 23 Jul 2026 12:11:46 -0700 Subject: [PATCH 5/5] docs: track cursor format retirement --- docs/architecture-memo.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/architecture-memo.md b/docs/architecture-memo.md index bece7ef..7bac76b 100644 --- a/docs/architecture-memo.md +++ b/docs/architecture-memo.md @@ -1356,6 +1356,17 @@ facets, deterministic comparison/timelines, tokenizer phrase search, smaller search/entries titles, exact locator interning, and Markdown presentation remain unsequenced until evidence gives them priority. +Cursor format convergence is also deferred. List and entries currently emit +anchored v2 continuations while accepting v1 offset continuations once; search +still emits v1. Promote a retirement plan only after a measured, exact search +keyset design justifies v2 search, every command has stopped emitting v1, and a +declared compatibility boundary permits removal. That plan must first preserve +v1 decoding during the emission transition, prove complete v2 traversal and +existing stale/query-mismatch failures, then remove the v1 decoder and legacy +offset paths together. Opaque cursors remain continuation tokens rather than +durable bookmarks, but their removal is still an explicit compatibility +decision. + Semantic search, an external plugin ABI, cloud/team indexes, native binaries, Homebrew, TUI/watch mode, orchestration integrations, opt-in automated changes, raw provider backup, destination-provider delivery, and automatic project edits