diff --git a/dev/plans/260723-doctor-maintenance-hot-paths.md b/dev/plans/260723-doctor-maintenance-hot-paths.md index 61f627f..95ec953 100644 --- a/dev/plans/260723-doctor-maintenance-hot-paths.md +++ b/dev/plans/260723-doctor-maintenance-hot-paths.md @@ -1,23 +1,27 @@ -# Bound doctor and maintenance work without weakening integrity +# Measure doctor and bound orphan maintenance without weakening integrity ## Goal -Make `doctor` and routine maintenance use bounded transient work while keeping -Sessions' exact, fail-closed evidence contracts. +Keep Sessions' exact, fail-closed evidence contracts while measuring doctor and +bounding routine orphan maintenance. -The first priority is doctor FTS verification. Today +The doctor feasibility slice is complete. Today `ftsProjectionSemanticContentIsValidReadOnly` builds a second complete FTS5 projection from every retained canonical content value in memory, then compares the complete expected and actual vocabularies. Its term ranges normally target 1,000,000 instances, but one term above that target is still compared as one unbounded range. On a large retained library this duplicates a corpus-sized search structure before useful progress is visible and is the most likely cause -of reports that doctor appears to hang. +of reports that doctor appears to hang. Human review accepted the measurement's +rejection of document-ID-bounded actual-vocabulary scans. Production doctor is +unchanged; successor investigation moved to +[the single-pass FTS feasibility plan](260723-doctor-single-pass-fts-feasibility.md). -The second priority is orphan repair. It currently pages all content rows, -tests reachability row by row, and executes a fenced zero-deletion batch for -windows containing only referenced content. The desired query should page only -orphan candidates while preserving the transaction-time reachability recheck. +The remaining executable scope is orphan repair. It currently pages all content +rows, tests reachability row by row, and executes a fenced zero-deletion batch +for windows containing only referenced content. The desired query should page +only orphan candidates while preserving the transaction-time reachability +recheck. Carrying certified cleanliness through a successful compact is outside this plan. Compaction deliberately invalidates the previous clean proof today. Any @@ -27,9 +31,9 @@ separately accepted measurement plan and ADR. Doctor remains an immutable, exact whole-library audit. Repair remains uncertified maintenance. The program changes no provider behavior, query result, public command grammar, structured output, exit code, or retention -policy. The feasibility measure, bounded doctor refactor, and orphan paging land -as separate PRs. The feasibility result requires explicit acceptance before the -bounded doctor refactor is authorized. +policy. The feasibility measure and orphan paging land as separate PRs. The +bounded doctor refactor in this program is retired; orphan paging remains +independently executable. ## Changes @@ -76,121 +80,29 @@ bounded doctor refactor is authorized. persistent state, interval accounting, and complete cleanup are hard assertions. -### 2. Verify expected FTS content in bounded exact intervals +### Slice 1 result -1. Do not start this section until the slice-1 feasibility result is explicitly - accepted. If it is rejected, preserve the current exact audit and replace - this design through a new or materially revised plan. -2. Add a private best-effort work observer in - `src/infrastructure/sqlite/fts-projection.ts`, composed from - `src/infrastructure/sqlite/sqlite-index-health.ts:inspectDatabaseHealth`. - Keep it out of the application health port and public doctor result. - Allowlisted aggregate fields are: - - canonical rows and UTF-8 bytes inspected; - - expected interval count and maximum interval rows/bytes; - - oversized-content interval count; - - term summaries and ordinary term ranges compared; - - oversized terms and coordinate ranges compared; - - actual/expected instances compared and maximum range size. -3. Isolate and swallow observer callback failures. They must not affect semantic - health, cleanup, stdout, stderr, or the exit code, and the observer must never - receive text, terms, content IDs, hashes, identities, paths, timestamps, SQL - text/errors, or lease values. -4. Refactor - `src/infrastructure/sqlite/fts-projection.ts:ftsProjectionSemanticContentIsValidReadOnly` - so it never retains one corpus-sized expected projection or a corpus-sized - interval list. Keep `PRAGMA temp_store = MEMORY` and fail closed unless - SQLite confirms memory-only TEMP storage. -5. Replace `loadExpectedDoctorProjectionInsideSavepoint` with interval-local - helpers such as `scanDoctorContentIntervals`, - `loadDoctorExpectedInterval`, and `dropDoctorExpectedProjection`. - Construct intervals by keyset-reading `content_id` and exact - `length(CAST(text AS BLOB))`, then load the corresponding canonical - `content_id,text` rows in signed-ID order. -6. Use fixed private admission targets of at most 512 canonical rows and exactly - `16 * 1024 * 1024` UTF-8 bytes per expected interval. A single canonical - value above the byte target is processed alone, counted as oversized, and - never truncated, skipped, split into invented canonical rows, or spilled to - disk. -7. For each interval: - - create a fresh contentless TEMP FTS5 table plus only the vocab tables - needed for that comparison; - - preserve the original signed row IDs; - - load inside a private savepoint that composes with an existing outer - transaction; - - compare docsize in both directions; - - compare exact term/document/column/offset instances in both directions; - - close iterators, roll back or release the savepoint, and drop every TEMP - object before advancing. -8. Cover actual document IDs exactly with half-open signed-ID intervals. The - first interval includes the actual prefix through its last canonical ID; - later intervals cover `(previousLastId,lastCanonicalId]`; after the final - interval, the actual docsize and vocabulary tail must be empty. This detects - extra actual documents before the first canonical row, between sparse - canonical IDs, and after the last row. An empty canonical table requires an - empty actual projection. -9. Derive each interval's actual and expected term summaries from the - `instance` vocabulary restricted by that interval's document bounds, grouped - by term with exact distinct-document and instance counts. The `doc` column of - the current `fts5vocab(...,'row')` table is a document count, not a document - ID; never filter or reuse whole-library row-vocabulary summaries as if it - were an interval coordinate. -10. The feasibility gate in slice 1 must already have shown that this exact - document-bounded access shape does not multiply whole-vocabulary work. - Bounded memory alone is not sufficient justification for accidental - superlinear CPU. -11. Preserve primary-error precedence. Load, comparison, iterator, or cleanup - failure makes semantic health false; cleanup noise must not replace the - original failure. Repeated calls and calls inside the current immutable - snapshot must leave no TEMP or persistent artifact. +The generated production-writer measurement is recorded in +[`docs/research/doctor-document-interval-feasibility.md`](../../docs/research/doctor-document-interval-feasibility.md). +Document-bounded actual-vocabulary queries retained the same virtual-table plan +shape as unbounded queries. On the large cohort, the proposed 512-row/16-MiB +admission rule produced 41 intervals, made total work 3.80 times slower, and +made grouped term-summary work 6.82 times slower. The actual-vocabulary probe's +peak RSS ratio was 0.96; it deliberately does not measure the expected-side +projection whose memory the full design intended to bound. -### 3. Stream term ranges and partition an oversized term +Human review accepted the recommendation on 2026-07-23. Sections 2–3 are retired +and must not be implemented. Production doctor behavior remains unchanged; +successor investigation is owned by +[the single-pass FTS feasibility plan](260723-doctor-single-pass-fts-feasibility.md). -1. Replace - `src/infrastructure/sqlite/fts-projection.ts:matchingDoctorTermRanges` and - the retained `DoctorTermRange[]` with immediate, streaming comparisons. - Walk actual and expected term summaries in UTF-8 binary order. Require exact - term, document-count, and instance-count equality before comparing - positions. -2. Accumulate complete ordinary terms only until either side reaches the - existing 1,000,000-instance target, compare that range in both directions, - then discard its bounds. Never sample terms, documents, or positions and - never replace exact instance equality with aggregate counts or hashes. -3. When one term exceeds the target, partition its exact ordered coordinate - space `(doc,col,offset)`. Build each half-open upper boundary with bounded - lookahead from both actual and expected vocabularies and choose the earlier - target-th coordinate, so neither side can place more than the target in one - comparison. Compare actual-only and expected-only rows before advancing. -4. Cover the first coordinate, signed 64-bit document IDs, boundaries that fall - on different sides, and the final tail. Equal total counts are not proof: - shifted documents, columns, or offsets must still fail. -5. Extend `test/infrastructure/sqlite-fts-projection.test.ts` with: - - healthy equality across row/byte intervals, sparse IDs, zero-token rows, - multibyte terms, and one oversized canonical value; - - missing, extra, and malformed docsize in the first, middle, and final - interval; - - extra actual documents before, between, and after canonical IDs; - - wrong terms with equal token counts, changed positions, cross-document - swaps, and unequal counts under the same term universe; - - a healthy oversized term spanning several coordinate ranges; - - actual-only, expected-only, shifted-document, and shifted-offset damage on - both sides of boundaries; - - an adversarial distribution that would exceed the target if boundaries - came from only one side; - - invalid private limits, load/comparison/observer/drop failures, outer - transaction composition, and repeated invocation. -6. Extend `test/infrastructure/sqlite-index-health.test.ts` to require the same - public `ReadyIndexHealth` decisions for healthy, docsize-damaged, and - semantic-damaged libraries. Extend `test/doctor-no-persistence.test.ts` with - a worker stopped after its first interval; the parent must see unchanged - main-file identity/stat, no WAL/SHM, and no persistent TEMP artifact. -7. Reconcile the current-behavior descriptions in - `docs/reference/cli-contract.md`, `docs/privacy.md`, - `docs/contributing/storage.md`, `docs/contributing/architecture.md`, - `docs/contributing/testing.md`, and `docs/architecture-memo.md`. Describe - exact interval coverage, the one-oversized-content exception, coordinate - partitioning, and the memory-only TEMP rule. Do not make the private work - observer part of the public CLI contract. +### 2–3. Retired: document-ID interval verification + +The accepted slice-1 decision rejects this design because document-bounded +actual-vocabulary scans multiply dominant work by interval count. The dependent +term-range proposal is not authorized independently in this program. Preserve +the current exact audit. Any reusable technique must be re-established through +the successor plan rather than copied from these retired sections. ### 4. Page only orphan repair candidates @@ -287,12 +199,10 @@ ordinary retained library. ## Verify -- For the feasibility delivery, run the script-contract tests and - `pnpm measure:doctor`; require exact generated query equality, immutable - persistent state, and cleanup, then stop for explicit acceptance. -- For bounded doctor after that acceptance, run the focused FTS projection, - index health, no-persistence, immutable snapshot, and measurement-contract - tests, then `pnpm measure:doctor`. +- The completed feasibility delivery requires the script-contract tests and + `pnpm measure:doctor`, with exact generated query equality, immutable + persistent state, cleanup, and the accepted decision recorded in the + feasibility report. - For orphan paging, run the repair-orphans, writer-coordination, FTS repair, clean-proof, and lifecycle tests. - Require exact semantic/corruption parity, bounded work counters, @@ -313,6 +223,5 @@ ordinary retained library. - Do not certify repair, forget, clear, failed compact, or crashed compact. - Do not claim secure erasure, partial-page repacking, or file shrink beyond observed whole-page reclamation. -- Stop and redesign bounded doctor if interval cleanup cannot remain - memory-only, exact coverage weakens, or actual-vocabulary work becomes - superlinear in interval count. +- Do not revive sections 2–3 without new measured evidence and an explicitly + accepted successor design. diff --git a/dev/plans/260723-doctor-single-pass-fts-feasibility.md b/dev/plans/260723-doctor-single-pass-fts-feasibility.md new file mode 100644 index 0000000..d67c48d --- /dev/null +++ b/dev/plans/260723-doctor-single-pass-fts-feasibility.md @@ -0,0 +1,139 @@ +# Prove a single-pass doctor FTS comparison before changing production + +## Goal + +Determine whether exact doctor FTS instance comparison can use one monotonic +actual stream and one monotonic expected stream instead of retained term ranges +plus repeated bidirectional `EXCEPT` queries. + +The accepted +[document-interval feasibility result](../../docs/research/doctor-document-interval-feasibility.md) +rejected document-ID-bounded actual-vocabulary scans because 41 +production-shaped intervals made dominant work 3.80 times slower. The current +production path in +`src/infrastructure/sqlite/fts-projection.ts:ftsProjectionSemanticContentIsValidReadOnly` +still builds one complete, corpus-sized, memory-only expected FTS projection, +streams actual and expected row vocabularies into a retained +`DoctorTermRange[]`, then runs two instance `EXCEPT` queries per range. + +This plan is an evidence gate, not a production executor plan. Its candidate +keeps the complete expected projection and existing exact docsize proof, so it +may reduce comparison CPU and make comparison-side state constant but cannot +claim bounded total memory. Production doctor remains unchanged until generated +evidence passes, the reliance on FTS5 vocabulary order is explicitly accepted, +and a separate production plan is approved. + +## Changes + +1. Extend `scripts/measure-doctor.ts` with a separate single-pass controller + mode, exposed as `pnpm measure:doctor:single-pass` in `package.json`. Reuse + the existing generated production-writer corpora, owned temporary root, + controller-issued worker authority, immutable clones, fresh child processes, + alternating strategy order, aggregate-only report, and success/failure + cleanup. Add full-only generated cohorts for a mixed ordinary-term index + whose current algorithm reports at least three production-sized term ranges + and for one repeated term above the 1,000,000-instance target. Keep the + existing `pnpm measure:doctor` arguments, corpora, and exact report unchanged. +2. Put candidate query and comparison code in a controller-independent + `scripts/doctor-single-pass-probe.ts` whose effects are limited to + connection-local memory-only TEMP state with complete cleanup. The controller + and deterministic tests must use this one measurement-only implementation. + Compare byte-identical clones with three strategies: + - the current production + `ftsProjectionSemanticContentIsValidReadOnly` result and total elapsed + time; + - a term-ordered direct-stream candidate; and + - a fully ordered + `ORDER BY term COLLATE BINARY, doc, col COLLATE BINARY, offset` control + that reveals the cost and TEMP plan of portable explicit ordering. + The candidate and control must build the same complete expected TEMP FTS and + perform the same bidirectional docsize proof before instance comparison. +3. For the candidate, prepare one actual and one expected + `fts5vocab(..., 'instance')` iterator ordered only by + `term COLLATE BINARY`. Decode SQLite integers as `bigint`. Validate every + coordinate as a non-empty term, signed 64-bit document ID, exact `text` + column, and non-negative signed-64-bit offset. Require each stream to be + strictly increasing by UTF-8 binary term, signed document ID, binary column, + then offset. Compare coordinates one-for-one and require both iterators to + reach EOF together. A malformed, duplicate, non-monotonic, unequal, missing, + or extra row fails closed. +4. Retain only the current and previous coordinate from each stream. Close both + iterators through the existing + `src/infrastructure/sqlite/iterator-cleanup.ts:closeSqliteIterators` + error-precedence pattern before dropping TEMP objects. Keep + `PRAGMA temp_store = MEMORY`, require the read-back value `2`, preserve the + private savepoint and any outer transaction, and make load, iteration, + comparison, rollback, release, or drop failure unhealthy. No comparison + term, range, document, position, count map, hash, or array may grow with the + corpus. +5. Capture normalized `EXPLAIN QUERY PLAN` facts and exact operation counters + for all strategies. Candidate admission requires exactly one actual and one + expected instance-vocabulary scan, no `EXCEPT`, no grouped summary query, no + TEMP B-tree, exact visited-row counts, and constant two-row lookahead. + Explicit full ordering is a control, not an acceptable fallback if it + materializes corpus-sized sort state. The real production baseline reports + only total elapsed time; do not invent unavailable baseline phases or modify + `src/` to expose them. Candidate and control phases are expected-index load, + docsize, instance comparison, cleanup, and total. Peak RSS remains supporting + evidence rather than a threshold. +6. Add `test/doctor-single-pass-measurement.test.ts` with two distinct proofs: + - import the measurement-only probe and require the same healthy/unhealthy + decisions as the current complete-vocabulary oracle for empty projections, + zero-token rows, sparse signed IDs including integer extrema, multibyte and + canonically distinct Unicode, missing or extra first/middle/final + coordinates, equal-count wrong terms, shifted documents/columns/offsets, + cross-document swaps, and docsize damage. Keep the routine contract corpus + bounded and assert constant lookahead with a reduced repeated-term target; + add a fixed-seed small-corpus matrix whose healthy and damaged cases must + agree with the oracle; and + - spawn the reduced controller contract and require recursive output + allowlisting, exact aggregate derivation, fixed query/row counters, + private modes, unchanged database bytes and metadata, absent WAL/SHM/ + journal artifacts, silent private-worker failure, and complete cleanup. + Exercise different healthy FTS segment topologies on equal content so the + monotonicity guard is not proved by one insertion shape. +7. Run the reduced ordering contract on the minimum supported Node runtime and + the current release runtime. Record Node and SQLite versions. The + [public `fts5vocab` contract](https://www.sqlite.org/fts5.html#the_fts5vocab_virtual_table_module) + exposes instance coordinates but does not promise the complete native tuple + order used by the candidate. A future runtime that changes that order must + fail a healthy library closed through the monotonicity guard; a production + plan may accept that compatibility dependency only through an explicit human + decision. +8. Record the result in + `docs/research/doctor-single-pass-fts-feasibility.md`. Require exact + correctness, immutability, privacy, cleanup, and structural work assertions + before interpreting time. The opt-in full measure must cover the existing + large high-unique-term cohort, the mixed corpus with at least three observed + baseline ranges, and the dominant repeated term crossing the current + 1,000,000-instance target. Run at least two complete invocations and report + median and p95 total ratios per cohort plus candidate phase scaling. A + candidate slower than the real production baseline in any named large cohort + is rejected. Non-regressive but noise-sized or inconsistent gains are + inconclusive, not acceptance. There is no automatic elapsed-time pass: + explicit human review decides whether repeated gains are meaningful and + whether the ordering dependency is acceptable before a separate production + executor plan may be created. + +## Verify + +- `pnpm test test/doctor-single-pass-measurement.test.ts` +- `pnpm measure:doctor:single-pass` +- `pnpm check` + +## Boundaries + +- Do not change `src/` production doctor behavior, public health/CLI contracts, + output, exit codes, provider reads, or persistent library state in this + feasibility slice. +- Do not describe direct instance streaming as bounded total memory. The full + expected TEMP FTS remains corpus-sized. +- Do not revive document-ID vocabulary scans, spill transcript-derived terms or + coordinates to disk, reimplement `unicode61`, load a native extension, open + the library writable for FTS5 `integrity-check`, sample evidence, or replace + exact coordinates with counts or hashes. +- Stop if deterministic ordering needs a TEMP B-tree, either vocabulary is + scanned more than once, comparison-side retained state grows with terms or + instances, any corruption decision differs, a hard immutability/privacy + assertion fails, or the candidate regresses any named large cohort. Treat + noise-sized or inconsistent gains as inconclusive and keep the current audit. diff --git a/dev/plans/README.md b/dev/plans/README.md index b9c085f..42e07ce 100644 --- a/dev/plans/README.md +++ b/dev/plans/README.md @@ -20,8 +20,12 @@ evidence-gated slices receive an executor plan only after their gate passes. replacement work, remove avoidable provider lookup/serialization and statement-per-content costs, then gate deeper transaction or WAL changes. 2. [Doctor and maintenance hot paths](260723-doctor-maintenance-hot-paths.md) — - make exact FTS verification memory-bounded, partition oversized terms, page - only orphan candidates, and gate any compact-proof optimization separately. -3. [Verified bounded session reads](260723-verified-bounded-session-reads.md) — + record the rejected document-interval gate, page only orphan candidates, and + keep compact-proof work separately gated. +3. [Single-pass doctor FTS feasibility](260723-doctor-single-pass-fts-feasibility.md) — + evaluate an exact term-ordered alternative without repeated + actual-vocabulary scans; gate any production refactor on corruption parity, + memory-only state, compatible ordering, and measured scaling. +4. [Verified bounded session reads](260723-verified-bounded-session-reads.md) — stream complete validation and the existing public-document digest while retaining only the requested bounded `show` or `export` selection. diff --git a/docs/architecture-memo.md b/docs/architecture-memo.md index 7bac76b..dc7332d 100644 --- a/docs/architecture-memo.md +++ b/docs/architecture-memo.md @@ -1331,6 +1331,18 @@ canonical reads, batches the TEMP FTS load transactionally, partitions exact FTS instance comparison by a fixed occurrence target, reports interactive phases, and supports aggregate opt-in timings without weakening its exact proof. The complete expected index and one oversized term remain corpus-dependent. + +Generated feasibility evidence rejected document-ID-bounded actual-vocabulary +scans: 41 production-shaped intervals made total work 3.80 times slower and +grouped term-summary work 6.82 times slower while retaining the same +virtual-table scan shape. The accepted decision preserves the current exact +whole-library audit and retires that interval design. A separate +[single-pass feasibility plan](../dev/plans/260723-doctor-single-pass-fts-feasibility.md) +may evaluate one monotonic actual/expected instance traversal as a CPU +optimization, but no production refactor is authorized until it proves exact +corruption parity, memory-only transient state, compatible ordering, and better +scaling. It does not solve the corpus-sized expected index. + Crash-safe index generations now use a transaction-bound schema-3 receipt to replace global recovery scans only at exact certified boundaries; any ambiguity keeps the complete validation fallback. Further lower total-memory doctor work, diff --git a/docs/contributing/commands.md b/docs/contributing/commands.md index 713bb21..527570d 100644 --- a/docs/contributing/commands.md +++ b/docs/contributing/commands.md @@ -11,6 +11,7 @@ | `pnpm format:docs:check` | Check Markdown formatting | No | No | | `pnpm lint` / `pnpm lint:fix` | Check or fix source/test lint | Fix variant only | No | | `pnpm measure:content-storage` | Compare legacy and current canonical-content layouts with fixed corpora | Temporary directory, removed | No | +| `pnpm measure:doctor` | Probe document-bounded exact FTS validation with generated libraries | Temporary directory, removed | No | | `pnpm measure:query-lineage` | Compare repeated and query-scoped lineage resolution | No | No | | `pnpm measure:entry-query` | Measure fixed textless entry queries through production SQLite seams | In-memory database, removed | No | | `pnpm measure:search-query` | Measure a fixed broad first-page search through production SQLite seams | In-memory database, removed | No | @@ -44,6 +45,18 @@ checks require exact-content ID reuse, collision coexistence, no target index on text, and a realistic target database no larger than 60% of the legacy layout. Timings are report-only and can vary by machine and SQLite version. +`pnpm measure:doctor` is an opt-in feasibility probe outside `pnpm check`. It +seeds two provider-free libraries through the production writer, measures one, +two, and many exact document-bounded FTS vocabulary strategies in alternating +fresh child processes, and runs exact equality and final-health checks outside +the measured workers. The many-interval strategy uses the proposed 512-row and +16-MiB admission limits. It reports only generated aggregate plan, timing, +memory, corpus, and integrity evidence. Immutable file state, absent sidecars, +owned permissions, and complete success/failure cleanup are required; elapsed +time and RSS are report-only. The probe is supported on macOS and Linux because +the private-file evidence requires POSIX modes. The result cannot authorize a +production refactor without explicit human acceptance. + `pnpm measure:query-lineage` is also opt-in and outside `pnpm check`. It compares rebuilding lineage state per resolution with one query-scoped resolver over a deterministic generic in-memory corpus. Exact result equality is required; diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index 3fc898d..893d86b 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -204,6 +204,19 @@ composition, forced load failure, cleanup, and exact semantic mismatches. Application, runtime, and CLI tests keep progress/timing best-effort, normal redirected stderr quiet, and doctor stdout/check/exit behavior unchanged. +`pnpm measure:doctor` is the opt-in, provider-free document-interval feasibility +probe. It seeds small and large generated libraries through the production +writer, measures one, two, and many document-bounded `fts5vocab` strategies in +alternating fresh child processes, and records normalized plan facts, elapsed +phases, and peak RSS. The many strategy uses the proposed 512-row and 16-MiB +admission limits. Separate exact comparisons require complete coordinate, +term-summary, docsize, interval, tail, final-health, immutable-file, sidecar, +permission, and cleanup equality. A reduced `--contract` mode owns deterministic +script, direct-worker isolation, and forced-failure coverage in Vitest on macOS +and Linux; Windows skips the POSIX-mode contract. Timings never pass or fail CI, +and the measurement cannot authorize a production doctor refactor; its recorded +result requires explicit human acceptance. + An accepted live doctor measurement may run against the ordinary retained library only as one process: do not open the database through `sqlite3` or any second SQLite connection while the immutable snapshot is active. Capture the diff --git a/docs/research/doctor-document-interval-feasibility.md b/docs/research/doctor-document-interval-feasibility.md new file mode 100644 index 0000000..ddde2d0 --- /dev/null +++ b/docs/research/doctor-document-interval-feasibility.md @@ -0,0 +1,148 @@ +# Doctor document-interval feasibility + +## Status + +Measurement and human review completed on 2026-07-23. Document-bounded +`fts5vocab(..., 'instance')` scans are rejected as the basis for bounded doctor +verification. + +Production `doctor` behavior is unchanged. Sections 2–3 of the originating plan +are retired; successor work is evidence-gated in +[the single-pass FTS feasibility plan](../../dev/plans/260723-doctor-single-pass-fts-feasibility.md). + +## Question + +Can exact doctor FTS verification divide the actual FTS vocabulary into many +document-ID intervals without repeatedly traversing the complete vocabulary? + +The proposed design would build and discard one bounded expected TEMP FTS +projection per canonical-content interval. That only helps if the matching +document-bounded reads of the actual `fts5vocab` projection also have bounded +work. + +## Method + +`pnpm measure:doctor` creates two provider-free libraries through the production +SQLite writer. Each includes: + +- repeated shared and unique generated text; +- zero-token and multibyte text; +- one generated value above the 16 MiB byte-interval target; and +- one generated term above a reduced measurement-only instance target. + +The measurement closes the writer, creates byte-identical mode-`0600` clones, +and runs every strategy in alternating fresh child processes. Measured workers +open the clone with `mode=ro&immutable=1`, require memory-only TEMP storage, and +consume: + +- raw `(term, doc, col, offset)` instance rows; and +- term summaries grouped by term with exact document and instance counts. + +The `one`, `two`, and `many` strategies cover the same document IDs and include +the required final actual-vocabulary tail query. `one` and `two` are equal-split +controls. `many` uses the proposed production admission rule: at most 512 +canonical rows or 16 MiB of UTF-8 text per interval, with an oversized value +processed alone. Exact coordinate and merged term-summary equality run +separately so their retained comparison state does not contaminate measured +peak RSS. + +Elapsed time and RSS are observations, not correctness thresholds. The OS page +cache is not flushed, so these are comparative warm measurements rather than +cold-start results. The probe is supported on macOS and Linux because its +private-file evidence relies on POSIX modes. + +Environment: + +- Node.js 24.18.0 +- SQLite 3.53.3 +- macOS arm64 + +## Result + +All values below are medians of three fresh-process samples. Peak RSS is the +largest child-process value across those samples. + +| Corpus | Content rows | FTS instances | Database bytes | +| ------ | -----------: | ------------: | -------------: | +| Small | 516 | 4,616 | 17,215,488 | +| Large | 20,004 | 180,008 | 25,833,472 | + +### Small corpus + +| Intervals | Queries per shape | Raw instances | Term summaries | Total | Peak RSS | +| --------: | ----------------: | ------------: | -------------: | -------: | --------: | +| 1 | 2 | 4.970 ms | 1.213 ms | 6.138 ms | 119.9 MiB | +| 2 | 3 | 5.551 ms | 1.423 ms | 6.976 ms | 117.7 MiB | +| 3 | 4 | 5.524 ms | 1.674 ms | 7.199 ms | 117.8 MiB | + +At three admission intervals, total elapsed was 1.17 times the one-interval +result. Raw instance work was 1.11 times slower and grouped term-summary work was +1.38 times slower. + +### Large corpus + +| Intervals | Queries per shape | Raw instances | Term summaries | Total | Peak RSS | +| --------: | ----------------: | ------------: | -------------: | ---------: | --------: | +| 1 | 2 | 114.368 ms | 35.653 ms | 150.365 ms | 169.7 MiB | +| 2 | 3 | 130.498 ms | 42.172 ms | 172.672 ms | 168.6 MiB | +| 41 | 42 | 330.567 ms | 243.201 ms | 571.614 ms | 162.9 MiB | + +At 41 production-shaped admission intervals, total elapsed was 3.80 times the +one-interval result. Raw instance work was 2.89 times slower and grouped +term-summary work was 6.82 times slower. The strategies returned the same +180,008 exact instances. + +Each query-count value is per measured shape: the worker runs that many raw +instance queries and that many grouped term-summary queries. Peak RSS in this +actual-vocabulary-only probe was 0.96 times the one-interval maximum. It does not +construct the proposed interval-local expected projection, so it neither proves +nor disproves that the complete design could reduce expected-side memory. + +Two subsequent complete invocations reproduced the large-corpus direction at +41 intervals: 3.64–3.73 times total, 2.74–2.82 times raw-instance, and 6.47–6.59 +times grouped term-summary work. Both passed every hard correctness gate. +Small-corpus timings were within process noise and are not decision evidence. + +For both query shapes, normalized `EXPLAIN QUERY PLAN` facts were identical for +unbounded, prefix, middle, and final-tail predicates: + +- one virtual-table scan; +- no TEMP B-tree for raw instance consumption; and +- two TEMP B-trees for grouped term summaries. + +The combined plan and scaling evidence is consistent with repeated +whole-vocabulary traversal rather than document-bounded access. + +## Correctness and privacy evidence + +Both cohorts required: + +- exact ordered coordinate equality between whole-vocabulary and concatenated + interval reads; +- exact merged term/document/instance summary equality; +- exact docsize coverage, including zero-token rows; +- disjoint, complete interval accounting and an empty final tail; +- healthy production index inspection after measurement; +- byte-identical initial clones; +- unchanged device, inode, mode, ownership, link count, size, modification and + change timestamps, and SHA-256 digest after immutable reads; +- absent WAL, SHM, and journal sidecars before and after; +- mode-`0700` owned temporary directories and mode-`0600` database files; and +- complete temporary cleanup on success and a forced measured-child failure. + +The report contains only generated aggregate data. Contract tests recursively +allowlist report fields and reject generated text, terms, identities, locators, +paths, hashes, timestamps, raw SQL, errors, and process IDs. + +## Accepted decision + +The document-ID interval design in sections 2–3 of the originating doctor plan +is rejected. It can bound the expected TEMP projection, but its +actual-vocabulary query shape introduces an interval-count CPU multiplier on +the dominant doctor path. + +Preserve the current exact audit until a replacement design is accepted. +The successor feasibility plan evaluates one direct actual/expected instance +stream as a possible CPU improvement. It does not claim bounded total memory: +the complete expected TEMP FTS remains corpus-sized until a safe, exact +tokenizer-ordering primitive exists. diff --git a/package.json b/package.json index aad8c6e..2482b6b 100644 --- a/package.json +++ b/package.json @@ -48,6 +48,7 @@ "lint:fix": "oxlint -c .oxlintrc.json --fix src scripts test vitest.config.ts", "measure:cli-startup": "node scripts/measure-cli-startup.ts", "measure:content-storage": "node scripts/measure-content-storage.ts", + "measure:doctor": "node scripts/measure-doctor.ts", "measure:entry-query": "node scripts/measure-entry-query.ts", "measure:indexing": "node scripts/measure-indexing.ts", "measure:indexing:codex": "node scripts/measure-codex-indexing.ts", diff --git a/scripts/measure-doctor.ts b/scripts/measure-doctor.ts new file mode 100644 index 0000000..bd1ab74 --- /dev/null +++ b/scripts/measure-doctor.ts @@ -0,0 +1,2048 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { createHash, randomBytes } from "node:crypto"; +import { + chmod, + copyFile, + lstat, + mkdir, + mkdtemp, + readFile, + readdir, + realpath, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { performance } from "node:perf_hooks"; +import { DatabaseSync, type StatementSync } from "node:sqlite"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +import { createDiscoveredSession } from "../src/application/source-input-fingerprint.ts"; +import type { IndexPaths } from "../src/application/ports/index-lifecycle.ts"; +import { + admitSessionObservation, + admitSessionReplacement, + type ValidatedSessionReplacement, +} from "../src/application/validate-session.ts"; +import { hashContent } from "../src/domain/content-hash.ts"; +import type { + SessionDocument, + SessionEntry, + SessionIdentity, + SourceInstance, +} from "../src/domain/session.ts"; +import { createSqliteIndexLifecycle } from "../src/infrastructure/sqlite/database.ts"; +import { resolveIndexPaths } from "../src/infrastructure/state/paths.ts"; + +const SCRIPT_PATH = fileURLToPath(import.meta.url); +const FAILURE_MESSAGE = "Doctor feasibility measurement failed\n"; +const PRIVATE_CANARY = "doctor-measurement-private-canary"; +const GENERATED_AT = "2026-07-23T12:00:00.000Z"; +const FINISHED_AT = "2026-07-23T12:01:00.000Z"; +const CLOSED_AT = "2026-07-23T12:02:00.000Z"; +const WRITER_TOKEN = "doctor-measurement-writer"; +const FTS_VOCAB_TABLE = "sessions_measure_actual_vocab"; +const AUTHORITY_MARKER = ".doctor-measurement-authority"; +const PRODUCTION_BYTE_INTERVAL_TARGET = 16 * 1024 * 1024; +const CONTRACT_SMALL_BYTE_TARGET = 64 * 1024; +const CONTRACT_LARGE_BYTE_TARGET = 128 * 1024; +const STRATEGIES = ["one", "two", "many"] as const; +const SIDECAR_SUFFIXES = ["-wal", "-shm", "-journal"] as const; + +type StrategyName = (typeof STRATEGIES)[number]; +type CohortName = "small" | "large"; +type WorkerMode = "seed" | "equality" | "measure" | "health"; + +interface CohortConfiguration { + readonly name: CohortName; + readonly uniqueValues: number; + readonly manyIntervals: number; + readonly rowIntervalTarget: number; + readonly byteIntervalTarget: number; + readonly repeatedTermTarget: number; +} + +interface ContentRow { + readonly contentId: number; + readonly bytes: number; +} + +interface DocumentInterval { + readonly lowerExclusive: number; + readonly upperInclusive: number; +} + +interface SeedWorkerReport { + readonly contentRows: number; + readonly contentBytes: number; + readonly entryRows: number; + readonly instanceRows: number; + readonly repeatedTermInstances: number; + readonly oversizedContentRows: number; + readonly zeroTokenContentRows: number; +} + +interface EqualityStrategyReport { + readonly name: StrategyName; + readonly intervalCount: number; + readonly intervals: readonly DocumentInterval[]; + readonly coordinateEquality: boolean; + readonly termSummaryEquality: boolean; + readonly docsizeCoverage: boolean; + readonly finalTailEmpty: boolean; +} + +interface EqualityWorkerReport { + readonly contentRows: number; + readonly instanceRows: number; + readonly termCount: number; + readonly strategies: readonly EqualityStrategyReport[]; +} + +interface PlanAggregate { + readonly rows: number; + readonly virtualTableScans: number; + readonly temporaryBtrees: number; +} + +interface PlanShapeReport { + readonly unbounded: PlanAggregate; + readonly prefix: PlanAggregate; + readonly middle: PlanAggregate; + readonly tail: PlanAggregate; + readonly prefixDiffersFromUnbounded: boolean; + readonly middleDiffersFromUnbounded: boolean; + readonly tailDiffersFromUnbounded: boolean; +} + +interface ProbePlanReport { + readonly rawInstances: PlanShapeReport; + readonly termSummaries: PlanShapeReport; +} + +interface MeasureWorkerReport { + readonly strategy: StrategyName; + readonly intervalCount: number; + readonly queriesPerShape: number; + readonly instanceRows: number; + readonly termSummaryRows: number; + readonly phases: { + readonly rawInstancesMs: number; + readonly termSummariesMs: number; + readonly totalMs: number; + }; + readonly peakRssBytes: number; + readonly memoryOnlyTemp: boolean; + readonly tempCleanup: boolean; + readonly plans: ProbePlanReport; +} + +interface HealthWorkerReport { + readonly healthy: boolean; +} + +interface NumericAggregate { + readonly samples: readonly number[]; + readonly median: number; + readonly p95: number; +} + +interface RssAggregate { + readonly samples: readonly number[]; + readonly max: number; +} + +interface StrategyMeasurementReport { + readonly name: StrategyName; + readonly intervalCount: number; + readonly bounds: { + readonly firstLowerExclusive: number; + readonly lastUpperInclusive: number; + readonly minimumIdSpan: number; + readonly maximumIdSpan: number; + }; + readonly queriesPerShape: number; + readonly instanceRows: number; + readonly termSummaryRows: number; + readonly phases: { + readonly rawInstancesMs: NumericAggregate; + readonly termSummariesMs: NumericAggregate; + readonly totalMs: NumericAggregate; + }; + readonly peakRssBytes: RssAggregate; +} + +interface CohortReport { + readonly name: CohortName; + readonly corpus: { + readonly sessions: number; + readonly entries: number; + readonly contentRows: number; + readonly contentBytes: number; + readonly instanceRows: number; + readonly repeatedTermTarget: number; + readonly repeatedTermInstances: number; + readonly rowIntervalTarget: number; + readonly byteIntervalTarget: number; + readonly oversizedContentRows: number; + readonly zeroTokenContentRows: number; + readonly databaseBytes: number; + }; + readonly cloneEquality: boolean; + readonly exactQueryEquality: boolean; + readonly intervalAccounting: boolean; + readonly finalHealth: boolean; + readonly persistentFileStateEqual: boolean; + readonly sidecarsAbsentBeforeAndAfter: boolean; + readonly ownedPermissions: boolean; + readonly plans: ProbePlanReport; + readonly strategies: readonly StrategyMeasurementReport[]; + readonly scaling: { + readonly twoToOneTotalRatio: number; + readonly manyToOneTotalRatio: number; + readonly manyToOneRawInstancesRatio: number; + readonly manyToOneTermSummariesRatio: number; + readonly manyToOnePeakRssRatio: number; + }; +} + +interface MeasurementReport { + readonly schemaVersion: 1; + readonly command: "measure-doctor"; + readonly mode: "contract" | "full"; + readonly acceptance: "accepted-rejection"; + readonly configuration: { + readonly timingRounds: number; + readonly intervalStrategies: readonly StrategyName[]; + readonly peakRssUnit: "bytes"; + readonly generatedOnly: true; + readonly productionDoctorUnchanged: true; + }; + readonly cohorts: readonly CohortReport[]; + readonly temporaryCleanup: true; +} + +interface FileSnapshot { + readonly device: bigint; + readonly inode: bigint; + readonly mode: bigint; + readonly uid: bigint; + readonly gid: bigint; + readonly links: bigint; + readonly size: bigint; + readonly modified: bigint; + readonly changed: bigint; + readonly digest: string; +} + +interface OwnedRoot { + readonly path: string; + readonly device: bigint; + readonly inode: bigint; +} + +const workerRequested = process.argv.includes("--worker"); +if (!workerRequested) { + try { + const report = await runController(); + process.stdout.write(`${JSON.stringify(report)}\n`); + } catch { + process.stderr.write(FAILURE_MESSAGE); + process.exitCode = 1; + } +} else { + try { + const workerMode = requiredWorkerMode(); + const report = await runWorker(workerMode); + process.stdout.write(`${JSON.stringify(report)}\n`); + } catch { + process.exitCode = 1; + } +} + +async function runController(): Promise { + assertControllerArguments(); + assert(process.platform !== "win32", "doctor feasibility measurement requires POSIX modes"); + const contract = process.argv.includes("--contract"); + const forceChildFailure = process.argv.includes("--force-child-failure"); + const configurations = cohortConfigurations(contract); + const timingRounds = contract ? 2 : 3; + const temporaryParent = process.env.SESSIONS_MEASURE_DOCTOR_TEMP_PARENT ?? tmpdir(); + const ownedRoot = await createMeasurementRoot(temporaryParent); + const rootPath = ownedRoot.path; + let report: MeasurementReport | undefined; + let operationFailure: unknown; + let cleanupFailure: unknown; + + try { + const capability = randomBytes(32).toString("hex"); + await createWorkerAuthority(rootPath, capability); + const cohorts: CohortReport[] = []; + for (const configuration of configurations) { + cohorts.push( + await measureCohort(rootPath, configuration, { + capability, + contract, + forceChildFailure, + timingRounds, + }), + ); + } + assert( + await hasMode(path.join(rootPath, AUTHORITY_MARKER), 0o600), + "measurement authority permissions changed", + ); + report = { + schemaVersion: 1, + command: "measure-doctor", + mode: contract ? "contract" : "full", + acceptance: "accepted-rejection", + configuration: { + timingRounds, + intervalStrategies: STRATEGIES, + peakRssUnit: "bytes", + generatedOnly: true, + productionDoctorUnchanged: true, + }, + cohorts, + temporaryCleanup: true, + }; + } catch (error) { + operationFailure = error; + } finally { + try { + await removeOwnedRoot(ownedRoot); + } catch (error) { + cleanupFailure = error; + } + } + + if (operationFailure !== undefined || cleanupFailure !== undefined) { + throw new AggregateError( + [operationFailure, cleanupFailure].filter((error) => error !== undefined), + "doctor feasibility measurement did not complete", + ); + } + assert(report !== undefined, "doctor feasibility measurement produced no report"); + return report; +} + +async function measureCohort( + root: string, + configuration: CohortConfiguration, + options: { + readonly capability: string; + readonly contract: boolean; + readonly forceChildFailure: boolean; + readonly timingRounds: number; + }, +): Promise { + const cohortRoot = path.join(root, configuration.name); + const seedDirectory = path.join(cohortRoot, "seed"); + await createOwnedDirectory(cohortRoot); + await createOwnedDirectory(seedDirectory); + const seedReport = readSeedWorkerReport( + runChild("seed", seedDirectory, configuration, options.contract, { + capability: options.capability, + root, + }), + ); + const seedPaths = pathsAt(seedDirectory); + await assertNoSidecars(seedPaths.database); + const seedSnapshotBefore = await snapshotFile(seedPaths.database); + const cloneDirectories = Object.fromEntries( + STRATEGIES.map((strategy) => [strategy, path.join(cohortRoot, strategy)]), + ) as Record; + const cloneSnapshots = {} as Record; + + for (const strategy of STRATEGIES) { + const directory = cloneDirectories[strategy]; + await createOwnedDirectory(directory); + const paths = pathsAt(directory); + await copyFile(seedPaths.database, paths.database); + await chmod(paths.database, 0o600); + await assertNoSidecars(paths.database); + const snapshot = await snapshotFile(paths.database); + assert.equal(snapshot.digest, seedSnapshotBefore.digest, "measurement clone bytes changed"); + assert.equal(snapshot.size, seedSnapshotBefore.size, "measurement clone size changed"); + cloneSnapshots[strategy] = snapshot; + } + + const equality = readEqualityWorkerReport( + runChild("equality", seedDirectory, configuration, options.contract, { + capability: options.capability, + root, + }), + ); + assert.equal(equality.contentRows, seedReport.contentRows, "equality content count changed"); + assert.equal(equality.instanceRows, seedReport.instanceRows, "equality instance count changed"); + assert.deepStrictEqual( + equality.strategies.map(({ name }) => name), + STRATEGIES, + "equality strategy order changed", + ); + + const samples = Object.fromEntries( + STRATEGIES.map((strategy) => [strategy, [] as MeasureWorkerReport[]]), + ) as Record; + let commonPlans: ProbePlanReport | undefined; + let failureInjected = false; + for (let round = 0; round < options.timingRounds; round += 1) { + for (const strategy of rotateStrategies(round)) { + const shouldFail: boolean = + options.forceChildFailure && configuration.name === "small" && !failureInjected; + failureInjected ||= shouldFail; + const raw = runChild("measure", cloneDirectories[strategy], configuration, options.contract, { + capability: options.capability, + root, + strategy, + forceFailure: shouldFail, + }); + const measurement = readMeasureWorkerReport(raw); + assert.equal(measurement.strategy, strategy, "measurement strategy changed"); + assert.equal( + measurement.intervalCount, + expectedIntervalCount(configuration, strategy), + "measurement interval count changed", + ); + assert.equal( + measurement.instanceRows, + equality.instanceRows, + "measurement skipped or duplicated FTS instances", + ); + assert(measurement.memoryOnlyTemp, "measurement TEMP storage was not memory-only"); + assert(measurement.tempCleanup, "measurement TEMP objects were not removed"); + if (commonPlans === undefined) commonPlans = measurement.plans; + else assert.deepStrictEqual(measurement.plans, commonPlans, "measurement plan changed"); + samples[strategy].push(measurement); + } + } + assert( + !options.forceChildFailure || failureInjected, + "forced doctor measurement failure was not injected", + ); + assert(commonPlans !== undefined, "doctor measurement captured no query plan"); + + let fileStateEqual = true; + let sidecarsAbsent = true; + let ownedPermissions = + (await hasMode(root, 0o700)) && + (await hasMode(path.join(root, AUTHORITY_MARKER), 0o600)) && + (await hasMode(cohortRoot, 0o700)) && + (await hasMode(seedDirectory, 0o700)); + for (const strategy of STRATEGIES) { + const paths = pathsAt(cloneDirectories[strategy]); + const after = await snapshotFile(paths.database); + fileStateEqual &&= sameFileSnapshot(cloneSnapshots[strategy], after); + sidecarsAbsent &&= await sidecarsAreAbsent(paths.database); + ownedPermissions &&= await hasMode(cloneDirectories[strategy], 0o700); + ownedPermissions &&= await hasMode(paths.database, 0o600); + } + + const seedSnapshotBeforeHealth = await snapshotFile(seedPaths.database); + const health = readHealthWorkerReport( + runChild("health", seedDirectory, configuration, options.contract, { + capability: options.capability, + root, + }), + ); + const seedSnapshotAfterHealth = await snapshotFile(seedPaths.database); + fileStateEqual &&= sameFileSnapshot(seedSnapshotBefore, seedSnapshotAfterHealth); + fileStateEqual &&= sameFileSnapshot(seedSnapshotBeforeHealth, seedSnapshotAfterHealth); + sidecarsAbsent &&= await sidecarsAreAbsent(seedPaths.database); + ownedPermissions &&= await hasMode(seedPaths.database, 0o600); + + const strategies = STRATEGIES.map((strategy) => { + const equalityStrategy = equality.strategies.find((candidate) => candidate.name === strategy); + assert(equalityStrategy !== undefined, "doctor equality strategy is absent"); + return summarizeStrategy(strategy, samples[strategy], equalityStrategy); + }); + const one = strategies[0]!; + const two = strategies[1]!; + const many = strategies[2]!; + const exactQueryEquality = equality.strategies.every( + (strategy) => strategy.coordinateEquality && strategy.termSummaryEquality, + ); + const intervalAccounting = equality.strategies.every( + (strategy) => + strategy.docsizeCoverage && + strategy.finalTailEmpty && + strategy.intervalCount === strategy.intervals.length, + ); + assert(exactQueryEquality, "doctor measurement exact query equality failed"); + assert(intervalAccounting, "doctor measurement interval accounting failed"); + assert(health.healthy, "doctor measurement final health failed"); + assert(fileStateEqual, "doctor measurement changed persistent database state"); + assert(sidecarsAbsent, "doctor measurement retained a SQLite sidecar"); + assert(ownedPermissions, "doctor measurement ownership permissions changed"); + + return { + name: configuration.name, + corpus: { + sessions: 1, + entries: seedReport.entryRows, + contentRows: seedReport.contentRows, + contentBytes: seedReport.contentBytes, + instanceRows: seedReport.instanceRows, + repeatedTermTarget: configuration.repeatedTermTarget, + repeatedTermInstances: seedReport.repeatedTermInstances, + rowIntervalTarget: configuration.rowIntervalTarget, + byteIntervalTarget: configuration.byteIntervalTarget, + oversizedContentRows: seedReport.oversizedContentRows, + zeroTokenContentRows: seedReport.zeroTokenContentRows, + databaseBytes: safeNumber(seedSnapshotBefore.size), + }, + cloneEquality: true, + exactQueryEquality, + intervalAccounting, + finalHealth: health.healthy, + persistentFileStateEqual: fileStateEqual, + sidecarsAbsentBeforeAndAfter: sidecarsAbsent, + ownedPermissions, + plans: commonPlans, + strategies, + scaling: { + twoToOneTotalRatio: ratio(two.phases.totalMs.median, one.phases.totalMs.median), + manyToOneTotalRatio: ratio(many.phases.totalMs.median, one.phases.totalMs.median), + manyToOneRawInstancesRatio: ratio( + many.phases.rawInstancesMs.median, + one.phases.rawInstancesMs.median, + ), + manyToOneTermSummariesRatio: ratio( + many.phases.termSummariesMs.median, + one.phases.termSummariesMs.median, + ), + manyToOnePeakRssRatio: ratio(many.peakRssBytes.max, one.peakRssBytes.max), + }, + }; +} + +function runChild( + mode: WorkerMode, + directory: string, + configuration: CohortConfiguration, + contract: boolean, + options: { + readonly capability: string; + readonly root: string; + readonly strategy?: StrategyName; + readonly forceFailure?: boolean; + }, +): unknown { + const arguments_ = [ + SCRIPT_PATH, + "--worker", + mode, + "--directory", + directory, + "--root", + options.root, + "--capability", + options.capability, + "--cohort", + configuration.name, + ]; + if (contract) arguments_.push("--contract"); + if (options.strategy !== undefined) arguments_.push("--strategy", options.strategy); + if (options.forceFailure === true) arguments_.push("--fail-after-open"); + + const result = spawnSync(process.execPath, arguments_, { + cwd: directory, + encoding: "utf8", + env: isolatedWorkerEnvironment(directory), + maxBuffer: 16 * 1024 * 1024, + timeout: contract ? 30_000 : 10 * 60_000, + }); + if (result.status !== 0 || result.signal !== null || result.stderr !== "") { + throw new Error("doctor feasibility worker failed"); + } + const lines = result.stdout.trimEnd().split("\n"); + assert.equal(lines.length, 1, "doctor feasibility worker emitted unexpected output"); + return JSON.parse(lines[0]!); +} + +function isolatedWorkerEnvironment(directory: string): NodeJS.ProcessEnv { + const isolatedHome = path.join(directory, "isolated-home"); + return { + ...process.env, + HOME: isolatedHome, + USERPROFILE: isolatedHome, + CODEX_HOME: path.join(directory, "unavailable-codex"), + CODEX_SQLITE_HOME: undefined, + SESSIONS_DATA_DIR: undefined, + SESSIONS_MEASURE_DOCTOR_TEMP_PARENT: undefined, + }; +} + +async function runWorker(mode: WorkerMode): Promise { + assertWorkerArguments(mode); + const directory = requiredOption("--directory"); + const root = requiredOption("--root"); + const capability = requiredOption("--capability"); + await assertWorkerAuthority(root, directory, capability); + const configuration = configurationFor( + process.argv.includes("--contract"), + requiredOption("--cohort"), + ); + if (mode === "seed") { + await assertSeedDirectoryEmpty(directory); + return seedWorker(directory, configuration); + } + if (mode === "equality") return equalityWorker(directory, configuration); + if (mode === "health") return healthWorker(directory); + return measureWorker( + directory, + configuration, + requiredStrategy(requiredOption("--strategy")), + process.argv.includes("--fail-after-open"), + ); +} + +async function seedWorker( + directory: string, + configuration: CohortConfiguration, +): Promise { + const paths = pathsAt(directory); + const lifecycle = measurementLifecycle(); + const writer = await lifecycle.openWriter(paths); + let operationFailure: unknown; + try { + const run = await writer.sessions.startRun({ + source: sourceFor(configuration), + startedAt: GENERATED_AT, + }); + await writer.sessions.replaceSession(run, replacementFor(configuration)); + const result = await writer.sessions.finishRun(run, { + status: "completed", + finishedAt: FINISHED_AT, + }); + assert.deepStrictEqual(result.counts, { + discovered: 1, + unchanged: 0, + updated: 1, + failed: 0, + missing: 0, + stale: 0, + }); + } catch (error) { + operationFailure = error; + } + let closeFailure: unknown; + try { + await writer.close(); + } catch (error) { + closeFailure = error; + } + if (operationFailure !== undefined || closeFailure !== undefined) { + throw new AggregateError( + [operationFailure, closeFailure].filter((error) => error !== undefined), + "doctor measurement seed failed", + ); + } + await chmod(paths.database, 0o600); + await assertNoSidecars(paths.database); + return inspectSeededCorpus(paths.database, configuration); +} + +async function inspectSeededCorpus( + databasePath: string, + configuration: CohortConfiguration, +): Promise { + const database = openImmutableDatabase(databasePath); + try { + configureMemoryTemp(database); + createActualVocab(database); + const content = database + .prepare( + `SELECT COUNT(*) AS rows, + COALESCE(SUM(length(CAST(text AS BLOB))), 0) AS bytes, + COALESCE(SUM( + CASE WHEN length(CAST(text AS BLOB)) > ? THEN 1 ELSE 0 END + ), 0) AS oversized + FROM sessions_content_values`, + ) + .get(configuration.byteIntervalTarget) as Record | undefined; + const entries = database.prepare("SELECT COUNT(*) AS rows FROM sessions_entries").get() as + | Record + | undefined; + const instances = database + .prepare(`SELECT COUNT(*) AS rows FROM temp.${FTS_VOCAB_TABLE}`) + .get() as Record | undefined; + const repeated = database + .prepare(`SELECT COUNT(*) AS rows FROM temp.${FTS_VOCAB_TABLE} WHERE term = ?`) + .get("repeated") as Record | undefined; + const indexedDocuments = database.prepare( + `SELECT DISTINCT doc FROM temp.${FTS_VOCAB_TABLE} ORDER BY doc`, + ); + indexedDocuments.setReadBigInts(true); + const indexedDocumentCount = indexedDocuments.all().length; + const contentRows = integerNumber(content?.rows); + const report = { + contentRows, + contentBytes: integerNumber(content?.bytes), + entryRows: integerNumber(entries?.rows), + instanceRows: integerNumber(instances?.rows), + repeatedTermInstances: integerNumber(repeated?.rows), + oversizedContentRows: integerNumber(content?.oversized), + zeroTokenContentRows: contentRows - indexedDocumentCount, + }; + assert( + report.repeatedTermInstances > configuration.repeatedTermTarget, + "generated repeated term did not cross the measurement target", + ); + assert.equal(report.oversizedContentRows, 1, "generated oversized content count changed"); + assert(report.zeroTokenContentRows >= 2, "generated zero-token content is absent"); + return report; + } finally { + dropActualVocab(database); + database.close(); + } +} + +async function equalityWorker( + directory: string, + configuration: CohortConfiguration, +): Promise { + const database = openImmutableDatabase(pathsAt(directory).database); + try { + configureMemoryTemp(database); + createActualVocab(database); + const contentRows = readContentRows(database); + const contentIds = contentRows.map(({ contentId }) => contentId); + const instanceRows = countRows( + database, + `SELECT COUNT(*) AS rows FROM temp.${FTS_VOCAB_TABLE}`, + ); + const referenceSummaries = readTermSummaries(database); + const strategies = STRATEGIES.map((name) => { + const intervals = intervalsForStrategy(contentRows, configuration, name); + return { + name, + intervalCount: intervals.length, + intervals, + coordinateEquality: coordinatesMatch(database, intervals), + termSummaryEquality: termSummariesMatch(database, intervals, referenceSummaries), + docsizeCoverage: docsizeCoverageIsExact(database, contentIds, intervals), + finalTailEmpty: finalTailIsEmpty(database, intervals), + }; + }); + assert( + strategies.every( + (strategy) => + strategy.coordinateEquality && + strategy.termSummaryEquality && + strategy.docsizeCoverage && + strategy.finalTailEmpty, + ), + "document interval equality failed", + ); + return { + contentRows: contentIds.length, + instanceRows, + termCount: referenceSummaries.size, + strategies, + }; + } finally { + dropActualVocab(database); + database.close(); + } +} + +async function measureWorker( + directory: string, + configuration: CohortConfiguration, + strategy: StrategyName, + failAfterOpen: boolean, +): Promise { + const database = openImmutableDatabase(pathsAt(directory).database); + let memoryOnlyTemp = false; + let tempCleanup = false; + try { + configureMemoryTemp(database); + memoryOnlyTemp = true; + createActualVocab(database); + if (failAfterOpen) throw new Error("forced doctor measurement failure"); + const intervals = intervalsForStrategy(readContentRows(database), configuration, strategy); + const plans = readProbePlans(database, intervals); + const totalStartedAt = performance.now(); + const rawStartedAt = performance.now(); + const instanceRows = consumeInstanceQueries(database, intervals); + const rawInstancesMs = performance.now() - rawStartedAt; + const summariesStartedAt = performance.now(); + const termSummaryRows = consumeTermSummaryQueries(database, intervals); + const termSummariesMs = performance.now() - summariesStartedAt; + const totalMs = performance.now() - totalStartedAt; + assert(finalTailIsEmpty(database, intervals), "measurement final FTS tail was not empty"); + dropActualVocab(database); + tempCleanup = actualVocabIsAbsent(database); + const maxRssKiB = process.resourceUsage().maxRSS; + assert(Number.isSafeInteger(maxRssKiB) && maxRssKiB > 0, "worker peak RSS is invalid"); + return { + strategy, + intervalCount: intervals.length, + queriesPerShape: intervals.length + 1, + instanceRows, + termSummaryRows, + phases: { + rawInstancesMs: roundMeasurement(rawInstancesMs), + termSummariesMs: roundMeasurement(termSummariesMs), + totalMs: roundMeasurement(totalMs), + }, + peakRssBytes: maxRssKiB * 1024, + memoryOnlyTemp, + tempCleanup, + plans, + }; + } finally { + if (!tempCleanup) dropActualVocab(database); + database.close(); + } +} + +async function healthWorker(directory: string): Promise { + const paths = pathsAt(directory); + const health = await measurementLifecycle().inspectHealth(paths); + assert(health.ok, "generated doctor measurement library is not healthy"); + return { healthy: true }; +} + +function measurementLifecycle() { + return createSqliteIndexLifecycle({ + now: () => new Date(CLOSED_AT), + writerToken: () => WRITER_TOKEN, + }); +} + +function replacementFor(configuration: CohortConfiguration): ValidatedSessionReplacement { + const identity = identityFor(configuration); + const candidate = createDiscoveredSession({ + identity, + inputs: [ + { + role: "transcript", + locator: { uri: `memory://${PRIVATE_CANARY}/${configuration.name}` }, + fingerprint: `revision-${configuration.name}`, + }, + ], + adapterVersion: "synthetic-doctor-measurement-v1", + }); + const observation = admitSessionObservation(candidate); + assert(observation.ok, "generated doctor measurement observation was rejected"); + const replacement = admitSessionReplacement(observation.observation, documentFor(configuration)); + assert(replacement.ok, "generated doctor measurement document was rejected"); + return replacement.replacement; +} + +function documentFor(configuration: CohortConfiguration): SessionDocument { + const identity = identityFor(configuration); + const texts = [ + "shared generated doctor evidence", + "shared generated doctor evidence", + " \t\n ", + "multibyte résumé 東京 evidence", + ...Array.from( + { length: configuration.uniqueValues }, + (_, ordinal) => + `repeated generated evidence unique-${String(ordinal).padStart(6, "0")} ${PRIVATE_CANARY}`, + ), + "!".repeat(configuration.byteIntervalTarget + 1), + ]; + return { + identity, + title: `Generated doctor measurement ${configuration.name}`, + workspace: "/generated/doctor-measurement", + createdAt: GENERATED_AT, + updatedAt: GENERATED_AT, + lineageCoverage: "complete", + relations: [], + entries: texts.map((text, ordinal) => entryFor(configuration, ordinal, text)), + }; +} + +function entryFor(configuration: CohortConfiguration, ordinal: number, text: string): SessionEntry { + return { + ordinal, + kind: "message", + actor: ordinal % 2 === 0 ? "human" : "model", + timestamp: GENERATED_AT, + sourceLocator: { + uri: `memory://${PRIVATE_CANARY}/${configuration.name}/entry/${String(ordinal)}`, + }, + content: [ + { + kind: "text", + ordinal: 0, + text, + contentHash: hashContent(text), + origin: "unknown", + originConfidence: "high", + sourceMetadata: { fixture: "generated-doctor-measurement" }, + }, + ], + }; +} + +function identityFor(configuration: CohortConfiguration): SessionIdentity { + return { + source: sourceFor(configuration), + nativeId: `generated-${configuration.name}`, + }; +} + +function sourceFor(configuration: CohortConfiguration): SourceInstance { + return { + kind: "synthetic-measurement", + instanceId: `doctor-${configuration.name}`, + }; +} + +function readContentRows(database: DatabaseSync): readonly ContentRow[] { + const statement = database.prepare( + `SELECT content_id, length(CAST(text AS BLOB)) AS bytes + FROM sessions_content_values + ORDER BY content_id`, + ); + statement.setReadBigInts(true); + return (statement.all() as readonly Record[]).map((row) => ({ + contentId: safeNumber(sqliteInteger(row.content_id)), + bytes: safeNumber(sqliteInteger(row.bytes)), + })); +} + +function intervalsForStrategy( + contentRows: readonly ContentRow[], + configuration: CohortConfiguration, + strategy: StrategyName, +): readonly DocumentInterval[] { + const contentIds = contentRows.map(({ contentId }) => contentId); + const intervals = + strategy === "many" + ? createAdmissionIntervals( + contentRows, + configuration.rowIntervalTarget, + configuration.byteIntervalTarget, + ) + : createEqualIntervals(contentIds, expectedIntervalCount(configuration, strategy)); + assert.equal( + intervals.length, + expectedIntervalCount(configuration, strategy), + "doctor measurement admission interval count changed", + ); + return intervals; +} + +function createEqualIntervals( + contentIds: readonly number[], + requestedCount: number, +): readonly DocumentInterval[] { + assert(contentIds.length > 0, "doctor measurement content IDs are empty"); + assert( + Number.isSafeInteger(requestedCount) && + requestedCount > 0 && + requestedCount <= contentIds.length, + "doctor measurement interval count is invalid", + ); + const intervals: DocumentInterval[] = []; + for (let index = 0; index < requestedCount; index += 1) { + const end = Math.floor(((index + 1) * contentIds.length) / requestedCount) - 1; + const upperInclusive = contentIds[end]; + assert(upperInclusive !== undefined, "doctor measurement interval upper bound is absent"); + const lowerExclusive = index === 0 ? contentIds[0]! - 1 : intervals[index - 1]!.upperInclusive; + intervals.push({ lowerExclusive, upperInclusive }); + } + assertIntervals(contentIds, intervals); + return intervals; +} + +function createAdmissionIntervals( + contentRows: readonly ContentRow[], + rowTarget: number, + byteTarget: number, +): readonly DocumentInterval[] { + assert(contentRows.length > 0, "doctor measurement content rows are empty"); + assert( + Number.isSafeInteger(rowTarget) && + rowTarget > 0 && + Number.isSafeInteger(byteTarget) && + byteTarget > 0, + "doctor measurement admission targets are invalid", + ); + const intervals: DocumentInterval[] = []; + let lowerExclusive = contentRows[0]!.contentId - 1; + let admittedRows = 0; + let admittedBytes = 0; + let lastContentId: number | undefined; + + const finishInterval = () => { + assert(lastContentId !== undefined, "doctor measurement admission interval is empty"); + intervals.push({ lowerExclusive, upperInclusive: lastContentId }); + lowerExclusive = lastContentId; + admittedRows = 0; + admittedBytes = 0; + lastContentId = undefined; + }; + + for (const row of contentRows) { + assert( + Number.isSafeInteger(row.bytes) && row.bytes >= 0, + "doctor measurement content byte length is invalid", + ); + if (admittedRows > 0 && (admittedRows >= rowTarget || admittedBytes + row.bytes > byteTarget)) { + finishInterval(); + } + admittedRows += 1; + admittedBytes += row.bytes; + lastContentId = row.contentId; + if (row.bytes > byteTarget) { + assert.equal(admittedRows, 1, "oversized doctor content was not admitted alone"); + finishInterval(); + } + } + if (admittedRows > 0) finishInterval(); + assertIntervals( + contentRows.map(({ contentId }) => contentId), + intervals, + ); + return intervals; +} + +function assertIntervals( + contentIds: readonly number[], + intervals: readonly DocumentInterval[], +): void { + let previousUpper: number | undefined; + for (const interval of intervals) { + assert( + Number.isSafeInteger(interval.lowerExclusive) && + Number.isSafeInteger(interval.upperInclusive) && + interval.lowerExclusive < interval.upperInclusive, + "doctor measurement interval is malformed", + ); + if (previousUpper !== undefined) { + assert.equal( + interval.lowerExclusive, + previousUpper, + "doctor measurement intervals are not contiguous", + ); + } + previousUpper = interval.upperInclusive; + } + for (const contentId of contentIds) { + assert.equal( + intervals.filter( + (interval) => contentId > interval.lowerExclusive && contentId <= interval.upperInclusive, + ).length, + 1, + "doctor measurement content ID was not covered exactly once", + ); + } +} + +function coordinatesMatch(database: DatabaseSync, intervals: readonly DocumentInterval[]): boolean { + const reference = database.prepare( + `SELECT term, doc, col, offset + FROM temp.${FTS_VOCAB_TABLE} + ORDER BY doc, term COLLATE BINARY, col COLLATE BINARY, offset`, + ); + reference.setReadBigInts(true); + const referenceIterator = reference.iterate()[Symbol.iterator]() as Iterator< + Record + >; + try { + for (const [index, interval] of intervals.entries()) { + const query = prepareCoordinateInterval(database, index === 0); + const parameters = + index === 0 + ? ([interval.upperInclusive] as const) + : ([interval.lowerExclusive, interval.upperInclusive] as const); + for (const row of query.iterate(...parameters) as Iterable>) { + const expected = referenceIterator.next(); + if (expected.done || !sameCoordinate(expected.value, row)) return false; + } + } + return referenceIterator.next().done === true; + } finally { + referenceIterator.return?.(); + } +} + +function prepareCoordinateInterval(database: DatabaseSync, prefix: boolean): StatementSync { + const statement = database.prepare( + `SELECT term, doc, col, offset + FROM temp.${FTS_VOCAB_TABLE} + WHERE ${prefix ? "doc <= ?" : "doc > ? AND doc <= ?"} + ORDER BY doc, term COLLATE BINARY, col COLLATE BINARY, offset`, + ); + statement.setReadBigInts(true); + return statement; +} + +function sameCoordinate(left: Record, right: Record): boolean { + return ( + left.term === right.term && + left.doc === right.doc && + left.col === right.col && + left.offset === right.offset + ); +} + +interface TermSummary { + readonly documents: bigint; + readonly instances: bigint; +} + +function readTermSummaries(database: DatabaseSync): ReadonlyMap { + const statement = database.prepare( + `SELECT term, COUNT(DISTINCT doc) AS documents, COUNT(*) AS instances + FROM temp.${FTS_VOCAB_TABLE} + GROUP BY term + ORDER BY term COLLATE BINARY`, + ); + statement.setReadBigInts(true); + const summaries = new Map(); + for (const row of statement.iterate() as Iterable>) { + const term = nonEmptyString(row.term); + summaries.set(term, { + documents: positiveSqliteInteger(row.documents), + instances: positiveSqliteInteger(row.instances), + }); + } + return summaries; +} + +function termSummariesMatch( + database: DatabaseSync, + intervals: readonly DocumentInterval[], + reference: ReadonlyMap, +): boolean { + const actual = new Map(); + for (const [index, interval] of intervals.entries()) { + const statement = prepareTermSummaryInterval(database, index === 0); + const parameters = + index === 0 + ? ([interval.upperInclusive] as const) + : ([interval.lowerExclusive, interval.upperInclusive] as const); + for (const row of statement.iterate(...parameters) as Iterable>) { + const term = nonEmptyString(row.term); + const existing = actual.get(term); + const documents = positiveSqliteInteger(row.documents); + const instances = positiveSqliteInteger(row.instances); + actual.set(term, { + documents: (existing?.documents ?? 0n) + documents, + instances: (existing?.instances ?? 0n) + instances, + }); + } + } + if (actual.size !== reference.size) return false; + for (const [term, expected] of reference) { + const value = actual.get(term); + if ( + value === undefined || + value.documents !== expected.documents || + value.instances !== expected.instances + ) { + return false; + } + } + return true; +} + +function prepareTermSummaryInterval(database: DatabaseSync, prefix: boolean): StatementSync { + const statement = database.prepare( + `SELECT term, COUNT(DISTINCT doc) AS documents, COUNT(*) AS instances + FROM temp.${FTS_VOCAB_TABLE} + WHERE ${prefix ? "doc <= ?" : "doc > ? AND doc <= ?"} + GROUP BY term + ORDER BY term COLLATE BINARY`, + ); + statement.setReadBigInts(true); + return statement; +} + +function docsizeCoverageIsExact( + database: DatabaseSync, + contentIds: readonly number[], + intervals: readonly DocumentInterval[], +): boolean { + const statement = database.prepare("SELECT id FROM sessions_content_fts_docsize ORDER BY id"); + statement.setReadBigInts(true); + const indexedIds = (statement.all() as readonly Record[]).map((row) => + safeNumber(sqliteInteger(row.id)), + ); + if ( + indexedIds.length !== contentIds.length || + indexedIds.some((contentId, index) => contentId !== contentIds[index]) + ) { + return false; + } + return contentIds.every( + (contentId) => + intervals.filter( + (interval) => contentId > interval.lowerExclusive && contentId <= interval.upperInclusive, + ).length === 1, + ); +} + +function finalTailIsEmpty(database: DatabaseSync, intervals: readonly DocumentInterval[]): boolean { + const upper = intervals.at(-1)?.upperInclusive; + assert(upper !== undefined, "doctor measurement final interval is absent"); + return ( + database + .prepare(`SELECT 1 AS unexpected FROM temp.${FTS_VOCAB_TABLE} WHERE doc > ? LIMIT 1`) + .get(upper) === undefined + ); +} + +function consumeInstanceQueries( + database: DatabaseSync, + intervals: readonly DocumentInterval[], +): number { + let rows = 0; + for (const [index, interval] of intervals.entries()) { + const statement = database.prepare( + `SELECT term, doc, col, offset + FROM temp.${FTS_VOCAB_TABLE} + WHERE ${index === 0 ? "doc <= ?" : "doc > ? AND doc <= ?"}`, + ); + statement.setReadBigInts(true); + const parameters = + index === 0 + ? ([interval.upperInclusive] as const) + : ([interval.lowerExclusive, interval.upperInclusive] as const); + for (const row of statement.iterate(...parameters) as Iterable>) { + validateCoordinate(row); + rows += 1; + } + } + const finalUpper = intervals.at(-1)!.upperInclusive; + const tail = database.prepare( + `SELECT term, doc, col, offset FROM temp.${FTS_VOCAB_TABLE} WHERE doc > ?`, + ); + tail.setReadBigInts(true); + for (const row of tail.iterate(finalUpper) as Iterable>) { + validateCoordinate(row); + rows += 1; + } + return rows; +} + +function consumeTermSummaryQueries( + database: DatabaseSync, + intervals: readonly DocumentInterval[], +): number { + let rows = 0; + for (const [index, interval] of intervals.entries()) { + const statement = prepareTermSummaryInterval(database, index === 0); + const parameters = + index === 0 + ? ([interval.upperInclusive] as const) + : ([interval.lowerExclusive, interval.upperInclusive] as const); + for (const row of statement.iterate(...parameters) as Iterable>) { + validateTermSummary(row); + rows += 1; + } + } + const tail = database.prepare( + `SELECT term, COUNT(DISTINCT doc) AS documents, COUNT(*) AS instances + FROM temp.${FTS_VOCAB_TABLE} + WHERE doc > ? + GROUP BY term + ORDER BY term COLLATE BINARY`, + ); + tail.setReadBigInts(true); + for (const row of tail.iterate(intervals.at(-1)!.upperInclusive) as Iterable< + Record + >) { + validateTermSummary(row); + rows += 1; + } + return rows; +} + +function validateCoordinate(row: Record): void { + nonEmptyString(row.term); + positiveSqliteInteger(row.doc); + nonEmptyString(row.col); + const offset = sqliteInteger(row.offset); + assert(offset >= 0n, "doctor measurement FTS offset is negative"); +} + +function validateTermSummary(row: Record): void { + nonEmptyString(row.term); + positiveSqliteInteger(row.documents); + positiveSqliteInteger(row.instances); +} + +function readProbePlans( + database: DatabaseSync, + intervals: readonly DocumentInterval[], +): ProbePlanReport { + const first = intervals[0]!; + const last = intervals.at(-1)!; + const middleLower = intervals.length > 1 ? intervals[1]!.lowerExclusive : first.lowerExclusive; + const middleUpper = intervals.length > 1 ? intervals[1]!.upperInclusive : first.upperInclusive; + return { + rawInstances: planShape( + database, + "SELECT term, doc, col, offset FROM temp.sessions_measure_actual_vocab%s", + { + prefix: [first.upperInclusive], + middle: [middleLower, middleUpper], + tail: [last.upperInclusive], + }, + ), + termSummaries: planShape( + database, + `SELECT term, COUNT(DISTINCT doc) AS documents, COUNT(*) AS instances + FROM temp.sessions_measure_actual_vocab%s + GROUP BY term + ORDER BY term COLLATE BINARY`, + { + prefix: [first.upperInclusive], + middle: [middleLower, middleUpper], + tail: [last.upperInclusive], + }, + ), + }; +} + +function planShape( + database: DatabaseSync, + template: string, + parameters: { + readonly prefix: readonly number[]; + readonly middle: readonly number[]; + readonly tail: readonly number[]; + }, +): PlanShapeReport { + const unboundedRows = explain(database, template.replace("%s", ""), []); + const prefixRows = explain( + database, + template.replace("%s", " WHERE doc <= ?"), + parameters.prefix, + ); + const middleRows = explain( + database, + template.replace("%s", " WHERE doc > ? AND doc <= ?"), + parameters.middle, + ); + const tailRows = explain(database, template.replace("%s", " WHERE doc > ?"), parameters.tail); + const unboundedSignature = planSignature(unboundedRows); + return { + unbounded: aggregatePlan(unboundedRows), + prefix: aggregatePlan(prefixRows), + middle: aggregatePlan(middleRows), + tail: aggregatePlan(tailRows), + prefixDiffersFromUnbounded: !sameStrings(unboundedSignature, planSignature(prefixRows)), + middleDiffersFromUnbounded: !sameStrings(unboundedSignature, planSignature(middleRows)), + tailDiffersFromUnbounded: !sameStrings(unboundedSignature, planSignature(tailRows)), + }; +} + +function explain( + database: DatabaseSync, + sql: string, + parameters: readonly number[], +): readonly Record[] { + return database.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...parameters) as readonly Record< + string, + unknown + >[]; +} + +function planSignature(rows: readonly Record[]): readonly string[] { + return rows.map((row) => + nonEmptyString(row.detail).replaceAll(FTS_VOCAB_TABLE, "").replace(/\s+/gu, " ").trim(), + ); +} + +function aggregatePlan(rows: readonly Record[]): PlanAggregate { + const details = rows.map((row) => nonEmptyString(row.detail)); + return { + rows: details.length, + virtualTableScans: details.filter((detail) => /^(?:SCAN|SEARCH) .+ VIRTUAL TABLE/u.test(detail)) + .length, + temporaryBtrees: details.filter((detail) => detail.includes("USE TEMP B-TREE")).length, + }; +} + +function sameStrings(left: readonly string[], right: readonly string[]): boolean { + return left.length === right.length && left.every((value, index) => value === right[index]); +} + +function createActualVocab(database: DatabaseSync): void { + database.exec( + `CREATE VIRTUAL TABLE temp.${FTS_VOCAB_TABLE} + USING fts5vocab(main, sessions_content_fts, 'instance')`, + ); +} + +function dropActualVocab(database: DatabaseSync): void { + database.exec(`DROP TABLE IF EXISTS temp.${FTS_VOCAB_TABLE}`); +} + +function actualVocabIsAbsent(database: DatabaseSync): boolean { + return ( + database + .prepare("SELECT 1 FROM sqlite_temp_schema WHERE name = ? LIMIT 1") + .get(FTS_VOCAB_TABLE) === undefined + ); +} + +function configureMemoryTemp(database: DatabaseSync): void { + database.exec("PRAGMA trusted_schema = OFF; PRAGMA temp_store = MEMORY"); + const row = database.prepare("PRAGMA temp_store").get() as Record | undefined; + assert.equal(row?.temp_store, 2, "doctor measurement TEMP storage is not memory-only"); +} + +function openImmutableDatabase(databasePath: string): DatabaseSync { + const url = pathToFileURL(databasePath); + url.searchParams.set("mode", "ro"); + url.searchParams.set("immutable", "1"); + return new DatabaseSync(url.href, { + allowExtension: false, + defensive: true, + enableDoubleQuotedStringLiterals: false, + enableForeignKeyConstraints: true, + readOnly: true, + timeout: 5_000, + }); +} + +function summarizeStrategy( + strategy: StrategyName, + samples: readonly MeasureWorkerReport[], + equality: EqualityStrategyReport, +): StrategyMeasurementReport { + assert(samples.length > 0, "doctor measurement strategy had no samples"); + const first = samples[0]!; + assert.equal( + first.intervalCount, + equality.intervalCount, + "doctor equality and measurement interval counts changed", + ); + for (const sample of samples) { + assert.equal(sample.strategy, strategy, "doctor measurement sample strategy changed"); + assert.equal( + sample.intervalCount, + first.intervalCount, + "doctor measurement sample interval count changed", + ); + assert.equal( + sample.queriesPerShape, + first.queriesPerShape, + "doctor measurement sample query count changed", + ); + assert.equal( + sample.instanceRows, + first.instanceRows, + "doctor measurement sample instance count changed", + ); + assert.equal( + sample.termSummaryRows, + first.termSummaryRows, + "doctor measurement sample term summary count changed", + ); + } + return { + name: strategy, + intervalCount: first.intervalCount, + bounds: summarizeIntervalBounds(equality.intervals), + queriesPerShape: first.queriesPerShape, + instanceRows: first.instanceRows, + termSummaryRows: first.termSummaryRows, + phases: { + rawInstancesMs: numericAggregate(samples.map((sample) => sample.phases.rawInstancesMs)), + termSummariesMs: numericAggregate(samples.map((sample) => sample.phases.termSummariesMs)), + totalMs: numericAggregate(samples.map((sample) => sample.phases.totalMs)), + }, + peakRssBytes: rssAggregate(samples.map((sample) => sample.peakRssBytes)), + }; +} + +function summarizeIntervalBounds( + intervals: readonly DocumentInterval[], +): StrategyMeasurementReport["bounds"] { + assert(intervals.length > 0, "doctor measurement interval bounds are empty"); + const spans = intervals.map((interval) => interval.upperInclusive - interval.lowerExclusive); + return { + firstLowerExclusive: intervals[0]!.lowerExclusive, + lastUpperInclusive: intervals.at(-1)!.upperInclusive, + minimumIdSpan: Math.min(...spans), + maximumIdSpan: Math.max(...spans), + }; +} + +function numericAggregate(values: readonly number[]): NumericAggregate { + assert(values.length > 0, "doctor measurement numeric aggregate is empty"); + assert( + values.every((value) => Number.isFinite(value) && value >= 0), + "doctor measurement elapsed sample is invalid", + ); + const sorted = [...values].sort((left, right) => left - right); + return { + samples: values, + median: percentile(sorted, 0.5), + p95: percentile(sorted, 0.95), + }; +} + +function rssAggregate(values: readonly number[]): RssAggregate { + assert( + values.length > 0 && values.every((value) => Number.isSafeInteger(value) && value > 0), + "doctor measurement RSS sample is invalid", + ); + return { samples: values, max: Math.max(...values) }; +} + +function percentile(sorted: readonly number[], fraction: number): number { + const index = Math.min(sorted.length - 1, Math.ceil(sorted.length * fraction) - 1); + return sorted[index]!; +} + +function ratio(numerator: number, denominator: number): number { + assert( + Number.isFinite(numerator) && Number.isFinite(denominator) && numerator >= 0 && denominator > 0, + "doctor measurement ratio inputs are invalid", + ); + return Number((numerator / denominator).toFixed(3)); +} + +function rotateStrategies(round: number): readonly StrategyName[] { + const offset = round % STRATEGIES.length; + return [...STRATEGIES.slice(offset), ...STRATEGIES.slice(0, offset)]; +} + +function expectedIntervalCount(configuration: CohortConfiguration, strategy: StrategyName): number { + if (strategy === "one") return 1; + if (strategy === "two") return 2; + return configuration.manyIntervals; +} + +function cohortConfigurations(contract: boolean): readonly CohortConfiguration[] { + if (contract) { + return [ + { + name: "small", + uniqueValues: 32, + manyIntervals: 4, + rowIntervalTarget: 12, + byteIntervalTarget: CONTRACT_SMALL_BYTE_TARGET, + repeatedTermTarget: 8, + }, + { + name: "large", + uniqueValues: 128, + manyIntervals: 8, + rowIntervalTarget: 20, + byteIntervalTarget: CONTRACT_LARGE_BYTE_TARGET, + repeatedTermTarget: 16, + }, + ]; + } + return [ + { + name: "small", + uniqueValues: 512, + manyIntervals: 3, + rowIntervalTarget: 512, + byteIntervalTarget: PRODUCTION_BYTE_INTERVAL_TARGET, + repeatedTermTarget: 64, + }, + { + name: "large", + uniqueValues: 20_000, + manyIntervals: 41, + rowIntervalTarget: 512, + byteIntervalTarget: PRODUCTION_BYTE_INTERVAL_TARGET, + repeatedTermTarget: 1_000, + }, + ]; +} + +function configurationFor(contract: boolean, name: string): CohortConfiguration { + const configuration = cohortConfigurations(contract).find((candidate) => candidate.name === name); + assert(configuration !== undefined, "doctor measurement cohort is invalid"); + return configuration; +} + +function readSeedWorkerReport(value: unknown): SeedWorkerReport { + const record = exactRecord(value, [ + "contentRows", + "contentBytes", + "entryRows", + "instanceRows", + "repeatedTermInstances", + "oversizedContentRows", + "zeroTokenContentRows", + ]); + return { + contentRows: positiveNumber(record.contentRows), + contentBytes: positiveNumber(record.contentBytes), + entryRows: positiveNumber(record.entryRows), + instanceRows: positiveNumber(record.instanceRows), + repeatedTermInstances: positiveNumber(record.repeatedTermInstances), + oversizedContentRows: positiveNumber(record.oversizedContentRows), + zeroTokenContentRows: positiveNumber(record.zeroTokenContentRows), + }; +} + +function readEqualityWorkerReport(value: unknown): EqualityWorkerReport { + const record = exactRecord(value, ["contentRows", "instanceRows", "termCount", "strategies"]); + assert(Array.isArray(record.strategies), "equality strategies are invalid"); + return { + contentRows: positiveNumber(record.contentRows), + instanceRows: positiveNumber(record.instanceRows), + termCount: positiveNumber(record.termCount), + strategies: record.strategies.map(readEqualityStrategyReport), + }; +} + +function readEqualityStrategyReport(value: unknown): EqualityStrategyReport { + const record = exactRecord(value, [ + "name", + "intervalCount", + "intervals", + "coordinateEquality", + "termSummaryEquality", + "docsizeCoverage", + "finalTailEmpty", + ]); + assert(Array.isArray(record.intervals), "equality intervals are invalid"); + const intervals = record.intervals.map((interval) => { + const bounds = exactRecord(interval, ["lowerExclusive", "upperInclusive"]); + return { + lowerExclusive: integerNumber(bounds.lowerExclusive), + upperInclusive: integerNumber(bounds.upperInclusive), + }; + }); + return { + name: requiredStrategy(record.name), + intervalCount: positiveNumber(record.intervalCount), + intervals, + coordinateEquality: trueBoolean(record.coordinateEquality), + termSummaryEquality: trueBoolean(record.termSummaryEquality), + docsizeCoverage: trueBoolean(record.docsizeCoverage), + finalTailEmpty: trueBoolean(record.finalTailEmpty), + }; +} + +function readMeasureWorkerReport(value: unknown): MeasureWorkerReport { + const record = exactRecord(value, [ + "strategy", + "intervalCount", + "queriesPerShape", + "instanceRows", + "termSummaryRows", + "phases", + "peakRssBytes", + "memoryOnlyTemp", + "tempCleanup", + "plans", + ]); + const phases = exactRecord(record.phases, ["rawInstancesMs", "termSummariesMs", "totalMs"]); + return { + strategy: requiredStrategy(record.strategy), + intervalCount: positiveNumber(record.intervalCount), + queriesPerShape: positiveNumber(record.queriesPerShape), + instanceRows: positiveNumber(record.instanceRows), + termSummaryRows: positiveNumber(record.termSummaryRows), + phases: { + rawInstancesMs: nonNegativeFinite(phases.rawInstancesMs), + termSummariesMs: nonNegativeFinite(phases.termSummariesMs), + totalMs: nonNegativeFinite(phases.totalMs), + }, + peakRssBytes: positiveNumber(record.peakRssBytes), + memoryOnlyTemp: trueBoolean(record.memoryOnlyTemp), + tempCleanup: trueBoolean(record.tempCleanup), + plans: readProbePlanReport(record.plans), + }; +} + +function readProbePlanReport(value: unknown): ProbePlanReport { + const record = exactRecord(value, ["rawInstances", "termSummaries"]); + return { + rawInstances: readPlanShapeReport(record.rawInstances), + termSummaries: readPlanShapeReport(record.termSummaries), + }; +} + +function readPlanShapeReport(value: unknown): PlanShapeReport { + const record = exactRecord(value, [ + "unbounded", + "prefix", + "middle", + "tail", + "prefixDiffersFromUnbounded", + "middleDiffersFromUnbounded", + "tailDiffersFromUnbounded", + ]); + return { + unbounded: readPlanAggregate(record.unbounded), + prefix: readPlanAggregate(record.prefix), + middle: readPlanAggregate(record.middle), + tail: readPlanAggregate(record.tail), + prefixDiffersFromUnbounded: booleanValue(record.prefixDiffersFromUnbounded), + middleDiffersFromUnbounded: booleanValue(record.middleDiffersFromUnbounded), + tailDiffersFromUnbounded: booleanValue(record.tailDiffersFromUnbounded), + }; +} + +function readPlanAggregate(value: unknown): PlanAggregate { + const record = exactRecord(value, ["rows", "virtualTableScans", "temporaryBtrees"]); + return { + rows: positiveNumber(record.rows), + virtualTableScans: nonNegativeInteger(record.virtualTableScans), + temporaryBtrees: nonNegativeInteger(record.temporaryBtrees), + }; +} + +function readHealthWorkerReport(value: unknown): HealthWorkerReport { + const record = exactRecord(value, ["healthy"]); + return { healthy: trueBoolean(record.healthy) }; +} + +function exactRecord(value: unknown, keys: readonly string[]): Record { + assert( + typeof value === "object" && value !== null && !Array.isArray(value), + "doctor measurement worker report is not an object", + ); + const record = value as Record; + assert.deepStrictEqual( + Object.keys(record).sort(), + [...keys].sort(), + "doctor measurement worker report fields changed", + ); + return record; +} + +function positiveNumber(value: unknown): number { + assert( + typeof value === "number" && Number.isSafeInteger(value) && value > 0, + "doctor measurement positive number is invalid", + ); + return value; +} + +function integerNumber(value: unknown): number { + assert( + typeof value === "number" && Number.isSafeInteger(value), + "doctor measurement integer is invalid", + ); + return value; +} + +function nonNegativeInteger(value: unknown): number { + assert( + typeof value === "number" && Number.isSafeInteger(value) && value >= 0, + "doctor measurement non-negative integer is invalid", + ); + return value; +} + +function nonNegativeFinite(value: unknown): number { + assert( + typeof value === "number" && Number.isFinite(value) && value >= 0, + "doctor measurement elapsed value is invalid", + ); + return value; +} + +function booleanValue(value: unknown): boolean { + assert(typeof value === "boolean", "doctor measurement boolean is invalid"); + return value; +} + +function trueBoolean(value: unknown): true { + assert.equal(value, true, "doctor measurement correctness assertion failed"); + return true; +} + +function requiredStrategy(value: unknown): StrategyName { + assert( + typeof value === "string" && STRATEGIES.includes(value as StrategyName), + "doctor measurement strategy is invalid", + ); + return value as StrategyName; +} + +function nonEmptyString(value: unknown): string { + assert(typeof value === "string" && value.length > 0, "doctor measurement text is invalid"); + return value; +} + +function sqliteInteger(value: unknown): bigint { + assert(typeof value === "bigint", "doctor measurement SQLite integer is invalid"); + return value; +} + +function positiveSqliteInteger(value: unknown): bigint { + const integer = sqliteInteger(value); + assert(integer > 0n, "doctor measurement SQLite integer is not positive"); + return integer; +} + +function safeNumber(value: bigint): number { + const converted = Number(value); + assert(Number.isSafeInteger(converted), "doctor measurement integer exceeds safe range"); + return converted; +} + +function countRows(database: DatabaseSync, sql: string): number { + const row = database.prepare(sql).get() as Record | undefined; + return integerNumber(row?.rows); +} + +function roundMeasurement(value: number): number { + return Number(value.toFixed(6)); +} + +function readOption(name: string): string | undefined { + const indexes = process.argv.flatMap((value, index) => (value === name ? [index] : [])); + assert(indexes.length <= 1, `duplicate ${name} option`); + const index = indexes[0]; + if (index === undefined) return undefined; + const value = process.argv[index + 1]; + assert(value !== undefined && !value.startsWith("--"), `missing ${name} value`); + return value; +} + +function requiredOption(name: string): string { + const value = readOption(name); + assert(value !== undefined, `missing ${name} option`); + return value; +} + +function requiredWorkerMode(): WorkerMode { + const mode = requiredOption("--worker"); + assert( + mode === "seed" || mode === "equality" || mode === "measure" || mode === "health", + "invalid doctor measurement worker mode", + ); + return mode; +} + +function assertControllerArguments(): void { + const allowed = new Set([SCRIPT_PATH, "--", "--contract", "--force-child-failure"]); + for (const value of process.argv.slice(1)) { + assert(allowed.has(value), "unexpected doctor measurement argument"); + } +} + +function assertWorkerArguments(mode: WorkerMode): void { + assert( + mode === "seed" || mode === "equality" || mode === "measure" || mode === "health", + "invalid doctor measurement worker mode", + ); + const valued = new Set([ + "--worker", + "--directory", + "--root", + "--capability", + "--cohort", + "--strategy", + ]); + const flags = new Set(["--contract", "--fail-after-open"]); + for (let index = 2; index < process.argv.length; index += 1) { + const value = process.argv[index]!; + if (flags.has(value)) continue; + assert(valued.has(value), "unexpected doctor measurement worker argument"); + index += 1; + assert(process.argv[index] !== undefined, "doctor measurement worker option value is absent"); + } + assert(path.isAbsolute(requiredOption("--directory")), "worker directory must be absolute"); + assert(path.isAbsolute(requiredOption("--root")), "worker root must be absolute"); + if (mode !== "measure") { + assert(readOption("--strategy") === undefined, "unexpected worker strategy"); + assert(!process.argv.includes("--fail-after-open"), "unexpected worker failpoint"); + } +} + +function pathsAt(directory: string): IndexPaths { + return resolveIndexPaths({ + platform: process.platform, + env: { SESSIONS_DATA_DIR: directory }, + homeDirectory: directory, + }); +} + +async function createOwnedDirectory(directory: string): Promise { + await mkdir(directory, { mode: 0o700, recursive: false }); + await chmod(directory, 0o700); + assert(await hasMode(directory, 0o700), "measurement directory permissions are not private"); +} + +async function createMeasurementRoot(parent: string): Promise { + const rootPath = await mkdtemp(path.join(parent, "sessions-doctor-measure-")); + try { + await chmod(rootPath, 0o700); + assert(await hasMode(rootPath, 0o700), "measurement root permissions are not private"); + return await captureOwnedRoot(rootPath); + } catch (operationError) { + let cleanupError: unknown; + try { + await removeOwnedRoot(await captureOwnedRoot(rootPath)); + } catch (error) { + cleanupError = error; + } + if (cleanupError === undefined) throw operationError; + throw new AggregateError( + [operationError, cleanupError], + "measurement root setup and cleanup both failed", + ); + } +} + +async function createWorkerAuthority(root: string, capability: string): Promise { + assert(/^[a-f0-9]{64}$/u.test(capability), "measurement capability is invalid"); + const marker = path.join(root, AUTHORITY_MARKER); + await writeFile(marker, capability, { encoding: "utf8", flag: "wx", mode: 0o600 }); + await chmod(marker, 0o600); + assert(await hasMode(marker, 0o600), "measurement authority permissions are not private"); +} + +async function assertSeedDirectoryEmpty(directory: string): Promise { + assert.deepStrictEqual( + await readdir(directory), + [], + "doctor measurement seed directory is not empty", + ); +} + +async function assertWorkerAuthority( + root: string, + directory: string, + capability: string, +): Promise { + assert(/^[a-f0-9]{64}$/u.test(capability), "worker capability is invalid"); + assert(path.basename(root).startsWith("sessions-doctor-measure-"), "worker root name is invalid"); + const [resolvedRoot, resolvedDirectory] = await Promise.all([ + realpath(root), + realpath(directory), + ]); + const relative = path.relative(resolvedRoot, resolvedDirectory); + assert( + relative !== "" && !relative.startsWith("..") && !path.isAbsolute(relative), + "worker directory is outside its owned root", + ); + const [rootStats, directoryStats, markerStats, markerContent] = await Promise.all([ + lstat(root, { bigint: true }), + lstat(directory, { bigint: true }), + lstat(path.join(root, AUTHORITY_MARKER), { bigint: true }), + readFile(path.join(root, AUTHORITY_MARKER), "utf8"), + ]); + assert( + rootStats.isDirectory() && + !rootStats.isSymbolicLink() && + directoryStats.isDirectory() && + !directoryStats.isSymbolicLink() && + markerStats.isFile() && + !markerStats.isSymbolicLink() && + rootStats.dev === directoryStats.dev && + rootStats.dev === markerStats.dev, + "worker authority ownership is invalid", + ); + assert.equal(markerContent, capability, "worker capability does not match its authority"); + assert(await hasMode(root, 0o700), "worker root permissions are not private"); + assert(await hasMode(directory, 0o700), "worker directory permissions are not private"); + assert( + await hasMode(path.join(root, AUTHORITY_MARKER), 0o600), + "worker authority permissions are not private", + ); +} + +async function captureOwnedRoot(root: string): Promise { + const stats = await lstat(root, { bigint: true }); + assert( + stats.isDirectory() && !stats.isSymbolicLink(), + "measurement root is not an owned directory", + ); + return { path: root, device: stats.dev, inode: stats.ino }; +} + +async function removeOwnedRoot(root: OwnedRoot): Promise { + const stats = await lstat(root.path, { bigint: true }); + assert( + stats.isDirectory() && + !stats.isSymbolicLink() && + stats.dev === root.device && + stats.ino === root.inode, + "measurement root ownership changed before cleanup", + ); + await rm(root.path, { force: true, recursive: true }); + try { + await lstat(root.path); + assert.fail("measurement root remained after cleanup"); + } catch (error) { + if (isMissing(error)) return; + throw error; + } +} + +async function snapshotFile(file: string): Promise { + const stats = await lstat(file, { bigint: true }); + assert(stats.isFile() && !stats.isSymbolicLink(), "measurement database is not an owned file"); + return { + device: stats.dev, + inode: stats.ino, + mode: stats.mode, + uid: stats.uid, + gid: stats.gid, + links: stats.nlink, + size: stats.size, + modified: stats.mtimeNs, + changed: stats.ctimeNs, + digest: createHash("sha256") + .update(await readFile(file)) + .digest("hex"), + }; +} + +function sameFileSnapshot(left: FileSnapshot, right: FileSnapshot): boolean { + return ( + left.device === right.device && + left.inode === right.inode && + left.mode === right.mode && + left.uid === right.uid && + left.gid === right.gid && + left.links === right.links && + left.size === right.size && + left.modified === right.modified && + left.changed === right.changed && + left.digest === right.digest + ); +} + +async function assertNoSidecars(database: string): Promise { + assert(await sidecarsAreAbsent(database), "measurement database retained a sidecar"); +} + +async function sidecarsAreAbsent(database: string): Promise { + for (const suffix of SIDECAR_SUFFIXES) { + try { + await lstat(`${database}${suffix}`); + return false; + } catch (error) { + if (isMissing(error)) continue; + throw error; + } + } + return true; +} + +async function hasMode(target: string, expected: number): Promise { + if (process.platform === "win32") return false; + const stats = await lstat(target); + return (stats.mode & 0o777) === expected; +} + +function isMissing(error: unknown): boolean { + return ( + typeof error === "object" && + error !== null && + "code" in error && + (error as { readonly code?: unknown }).code === "ENOENT" + ); +} diff --git a/test/doctor-measurement.test.ts b/test/doctor-measurement.test.ts new file mode 100644 index 0000000..b75e110 --- /dev/null +++ b/test/doctor-measurement.test.ts @@ -0,0 +1,570 @@ +import { spawnSync } from "node:child_process"; +import { chmod, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { afterEach, describe, expect, test } from "vitest"; + +const SCRIPT = path.resolve(import.meta.dirname, "../scripts/measure-doctor.ts"); +const FAILURE_MESSAGE = "Doctor feasibility measurement failed\n"; +const PRIVATE_CANARY = "doctor-measurement-private-canary"; +const temporaryDirectories: string[] = []; + +afterEach(async () => { + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { force: true, recursive: true })), + ); +}); + +describe.skipIf(process.platform === "win32")("doctor feasibility measurement", () => { + test("uses generated exact evidence and leaves no persistent or private output", async () => { + const sandbox = await createSandbox(); + const temporaryParent = path.join(sandbox, "temporary"); + await mkdir(temporaryParent, { mode: 0o700 }); + await chmod(temporaryParent, 0o700); + const result = runMeasurement(sandbox, temporaryParent, ["--contract"]); + + expect(result.status).toBe(0); + expect(result.signal).toBeNull(); + expect(result.stderr).toBe(""); + expect(result.stdout.endsWith("\n")).toBe(true); + expect(result.stdout.trimEnd().split("\n")).toHaveLength(1); + expect(result.stdout).not.toContain(PRIVATE_CANARY); + expect(result.stdout).not.toContain("memory://"); + expect(result.stdout).not.toContain("sessions-doctor-measure-"); + expect(result.stdout).not.toContain(sandbox); + + const report = readReport(JSON.parse(result.stdout)); + expect(report).toMatchObject({ + schemaVersion: 1, + command: "measure-doctor", + mode: "contract", + acceptance: "accepted-rejection", + configuration: { + timingRounds: 2, + intervalStrategies: ["one", "two", "many"], + peakRssUnit: "bytes", + generatedOnly: true, + productionDoctorUnchanged: true, + }, + temporaryCleanup: true, + }); + expect(report.cohorts.map((cohort) => cohort.name)).toEqual(["small", "large"]); + expect(report.cohorts.map((cohort) => cohort.corpus.rowIntervalTarget)).toEqual([12, 20]); + expect(report.cohorts.map((cohort) => cohort.strategies[2]?.intervalCount)).toEqual([4, 8]); + for (const cohort of report.cohorts) assertHealthyCohort(cohort, report.configuration); + await expect(readdir(temporaryParent)).resolves.toEqual([]); + }, 60_000); + + test("sanitizes a measured-child failure and removes every owned temporary file", async () => { + const sandbox = await createSandbox(); + const temporaryParent = path.join(sandbox, "temporary"); + await mkdir(temporaryParent, { mode: 0o700 }); + await chmod(temporaryParent, 0o700); + const result = runMeasurement(sandbox, temporaryParent, [ + "--contract", + "--force-child-failure", + ]); + + expect(result.status).toBe(1); + expect(result.signal).toBeNull(); + expect(result.stdout).toBe(""); + expect(result.stderr).toBe(FAILURE_MESSAGE); + expect(result.stderr).not.toContain(PRIVATE_CANARY); + expect(result.stderr).not.toContain("sessions-doctor-measure-"); + expect(result.stderr).not.toContain(sandbox); + await expect(readdir(temporaryParent)).resolves.toEqual([]); + }, 60_000); + + test("rejects a direct seed worker without authority and preserves an existing library", async () => { + const root = await mkdtemp(path.join(tmpdir(), "sessions-doctor-measure-")); + temporaryDirectories.push(root); + await chmod(root, 0o700); + const library = path.join(root, "sentinel-library"); + await mkdir(library, { mode: 0o700 }); + await chmod(library, 0o700); + const database = path.join(library, "sessions.sqlite3"); + const sentinel = Buffer.from("existing-sessions-library"); + await writeFile(database, sentinel, { mode: 0o600 }); + const entriesBefore = await readdir(library); + + const result = runDirectWorker(root, library); + + expect(result.status).toBe(1); + expect(result.signal).toBeNull(); + expect(result.stdout).toBe(""); + expect(result.stderr).toBe(""); + await expect(readdir(library)).resolves.toEqual(entriesBefore); + await expect(readFile(database)).resolves.toEqual(sentinel); + }); +}); + +async function createSandbox(): Promise { + const sandbox = await mkdtemp(path.join(tmpdir(), "sessions-doctor-measurement-test-")); + temporaryDirectories.push(sandbox); + await chmod(sandbox, 0o700); + return sandbox; +} + +function runMeasurement(sandbox: string, temporaryParent: string, arguments_: readonly string[]) { + const isolatedHome = path.join(sandbox, "home"); + return spawnSync(process.execPath, [SCRIPT, ...arguments_], { + cwd: sandbox, + encoding: "utf8", + timeout: 60_000, + env: { + ...process.env, + HOME: isolatedHome, + USERPROFILE: isolatedHome, + CODEX_HOME: path.join(sandbox, "unavailable-codex"), + CODEX_SQLITE_HOME: undefined, + SESSIONS_DATA_DIR: undefined, + SESSIONS_MEASURE_DOCTOR_TEMP_PARENT: temporaryParent, + }, + }); +} + +function runDirectWorker(root: string, library: string) { + return spawnSync( + process.execPath, + [ + SCRIPT, + "--worker", + "seed", + "--directory", + library, + "--root", + root, + "--cohort", + "small", + "--contract", + ], + { + cwd: library, + encoding: "utf8", + timeout: 10_000, + env: { + ...process.env, + HOME: path.join(root, "home"), + USERPROFILE: path.join(root, "home"), + CODEX_HOME: path.join(root, "unavailable-codex"), + CODEX_SQLITE_HOME: undefined, + SESSIONS_DATA_DIR: undefined, + SESSIONS_MEASURE_DOCTOR_TEMP_PARENT: undefined, + }, + }, + ); +} + +interface MeasurementReport { + readonly schemaVersion: number; + readonly command: string; + readonly mode: string; + readonly acceptance: string; + readonly configuration: { + readonly timingRounds: number; + readonly intervalStrategies: readonly string[]; + readonly peakRssUnit: string; + readonly generatedOnly: boolean; + readonly productionDoctorUnchanged: boolean; + }; + readonly cohorts: readonly CohortReport[]; + readonly temporaryCleanup: boolean; +} + +interface CohortReport { + readonly name: string; + readonly corpus: { + readonly sessions: number; + readonly entries: number; + readonly contentRows: number; + readonly contentBytes: number; + readonly instanceRows: number; + readonly repeatedTermTarget: number; + readonly repeatedTermInstances: number; + readonly rowIntervalTarget: number; + readonly byteIntervalTarget: number; + readonly oversizedContentRows: number; + readonly zeroTokenContentRows: number; + readonly databaseBytes: number; + }; + readonly cloneEquality: boolean; + readonly exactQueryEquality: boolean; + readonly intervalAccounting: boolean; + readonly finalHealth: boolean; + readonly persistentFileStateEqual: boolean; + readonly sidecarsAbsentBeforeAndAfter: boolean; + readonly ownedPermissions: boolean; + readonly plans: ProbePlans; + readonly strategies: readonly StrategyReport[]; + readonly scaling: { + readonly twoToOneTotalRatio: number; + readonly manyToOneTotalRatio: number; + readonly manyToOneRawInstancesRatio: number; + readonly manyToOneTermSummariesRatio: number; + readonly manyToOnePeakRssRatio: number; + }; +} + +interface ProbePlans { + readonly rawInstances: PlanShape; + readonly termSummaries: PlanShape; +} + +interface PlanShape { + readonly unbounded: PlanAggregate; + readonly prefix: PlanAggregate; + readonly middle: PlanAggregate; + readonly tail: PlanAggregate; + readonly prefixDiffersFromUnbounded: boolean; + readonly middleDiffersFromUnbounded: boolean; + readonly tailDiffersFromUnbounded: boolean; +} + +interface PlanAggregate { + readonly rows: number; + readonly virtualTableScans: number; + readonly temporaryBtrees: number; +} + +interface StrategyReport { + readonly name: string; + readonly intervalCount: number; + readonly bounds: { + readonly firstLowerExclusive: number; + readonly lastUpperInclusive: number; + readonly minimumIdSpan: number; + readonly maximumIdSpan: number; + }; + readonly queriesPerShape: number; + readonly instanceRows: number; + readonly termSummaryRows: number; + readonly phases: { + readonly rawInstancesMs: NumericAggregate; + readonly termSummariesMs: NumericAggregate; + readonly totalMs: NumericAggregate; + }; + readonly peakRssBytes: { + readonly samples: readonly number[]; + readonly max: number; + }; +} + +interface NumericAggregate { + readonly samples: readonly number[]; + readonly median: number; + readonly p95: number; +} + +function readReport(value: unknown): MeasurementReport { + const report = exactRecord(value, [ + "schemaVersion", + "command", + "mode", + "acceptance", + "configuration", + "cohorts", + "temporaryCleanup", + ]); + const configuration = exactRecord(report.configuration, [ + "timingRounds", + "intervalStrategies", + "peakRssUnit", + "generatedOnly", + "productionDoctorUnchanged", + ]); + expect(Array.isArray(report.cohorts)).toBe(true); + expect(Array.isArray(configuration.intervalStrategies)).toBe(true); + return { + schemaVersion: numberValue(report.schemaVersion), + command: stringValue(report.command), + mode: stringValue(report.mode), + acceptance: stringValue(report.acceptance), + configuration: { + timingRounds: numberValue(configuration.timingRounds), + intervalStrategies: (configuration.intervalStrategies as unknown[]).map(stringValue), + peakRssUnit: stringValue(configuration.peakRssUnit), + generatedOnly: booleanValue(configuration.generatedOnly), + productionDoctorUnchanged: booleanValue(configuration.productionDoctorUnchanged), + }, + cohorts: (report.cohorts as unknown[]).map(readCohort), + temporaryCleanup: booleanValue(report.temporaryCleanup), + }; +} + +function readCohort(value: unknown): CohortReport { + const cohort = exactRecord(value, [ + "name", + "corpus", + "cloneEquality", + "exactQueryEquality", + "intervalAccounting", + "finalHealth", + "persistentFileStateEqual", + "sidecarsAbsentBeforeAndAfter", + "ownedPermissions", + "plans", + "strategies", + "scaling", + ]); + const corpus = exactRecord(cohort.corpus, [ + "sessions", + "entries", + "contentRows", + "contentBytes", + "instanceRows", + "repeatedTermTarget", + "repeatedTermInstances", + "rowIntervalTarget", + "byteIntervalTarget", + "oversizedContentRows", + "zeroTokenContentRows", + "databaseBytes", + ]); + const scaling = exactRecord(cohort.scaling, [ + "twoToOneTotalRatio", + "manyToOneTotalRatio", + "manyToOneRawInstancesRatio", + "manyToOneTermSummariesRatio", + "manyToOnePeakRssRatio", + ]); + expect(Array.isArray(cohort.strategies)).toBe(true); + return { + name: stringValue(cohort.name), + corpus: mapNumbers(corpus) as CohortReport["corpus"], + cloneEquality: booleanValue(cohort.cloneEquality), + exactQueryEquality: booleanValue(cohort.exactQueryEquality), + intervalAccounting: booleanValue(cohort.intervalAccounting), + finalHealth: booleanValue(cohort.finalHealth), + persistentFileStateEqual: booleanValue(cohort.persistentFileStateEqual), + sidecarsAbsentBeforeAndAfter: booleanValue(cohort.sidecarsAbsentBeforeAndAfter), + ownedPermissions: booleanValue(cohort.ownedPermissions), + plans: readPlans(cohort.plans), + strategies: (cohort.strategies as unknown[]).map(readStrategy), + scaling: mapNumbers(scaling) as CohortReport["scaling"], + }; +} + +function readPlans(value: unknown): ProbePlans { + const plans = exactRecord(value, ["rawInstances", "termSummaries"]); + return { + rawInstances: readPlanShape(plans.rawInstances), + termSummaries: readPlanShape(plans.termSummaries), + }; +} + +function readPlanShape(value: unknown): PlanShape { + const shape = exactRecord(value, [ + "unbounded", + "prefix", + "middle", + "tail", + "prefixDiffersFromUnbounded", + "middleDiffersFromUnbounded", + "tailDiffersFromUnbounded", + ]); + return { + unbounded: readPlanAggregate(shape.unbounded), + prefix: readPlanAggregate(shape.prefix), + middle: readPlanAggregate(shape.middle), + tail: readPlanAggregate(shape.tail), + prefixDiffersFromUnbounded: booleanValue(shape.prefixDiffersFromUnbounded), + middleDiffersFromUnbounded: booleanValue(shape.middleDiffersFromUnbounded), + tailDiffersFromUnbounded: booleanValue(shape.tailDiffersFromUnbounded), + }; +} + +function readPlanAggregate(value: unknown): PlanAggregate { + return mapNumbers( + exactRecord(value, ["rows", "virtualTableScans", "temporaryBtrees"]), + ) as unknown as PlanAggregate; +} + +function readStrategy(value: unknown): StrategyReport { + const strategy = exactRecord(value, [ + "name", + "intervalCount", + "bounds", + "queriesPerShape", + "instanceRows", + "termSummaryRows", + "phases", + "peakRssBytes", + ]); + const bounds = exactRecord(strategy.bounds, [ + "firstLowerExclusive", + "lastUpperInclusive", + "minimumIdSpan", + "maximumIdSpan", + ]); + const phases = exactRecord(strategy.phases, ["rawInstancesMs", "termSummariesMs", "totalMs"]); + const rss = exactRecord(strategy.peakRssBytes, ["samples", "max"]); + expect(Array.isArray(rss.samples)).toBe(true); + return { + name: stringValue(strategy.name), + intervalCount: numberValue(strategy.intervalCount), + bounds: mapNumbers(bounds) as StrategyReport["bounds"], + queriesPerShape: numberValue(strategy.queriesPerShape), + instanceRows: numberValue(strategy.instanceRows), + termSummaryRows: numberValue(strategy.termSummaryRows), + phases: { + rawInstancesMs: readNumericAggregate(phases.rawInstancesMs), + termSummariesMs: readNumericAggregate(phases.termSummariesMs), + totalMs: readNumericAggregate(phases.totalMs), + }, + peakRssBytes: { + samples: (rss.samples as unknown[]).map(numberValue), + max: numberValue(rss.max), + }, + }; +} + +function readNumericAggregate(value: unknown): NumericAggregate { + const aggregate = exactRecord(value, ["samples", "median", "p95"]); + expect(Array.isArray(aggregate.samples)).toBe(true); + return { + samples: (aggregate.samples as unknown[]).map(numberValue), + median: numberValue(aggregate.median), + p95: numberValue(aggregate.p95), + }; +} + +function assertHealthyCohort( + cohort: CohortReport, + configuration: MeasurementReport["configuration"], +): void { + for (const count of Object.values(cohort.corpus)) expectPositiveSafeInteger(count); + expect(cohort.corpus.sessions).toBe(1); + expect(cohort.corpus.entries).toBeGreaterThan(0); + expect(cohort.corpus.contentRows).toBeGreaterThan(0); + expect(cohort.corpus.contentBytes).toBeGreaterThan(cohort.corpus.byteIntervalTarget); + expect(cohort.corpus.repeatedTermInstances).toBeGreaterThan(cohort.corpus.repeatedTermTarget); + expect(cohort.corpus.oversizedContentRows).toBe(1); + expect(cohort.corpus.zeroTokenContentRows).toBeGreaterThanOrEqual(2); + expect(cohort.cloneEquality).toBe(true); + expect(cohort.exactQueryEquality).toBe(true); + expect(cohort.intervalAccounting).toBe(true); + expect(cohort.finalHealth).toBe(true); + expect(cohort.persistentFileStateEqual).toBe(true); + expect(cohort.sidecarsAbsentBeforeAndAfter).toBe(true); + expect(cohort.ownedPermissions).toBe(true); + expect(cohort.strategies.map((strategy) => strategy.name)).toEqual(["one", "two", "many"]); + for (const strategy of cohort.strategies) { + for (const count of [ + strategy.intervalCount, + strategy.queriesPerShape, + strategy.instanceRows, + strategy.termSummaryRows, + ]) { + expectPositiveSafeInteger(count); + } + for (const bound of Object.values(strategy.bounds)) { + expect(Number.isSafeInteger(bound)).toBe(true); + } + expect(strategy.bounds.firstLowerExclusive).toBeLessThan(strategy.bounds.lastUpperInclusive); + expect(strategy.bounds.minimumIdSpan).toBeGreaterThan(0); + expect(strategy.bounds.maximumIdSpan).toBeGreaterThanOrEqual(strategy.bounds.minimumIdSpan); + expect(strategy.queriesPerShape).toBe(strategy.intervalCount + 1); + expect(strategy.instanceRows).toBe(cohort.corpus.instanceRows); + expect(strategy.phases.rawInstancesMs.samples).toHaveLength(configuration.timingRounds); + expect(strategy.phases.termSummariesMs.samples).toHaveLength(configuration.timingRounds); + expect(strategy.phases.totalMs.samples).toHaveLength(configuration.timingRounds); + expect(strategy.peakRssBytes.samples).toHaveLength(configuration.timingRounds); + expect(strategy.peakRssBytes.max).toBeGreaterThan(0); + for (const sample of strategy.peakRssBytes.samples) expectPositiveSafeInteger(sample); + for (const sample of [ + ...strategy.phases.rawInstancesMs.samples, + ...strategy.phases.termSummariesMs.samples, + ...strategy.phases.totalMs.samples, + ]) { + expect(Number.isFinite(sample)).toBe(true); + expect(sample).toBeGreaterThanOrEqual(0); + } + assertNumericAggregate(strategy.phases.rawInstancesMs); + assertNumericAggregate(strategy.phases.termSummariesMs); + assertNumericAggregate(strategy.phases.totalMs); + expect(strategy.peakRssBytes.max).toBe(Math.max(...strategy.peakRssBytes.samples)); + } + const [one, two, many] = cohort.strategies; + expect(one).toBeDefined(); + expect(two).toBeDefined(); + expect(many).toBeDefined(); + expect(many!.bounds.maximumIdSpan).toBeLessThanOrEqual(cohort.corpus.rowIntervalTarget); + for (const shape of [cohort.plans.rawInstances, cohort.plans.termSummaries]) { + for (const aggregate of [shape.unbounded, shape.prefix, shape.middle, shape.tail]) { + expectPositiveSafeInteger(aggregate.rows); + expectNonNegativeSafeInteger(aggregate.virtualTableScans); + expectNonNegativeSafeInteger(aggregate.temporaryBtrees); + } + } + expect(cohort.scaling).toEqual({ + twoToOneTotalRatio: roundedRatio(two!.phases.totalMs.median, one!.phases.totalMs.median), + manyToOneTotalRatio: roundedRatio(many!.phases.totalMs.median, one!.phases.totalMs.median), + manyToOneRawInstancesRatio: roundedRatio( + many!.phases.rawInstancesMs.median, + one!.phases.rawInstancesMs.median, + ), + manyToOneTermSummariesRatio: roundedRatio( + many!.phases.termSummariesMs.median, + one!.phases.termSummariesMs.median, + ), + manyToOnePeakRssRatio: roundedRatio(many!.peakRssBytes.max, one!.peakRssBytes.max), + }); +} + +function assertNumericAggregate(aggregate: NumericAggregate): void { + const sorted = [...aggregate.samples].sort((left, right) => left - right); + expect(aggregate.median).toBe(percentile(sorted, 0.5)); + expect(aggregate.p95).toBe(percentile(sorted, 0.95)); +} + +function percentile(sorted: readonly number[], fraction: number): number { + return sorted[Math.min(sorted.length - 1, Math.ceil(sorted.length * fraction) - 1)]!; +} + +function roundedRatio(numerator: number, denominator: number): number { + return Number((numerator / denominator).toFixed(3)); +} + +function expectPositiveSafeInteger(value: number): void { + expect(Number.isSafeInteger(value)).toBe(true); + expect(value).toBeGreaterThan(0); +} + +function expectNonNegativeSafeInteger(value: number): void { + expect(Number.isSafeInteger(value)).toBe(true); + expect(value).toBeGreaterThanOrEqual(0); +} + +function exactRecord(value: unknown, keys: readonly string[]): Record { + expect(typeof value).toBe("object"); + expect(value).not.toBeNull(); + expect(Array.isArray(value)).toBe(false); + const record = value as Record; + expect(Object.keys(record).sort()).toEqual([...keys].sort()); + return record; +} + +function mapNumbers(record: Record): Record { + return Object.fromEntries( + Object.entries(record).map(([key, value]) => [key, numberValue(value)]), + ); +} + +function numberValue(value: unknown): number { + expect(typeof value).toBe("number"); + expect(Number.isFinite(value as number)).toBe(true); + return value as number; +} + +function stringValue(value: unknown): string { + expect(typeof value).toBe("string"); + return value as string; +} + +function booleanValue(value: unknown): boolean { + expect(typeof value).toBe("boolean"); + return value as boolean; +}