diff --git a/README.md b/README.md index 6ceee97..28f96c6 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Two ways in, depending on what you're doing: - *"Spawn live agents to improve a KB"* → pass an `updateKnowledge` callback to `improveKnowledgeBase`; runtime-backed supervisors live in `@tangle-network/agent-runtime`. - *"Tune retrieval for a knowledge base"* → `runRetrievalImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section. - *"Run an operator-grade KB improvement cycle"* → `improveKnowledgeBase` in the [Agent-Eval integration](#agent-eval-integration) section. - It creates a candidate KB, lets agents or deterministic hooks improve it, runs configured evals, can be resumed, and promotes only when the live KB has not changed underneath it. + It creates a candidate KB, lets agents or deterministic hooks improve it, runs configured evals, and returns exact candidate identity for later approval. - *"Expose the lower-level RAG lifecycle phases"* → `runRagKnowledgeImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section. It exposes retrieval tuning, gap diagnosis, knowledge acquisition/update, answer-quality checks, and promotion as one typed lifecycle. - *"Evaluate RAG answers or a wiki/KB"* → `ragAnswerQualityJudge`, `createRagAnswerQualityHook`, and `scoreKnowledgeBaseIndex` in the [Agent-Eval integration](#agent-eval-integration) section. @@ -53,8 +53,8 @@ Storage stays consumer-owned via `KbStore` (`MemoryKbStore`, `FileSystemKbStore` | RAG | Retrieval eval, source-span labels, answer quality checks, missing/stale/noisy-source diagnosis | `runRagKnowledgeImprovementLoop` | | Memory DB | Adapter contract that turns memory hits into source-like evidence and benchmark rows | `/memory` plus `runMemoryAdapterBenchmark` | | From scratch | Empty KB layout, source ingestion, write-block application, indexing, readiness checks | CLI `init` or `runKnowledgeResearchLoop` | -| Existing KB | Candidate workspace, resume state, base-hash conflict check, promotion only after evals pass | `improveKnowledgeBase` | -| Parallel agents | Per-run locks, isolated candidates, `promote: false` handoff, retry with the same `runId` | `improveKnowledgeBase` with runtime workers | +| Existing KB | Candidate workspace, resume state, base-hash conflict check, exact approved promotion | `improveKnowledgeBase` | +| Parallel agents | Per-run locks, isolated candidates, frozen candidate handoff, retry with the same `runId` | `improveKnowledgeBase` with runtime workers | | One-call agent job | Runtime supervisor plus all KB mechanics above | `runKnowledgeImprovementJob` from `@tangle-network/agent-runtime/knowledge` | ## CLI @@ -119,7 +119,7 @@ from `@tangle-network/agent-knowledge`. without this package hardcoding an agent runner. - `improveKnowledgeBase()` wraps that lifecycle with durable candidate state, a per-run lock, resume support, isolated candidate workspaces, KB quality - scoring, and conflict-safe promotion. + scoring, and an exact candidate reference for explicit promotion. Use it when running agents in loops against a real KB rather than only exposing phase hooks. - `evaluateKnowledgeBaseReadiness()` checks one KB root without running an @@ -223,13 +223,14 @@ const result = await runKnowledgeBenchmarkSuite({ respond: async ({ case: testCase }) => { if (testCase.taskKind !== 'retrieval') return { hits: [] } const hits = await retrieveFromYourKb(testCase.query) - return { hits, costUsd: 0.001 } + return { hits } }, }) console.log(result.report.score.mean) ``` +Billable responders must execute paid work through `context.cost.runPaidCall()`; `costUsd` on an artifact is display-only. Use `buildRetrievalBenchmarkCasesFromQrels()` for qrels-backed retrieval datasets. The smoke pack proves that every declared benchmark family is wired through the runner; full BEIR, MTEB, MS MARCO, TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA, KILT, RAGTruth, and FaithBench runs should pass real dataset rows through the same case shapes. Use `KnowledgeAnswerBenchmarkCase` for CRAG/HotpotQA/KILT-style answer checks and RAGTruth/FaithBench-style hallucination checks by encoding required claims, forbidden claims, and expected source IDs. @@ -272,6 +273,7 @@ const kbQuality = scoreKnowledgeBaseIndex(index, { Use `runRagKnowledgeImprovementLoop` when the product question is broader than retrieval: can the system find the gaps, gather or update knowledge, prove generated answers still behave, and decide whether to promote? `agent-knowledge` owns the knowledge/eval contract; the caller supplies the research, coding, connector, and answer-eval hooks. +This lower-level loop returns a promotion recommendation; it never writes to the live knowledge base. ```ts import { runRagKnowledgeImprovementLoop } from '@tangle-network/agent-knowledge' @@ -299,12 +301,17 @@ console.log(result.promotion) ``` Use `improveKnowledgeBase` when a program should own the candidate workspace and promotion mechanics. -The wrapper composes the same RAG lifecycle, but adds resumable state under `.agent-knowledge/improvements//`, a lock lease for parallel operators, and a base-hash check before copying the candidate over the live KB. +The wrapper composes the same RAG lifecycle, but adds resumable state under `.agent-knowledge/improvements//`, a lock lease for parallel operators, and exact candidate bytes that can be approved later. ```ts -import { improveKnowledgeBase } from '@tangle-network/agent-knowledge' +import { + improveKnowledgeBase, + knowledgeImprovementCandidateRef, + promoteKnowledgeCandidate, + withKnowledgeImprovementCandidate, +} from '@tangle-network/agent-knowledge' -const result = await improveKnowledgeBase({ +const staged = await improveKnowledgeBase({ root: './kb', goal: 'Improve support refund-policy knowledge', readinessSpecs, @@ -320,11 +327,24 @@ const result = await improveKnowledgeBase({ requiredPhases: ['knowledge-update', 'retrieval-tuning', 'answer-quality'], }) -console.log(result.promoted, result.candidate?.evaluation) +const candidate = knowledgeImprovementCandidateRef(staged) +console.log(staged.evaluation, candidate) + +await withKnowledgeImprovementCandidate({ root: './kb', candidate }, async (snapshot) => { + await inspectCandidateFiles(snapshot.root, snapshot.evaluation) +}) + +// Call this only after your product records approval for this exact candidate. +const promoted = await promoteKnowledgeCandidate({ root: './kb', candidate }) +console.log(promoted.promoted) ``` -Pass `promote: false` to leave the candidate workspace open for another agent or a human edit. -Calling `improveKnowledgeBase` again with the same `runId` re-evaluates that candidate and promotes it only if the original live KB hash still matches. +`improveKnowledgeBase` stages a measured candidate by default and does not change the live knowledge base. +Calling it again with the same `runId` resumes interrupted work. +`withKnowledgeImprovementCandidate` materializes the measured bytes in an isolated temporary directory for the callback, checks them again afterward, and removes the directory. +`promoteKnowledgeCandidate` applies only the frozen bytes identified by the approved candidate reference, and refuses if the live base changed. +The current release intentionally accepts only its strict run-state format; incomplete runs created by 1.x must be completed or restarted before upgrading. +The exact candidate workflow requires Linux; other knowledge, retrieval, and evaluation APIs remain cross-platform. If a required phase is missing its hook, the loop throws. That keeps the public API from reporting a fake “RAG improved” result when the caller only wired retrieval or only wired a researcher. @@ -558,7 +578,7 @@ await runTwoAgentResearchLoop({ `agent-knowledge` owns knowledge state and measurement. It deliberately does not own an agent runner. Live agent orchestration belongs in `@tangle-network/agent-runtime`. -Use the one-call runtime job when you want agents, candidate workspaces, readiness checks, promotion, and spend measurement wired together: +Use the one-call runtime job when you want agents, candidate workspaces, readiness checks, exact candidate handoff, and spend measurement wired together: ```ts import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' diff --git a/package.json b/package.json index ac0b263..bbc7b23 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tangle-network/agent-knowledge", - "version": "1.12.1", + "version": "2.0.0", "description": "Source-grounded, eval-gated knowledge growth primitives for agents.", "homepage": "https://github.com/tangle-network/agent-knowledge#readme", "repository": { @@ -68,7 +68,8 @@ "format": "biome format --write src tests" }, "dependencies": { - "@tangle-network/agent-eval": "^0.115.1", + "@tangle-network/agent-eval": "^0.118.3", + "proper-lockfile": "4.1.2", "zod": "^4.3.6" }, "devDependencies": { @@ -76,6 +77,7 @@ "@neo4j-labs/agent-memory": "0.4.0", "@tangle-network/sandbox": "^0.9.7", "@types/node": "^25.6.0", + "@types/proper-lockfile": "4.1.4", "tsup": "^8.0.0", "tsx": "^4.22.4", "typescript": "^5.7.0", @@ -86,10 +88,14 @@ "minimumReleaseAgeExclude": [ "@tangle-network/agent-eval", "@tangle-network/sandbox" - ] + ], + "overrides": { + "hono": "4.12.30", + "ws": "8.21.0" + } }, "engines": { - "node": ">=20" + "node": ">=20.19.0" }, "license": "MIT", "packageManager": "pnpm@10.28.0" diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7a45888..66f1ff5 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -4,13 +4,20 @@ settings: autoInstallPeers: true excludeLinksFromLockfile: false +overrides: + hono: 4.12.30 + ws: 8.21.0 + importers: .: dependencies: '@tangle-network/agent-eval': - specifier: ^0.115.1 - version: 0.115.1(typescript@5.9.3) + specifier: ^0.118.3 + version: 0.118.3(typescript@5.9.3) + proper-lockfile: + specifier: 4.1.2 + version: 4.1.2 zod: specifier: ^4.3.6 version: 4.4.2 @@ -27,6 +34,9 @@ importers: '@types/node': specifier: ^25.6.0 version: 25.6.0 + '@types/proper-lockfile': + specifier: 4.1.4 + version: 4.1.4 tsup: specifier: ^8.0.0 version: 8.5.1(postcss@8.5.13)(tsx@4.22.4)(typescript@5.9.3)(yaml@2.8.4) @@ -428,7 +438,7 @@ packages: resolution: {integrity: sha512-jI9yMDyFpqBeSighf/zlXnQG/nl9AyBc6aAgy4XtxJMyt/CNyJpvPfzDD+bCc2zAOmhhqtF6TnmIaY+xV4mIrw==} engines: {node: '>=20'} peerDependencies: - hono: ^4 + hono: 4.12.30 '@jridgewell/gen-mapping@0.3.13': resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==} @@ -617,8 +627,11 @@ packages: '@tangle-network/agent-core@0.3.8': resolution: {integrity: sha512-bZfVpdiFjXbcQwSxSQABdtXEmUuapWChCcXrHkP6fAnwqEy9hWEfeZLlMZUy+1XaOfpTaMLUOEAwYhXfN018wQ==} - '@tangle-network/agent-eval@0.115.1': - resolution: {integrity: sha512-Yd487u8v0c8/+On9kaYGASS88oLhOJ5p0zwRMGdbWRSWC1UODbIgk1Fqdyq6EI60v+lZ+yDg/7LHkIrNi4kjPA==} + '@tangle-network/agent-core@0.4.11': + resolution: {integrity: sha512-5B1IjrJ8xDR7w8Hv/MSk2ixul6NEJQ5Ftzo+z6l7ipeFpc0yaf1+Kkml4HDUEmGW/rqdrNpBf9/MpT7i/SY0PA==} + + '@tangle-network/agent-eval@0.118.3': + resolution: {integrity: sha512-+HQaUHlvT5SH5jCcj7CgOfxnbOv4bqcOucE8r7ZHdo+JgaE9mRbXUTaaYe5wPSqHDUZb5SVtks70mz9olT5Lzg==} engines: {node: '>=20'} hasBin: true @@ -628,8 +641,8 @@ packages: '@tangle-network/agent-interface@0.17.1': resolution: {integrity: sha512-B7dRJTo0HSUtgBCB1VMwkTFYkLUaRr/4BcRglrQuGhGUwOzKv1RYyMejOVh5M3a5AagY9N79f7GYbjcA3UmnIA==} - '@tangle-network/agent-interface@0.22.0': - resolution: {integrity: sha512-7fsJhNdvTmOB1X9E2owl06jzyrqaN+jMkOPVKbK7dvNqQvf627PowCtL/edhUqEEv+K0FWtkwG3R3xqjX7VNZg==} + '@tangle-network/agent-interface@0.26.0': + resolution: {integrity: sha512-/z4HavFr/9AbaHxi/13bFP9iSUt8oitZbw4wS9g9KAt2TeN2ec5/A2C106udWkKIETZLnu465TfU+K4KDVNkKw==} '@tangle-network/sandbox@0.9.7': resolution: {integrity: sha512-9pCwJ5MlF7RUpp0AQKQDFyR0yu+E0udEhWkqhrlb/RuoJxlt72zVPuzO4FnMb1MZTkfjStmomC3k5xQyqi1YSA==} @@ -672,6 +685,12 @@ packages: '@types/node@25.6.0': resolution: {integrity: sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ==} + '@types/proper-lockfile@4.1.4': + resolution: {integrity: sha512-uo2ABllncSqg9F1D4nugVl9v93RmjxF6LJzQLMLDdPaXCUIDPeOJ21Gbqi43xNKzBi/WQ0Q0dICqufzQbMjipQ==} + + '@types/retry@0.12.5': + resolution: {integrity: sha512-3xSjTp3v03X/lSQLkczaN9UIEwJMoMCA1+Nb5HfbJEQWogdeQIyVtTvxPXDQjZ5zws8rFQfVfRdz03ARihPJgw==} + '@vitest/expect@3.2.4': resolution: {integrity: sha512-Io0yyORnB6sikFlt8QW5K7slY4OjqNX9jmJQ02QDda8lyM6B5oNgVWoSoKPac8/kgnCUzuHQKrSLtu/uOqqrig==} @@ -817,14 +836,17 @@ packages: engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} os: [darwin] - hono@4.12.16: - resolution: {integrity: sha512-jN0ZewiNAWSe5khM3EyCmBb250+b40wWbwNILNfEvq84VREWwOIkuUsFONk/3i3nqkz7Oe1PcpM2mwQEK2L9Kg==} + graceful-fs@4.2.11: + resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==} + + hono@4.12.30: + resolution: {integrity: sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog==} engines: {node: '>=16.9.0'} isows@1.0.7: resolution: {integrity: sha512-I1fSfDCZL5P0v33sVqeTDSpcstAg/N+wF5HS033mogOVIp4B+oHC7oOCsA3axAbBSGTJ8QubbNmnIRN/h8U7hg==} peerDependencies: - ws: '*' + ws: 8.21.0 joycon@3.1.1: resolution: {integrity: sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw==} @@ -922,6 +944,9 @@ packages: resolution: {integrity: sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==} engines: {node: ^10 || ^12 || >=14} + proper-lockfile@4.1.2: + resolution: {integrity: sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA==} + readdirp@4.1.2: resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} engines: {node: '>= 14.18.0'} @@ -930,6 +955,10 @@ packages: resolution: {integrity: sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==} engines: {node: '>=8'} + retry@0.12.0: + resolution: {integrity: sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==} + engines: {node: '>= 4'} + rollup@4.60.2: resolution: {integrity: sha512-J9qZyW++QK/09NyN/zeO0dG/1GdGfyp9lV8ajHnRVLfo/uFsbji5mHnDgn/qYdUHyCkM2N+8VyspgZclfAh0eQ==} engines: {node: '>=18.0.0', npm: '>=8.0.0'} @@ -938,6 +967,9 @@ packages: siginfo@2.0.0: resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==} + signal-exit@3.0.7: + resolution: {integrity: sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==} + source-map-js@1.2.1: resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} engines: {node: '>=0.10.0'} @@ -1117,8 +1149,8 @@ packages: engines: {node: '>=8'} hasBin: true - ws@8.18.3: - resolution: {integrity: sha512-PEIGCY5tSlUt50cqyMXfCzX+oOPqN0vuGqWzbcJ2xvnkzkq46oOpz7dQaTDBdfICb4N14+GARUDw2XV2N4tvzg==} + ws@8.21.0: + resolution: {integrity: sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==} engines: {node: '>=10.0.0'} peerDependencies: bufferutil: ^4.0.1 @@ -1347,9 +1379,9 @@ snapshots: '@esbuild/win32-x64@0.28.1': optional: true - '@hono/node-server@2.0.1(hono@4.12.16)': + '@hono/node-server@2.0.1(hono@4.12.30)': dependencies: - hono: 4.12.16 + hono: 4.12.30 '@jridgewell/gen-mapping@0.3.13': dependencies: @@ -1489,14 +1521,20 @@ snapshots: '@tangle-network/agent-interface': 0.17.1 zod: 4.4.3 - '@tangle-network/agent-eval@0.115.1(typescript@5.9.3)': + '@tangle-network/agent-core@0.4.11': + dependencies: + '@tangle-network/agent-interface': 0.26.0 + zod: 4.4.3 + + '@tangle-network/agent-eval@0.118.3(typescript@5.9.3)': dependencies: '@asteasolutions/zod-to-openapi': 8.5.0(zod@4.4.3) '@ax-llm/ax': 19.0.45(zod@4.4.3) - '@hono/node-server': 2.0.1(hono@4.12.16) - '@tangle-network/agent-interface': 0.22.0 + '@hono/node-server': 2.0.1(hono@4.12.30) + '@tangle-network/agent-core': 0.4.11 + '@tangle-network/agent-interface': 0.26.0 '@tangle-network/tcloud': 0.4.14(typescript@5.9.3)(zod@4.4.3) - hono: 4.12.16 + hono: 4.12.30 zod: 4.4.3 transitivePeerDependencies: - '@mastra/core' @@ -1515,8 +1553,9 @@ snapshots: dependencies: zod: 4.4.3 - '@tangle-network/agent-interface@0.22.0': + '@tangle-network/agent-interface@0.26.0': dependencies: + '@noble/hashes': 1.8.0 zod: 4.4.3 '@tangle-network/sandbox@0.9.7(viem@2.48.8(typescript@5.9.3)(zod@4.4.2))': @@ -1559,6 +1598,12 @@ snapshots: dependencies: undici-types: 7.19.2 + '@types/proper-lockfile@4.1.4': + dependencies: + '@types/retry': 0.12.5 + + '@types/retry@0.12.5': {} + '@vitest/expect@3.2.4': dependencies: '@types/chai': 5.2.3 @@ -1730,11 +1775,13 @@ snapshots: fsevents@2.3.3: optional: true - hono@4.12.16: {} + graceful-fs@4.2.11: {} - isows@1.0.7(ws@8.18.3): + hono@4.12.30: {} + + isows@1.0.7(ws@8.21.0): dependencies: - ws: 8.18.3 + ws: 8.21.0 joycon@3.1.1: {} @@ -1820,10 +1867,18 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + proper-lockfile@4.1.2: + dependencies: + graceful-fs: 4.2.11 + retry: 0.12.0 + signal-exit: 3.0.7 + readdirp@4.1.2: {} resolve-from@5.0.0: {} + retry@0.12.0: {} + rollup@4.60.2: dependencies: '@types/estree': 1.0.8 @@ -1857,6 +1912,8 @@ snapshots: siginfo@2.0.0: {} + signal-exit@3.0.7: {} + source-map-js@1.2.1: {} source-map@0.7.6: {} @@ -1953,9 +2010,9 @@ snapshots: '@scure/bip32': 1.7.0 '@scure/bip39': 1.6.0 abitype: 1.2.3(typescript@5.9.3)(zod@4.4.3) - isows: 1.0.7(ws@8.18.3) + isows: 1.0.7(ws@8.21.0) ox: 0.14.20(typescript@5.9.3)(zod@4.4.3) - ws: 8.18.3 + ws: 8.21.0 optionalDependencies: typescript: 5.9.3 transitivePeerDependencies: @@ -2044,7 +2101,7 @@ snapshots: siginfo: 2.0.0 stackback: 0.0.2 - ws@8.18.3: {} + ws@8.21.0: {} yaml@2.8.4: {} diff --git a/src/benchmarks/index.ts b/src/benchmarks/index.ts index 2696dbf..c64832e 100644 --- a/src/benchmarks/index.ts +++ b/src/benchmarks/index.ts @@ -160,6 +160,7 @@ export interface KnowledgeBenchmarkArtifact { citedEventIds?: readonly string[] usedMemoryIds?: readonly string[] actorIds?: readonly string[] + /** Informational copy. Billable responders account through context.cost.runPaidCall. */ costUsd?: number durationMs?: number metadata?: Record @@ -576,7 +577,6 @@ export function respondToIndustryRagBenchmarkSmokeCase(input: { const hit = hitForExpectedTarget(expected, testCase.id) return { hits: [hit], - costUsd: 0.001, durationMs: 1, metadata: { smoke: true, @@ -590,7 +590,6 @@ export function respondToIndustryRagBenchmarkSmokeCase(input: { .filter((fragment): fragment is string => Boolean(fragment)) .join(' '), citedSourceIds: testCase.expectedSourceIds ?? [], - costUsd: 0.001, durationMs: 1, metadata: { smoke: true, @@ -674,7 +673,6 @@ export function respondToIndustryMemoryBenchmarkSmokeCase(input: { rememberedFacts: facts ?? [], citedEventIds: testCase.expectedEventIds ?? [], actorIds: testCase.expectedActorIds ?? [], - costUsd: 0.001, durationMs: 1, metadata: { smoke: true, @@ -847,54 +845,80 @@ export function createMemoryAdapterBenchmarkResponder(options: { costUsdPerCase?: number now?: () => Date }): KnowledgeBenchmarkResponder { - return async ({ case: testCase }) => { + return async ({ case: testCase, context: dispatchContext }) => { if (!isKnowledgeMemoryBenchmarkCase(testCase)) { return { answer: '', metadata: { candidateId: options.candidateId, skipped: true } } } - const startedAt = Date.now() - const scope = benchmarkMemoryScope(options.candidateId, testCase, options.scope) - for (const event of testCase.events) { - await options.adapter.write({ - id: event.id, - kind: 'message', - text: event.text, - role: event.actorId === 'user' ? 'user' : 'assistant', - title: `${testCase.id}:${event.id}`, + const costUsd = options.costUsdPerCase ?? 0 + if (!Number.isFinite(costUsd) || costUsd < 0) { + throw new Error(`memory adapter costUsdPerCase must be non-negative finite, got ${costUsd}`) + } + + const execute = async (): Promise => { + const startedAt = Date.now() + const scope = benchmarkMemoryScope(options.candidateId, testCase, options.scope) + for (const event of testCase.events) { + await options.adapter.write({ + id: event.id, + kind: 'message', + text: event.text, + role: event.actorId === 'user' ? 'user' : 'assistant', + title: `${testCase.id}:${event.id}`, + scope, + metadata: compactObject({ + benchmarkCaseId: testCase.id, + eventId: event.id, + actorId: event.actorId, + sessionId: event.sessionId, + timestamp: event.timestamp, + ...event.metadata, + }) as Record, + }) + } + + const adapterContext = await options.adapter.getContext(testCase.prompt, { scope, - metadata: compactObject({ + limit: options.searchLimit ?? 1, + metadata: { benchmarkCaseId: testCase.id, - eventId: event.id, - actorId: event.actorId, - sessionId: event.sessionId, - timestamp: event.timestamp, - ...event.metadata, - }) as Record, + candidateId: options.candidateId, + }, }) + const hits = adapterContext.hits + return { + answer: adapterContext.text, + rememberedFacts: hits.map((hit) => hit.text), + citedEventIds: unique(hits.map(memoryEventId).filter((id): id is string => Boolean(id))), + usedMemoryIds: hits.map((hit) => hit.id), + actorIds: unique(hits.map(memoryActorId).filter((id): id is string => Boolean(id))), + costUsd, + durationMs: Math.max(0, Date.now() - startedAt), + metadata: { + candidateId: options.candidateId, + adapterId: options.adapter.id, + hitCount: hits.length, + }, + } } - const context = await options.adapter.getContext(testCase.prompt, { - scope, - limit: options.searchLimit ?? 1, - metadata: { - benchmarkCaseId: testCase.id, - candidateId: options.candidateId, - }, + if (costUsd === 0) return execute() + const receipt = { + model: options.adapter.id, + inputTokens: 0, + outputTokens: 0, + usageUnknown: true, + actualCostUsd: costUsd, + } as const + const paid = await dispatchContext.cost.runPaidCall({ + actor: `agent-knowledge:memory-adapter:${options.adapter.id}`, + model: options.adapter.id, + maximumCharge: { externallyEnforcedMaximumUsd: costUsd }, + execute, + receipt: () => receipt, + receiptFromError: () => receipt, }) - const hits = context.hits - return { - answer: context.text, - rememberedFacts: hits.map((hit) => hit.text), - citedEventIds: unique(hits.map(memoryEventId).filter((id): id is string => Boolean(id))), - usedMemoryIds: hits.map((hit) => hit.id), - actorIds: unique(hits.map(memoryActorId).filter((id): id is string => Boolean(id))), - costUsd: options.costUsdPerCase ?? 0, - durationMs: Math.max(0, Date.now() - startedAt), - metadata: { - candidateId: options.candidateId, - adapterId: options.adapter.id, - hitCount: hits.length, - }, - } + if (!paid.succeeded) throw paid.error + return paid.value } } @@ -1147,7 +1171,6 @@ export async function runKnowledgeBenchmarkSuite { const artifact = await options.respond({ case: scenario.case, scenario, context }) - observeArtifactCost(context, artifact) return artifact } const campaign = await runCampaign({ @@ -1831,17 +1854,6 @@ function formatNumber(value: number): string { return value.toFixed(value === 0 || Math.abs(value) >= 10 ? 0 : 3) } -function observeArtifactCost(context: DispatchContext, artifact: unknown): void { - const costUsd = (artifact as { costUsd?: unknown })?.costUsd - if (costUsd === undefined) return - if (typeof costUsd !== 'number' || !Number.isFinite(costUsd) || costUsd < 0) { - throw new Error( - `benchmark artifact costUsd must be non-negative finite, got ${String(costUsd)}`, - ) - } - context.cost.observe(costUsd, 'agent-knowledge:benchmark') -} - function compactObject(value: unknown): unknown { if (Array.isArray(value)) return value.map(compactObject) if (!value || typeof value !== 'object') return value diff --git a/src/durable-fs.ts b/src/durable-fs.ts new file mode 100644 index 0000000..16a4804 --- /dev/null +++ b/src/durable-fs.ts @@ -0,0 +1,332 @@ +import { randomUUID } from 'node:crypto' +import { constants } from 'node:fs' +import { + type FileHandle, + lstat, + mkdir, + open, + readdir, + realpath, + rename, + rm, +} from 'node:fs/promises' +import { basename, dirname, isAbsolute, resolve } from 'node:path' + +const DIRECTORY_FLAGS = constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW +const CREATE_FILE_FLAGS = + constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW +const READ_FILE_FLAGS = constants.O_RDONLY | constants.O_NOFOLLOW + +export interface RegularFileSnapshot { + bytes: Buffer + mode: number +} + +export async function writeFileDurable( + path: string, + data: string | Buffer, + options: { encoding?: BufferEncoding; mode?: number } = {}, +): Promise { + const directory = dirname(path) + await mkdir(directory, { recursive: true }) + await withDirectoryHandle(directory, async (handle, anchoredDirectory) => { + const target = resolve(anchoredDirectory, basename(path)) + const existingMode = await regularFileModeOrUndefined(target) + const requestedMode = options.mode ?? existingMode + const temporary = resolve(anchoredDirectory, `.${basename(path)}.${randomUUID()}.tmp`) + let temporaryHandle: FileHandle | undefined + try { + temporaryHandle = await open(temporary, CREATE_FILE_FLAGS, requestedMode ?? 0o666) + await temporaryHandle.writeFile( + data, + options.encoding === undefined ? undefined : { encoding: options.encoding }, + ) + if (requestedMode !== undefined) await temporaryHandle.chmod(requestedMode) + await temporaryHandle.sync() + await temporaryHandle.close() + temporaryHandle = undefined + await rename(temporary, target) + await syncDirectoryHandle(handle) + } finally { + await temporaryHandle?.close().catch(() => undefined) + await rm(temporary, { force: true }).catch(() => undefined) + } + }) +} + +export async function writeJsonDurable(path: string, value: unknown): Promise { + await writeFileDurable(path, `${JSON.stringify(value, null, 2)}\n`, { encoding: 'utf8' }) +} + +export async function writeFileDurableWithinRoot( + root: string, + relativePath: string, + data: string | Buffer, + options: { encoding?: BufferEncoding; mode?: number } = {}, +): Promise { + await withSafeDescendant(root, relativePath, (path) => writeFileDurable(path, data, options)) +} + +export async function writeJsonDurableWithinRoot( + root: string, + relativePath: string, + value: unknown, +): Promise { + await writeFileDurableWithinRoot(root, relativePath, `${JSON.stringify(value, null, 2)}\n`, { + encoding: 'utf8', + }) +} + +export async function renameDurable(source: string, target: string): Promise { + const sourceDir = dirname(source) + const targetDir = dirname(target) + await withDirectoryHandle(sourceDir, async (sourceHandle, anchoredSourceDir) => { + await withDirectoryHandle(targetDir, async (targetHandle, anchoredTargetDir) => { + await rename( + resolve(anchoredSourceDir, basename(source)), + resolve(anchoredTargetDir, basename(target)), + ) + await syncDirectoryHandle(sourceHandle) + if (targetDir !== sourceDir) await syncDirectoryHandle(targetHandle) + }) + }) +} + +export async function removeDurable(path: string): Promise { + await withDirectoryHandle(dirname(path), async (handle, anchoredDirectory) => { + await rm(resolve(anchoredDirectory, basename(path)), { force: true }) + await syncDirectoryHandle(handle) + }) +} + +export async function syncDirectory(path: string): Promise { + await withDirectoryHandle(path, syncDirectoryHandle) +} + +/** + * Keeps the target's parent directory open while the operation runs. On Linux, + * `/proc/self/fd` anchors every lookup to that directory even if an ancestor is + * renamed concurrently. + */ +export async function withSafeDescendant( + root: string, + relativePath: string, + use: (path: string) => Promise | T, +): Promise { + const normalized = normalizeRelativePath(relativePath) + const parts = normalized.split('/') + const filename = parts.pop()! + const directory = await openSafeDirectoryTree(root, parts.join('/'), true) + try { + const target = resolve(anchoredDirectoryPath(directory.handle, directory.path), filename) + await regularFileModeOrUndefined(target) + return await use(target) + } finally { + await directory.handle.close() + } +} + +export async function withSafeDirectory( + root: string, + relativePath: string, + create: boolean, + use: (path: string) => Promise | T, +): Promise { + const directory = await openSafeDirectoryTree(root, relativePath, create) + try { + return await use(anchoredDirectoryPath(directory.handle, directory.path)) + } finally { + await directory.handle.close() + } +} + +export async function readRegularFileNoFollow(path: string): Promise { + let handle: FileHandle | undefined + try { + handle = await open(path, READ_FILE_FLAGS) + const entry = await handle.stat() + if (!entry.isFile()) throw new Error(`knowledge path is not a regular file: ${path}`) + return { bytes: await handle.readFile(), mode: entry.mode & 0o777 } + } catch (error) { + const code = (error as NodeJS.ErrnoException | null)?.code + if (code === 'ELOOP' || code === 'ENOTDIR') { + throw new Error(`knowledge path is not a regular file: ${path}`, { cause: error }) + } + throw error + } finally { + await handle?.close().catch(() => undefined) + } +} + +export async function readRegularFileWithinRoot( + root: string, + relativePath: string, +): Promise { + const normalized = normalizeRelativePath(relativePath) + const parts = normalized.split('/') + const filename = parts.pop()! + const directory = await openSafeDirectoryTree(root, parts.join('/'), false) + try { + return await readRegularFileNoFollow( + resolve(anchoredDirectoryPath(directory.handle, directory.path), filename), + ) + } finally { + await directory.handle.close() + } +} + +export async function listRegularFilesWithinRoot( + root: string, + relativeDirectory: string, +): Promise> { + const normalized = normalizeRelativePath(relativeDirectory) + return withSafeDirectory(root, normalized, false, (directory) => + listRegularFilesFromOpenDirectory(directory, normalized), + ) +} + +export function isMissingFile(error: unknown): boolean { + return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT' +} + +async function withDirectoryHandle( + path: string, + use: (handle: FileHandle, anchoredPath: string) => Promise | T, +): Promise { + const handle = await openDirectory(path) + try { + return await use(handle, anchoredDirectoryPath(handle, path)) + } finally { + await handle.close() + } +} + +async function listRegularFilesFromOpenDirectory( + directory: string, + relativeDirectory: string, +): Promise> { + const out: Array = [] + for (const entry of await readdir(directory, { withFileTypes: true })) { + const relativePath = `${relativeDirectory}/${entry.name}` + if (entry.isDirectory()) { + out.push( + ...(await withSafeDirectory(directory, entry.name, false, (child) => + listRegularFilesFromOpenDirectory(child, relativePath), + )), + ) + } else if (entry.isFile()) { + out.push({ + path: relativePath, + ...(await readRegularFileNoFollow(resolve(directory, entry.name))), + }) + } else { + throw new Error(`knowledge tree contains an unsupported filesystem entry: ${relativePath}`) + } + } + return out +} + +async function openSafeDirectoryTree( + root: string, + relativePath: string, + create: boolean, +): Promise<{ handle: FileHandle; path: string }> { + if (create) await mkdir(root, { recursive: true }) + const resolvedRoot = isKernelAnchoredPath(root) ? root : await realpath(root) + const normalized = relativePath === '' ? '' : normalizeRelativePath(relativePath) + let currentPath = resolvedRoot + let currentHandle = await openDirectory(resolvedRoot) + try { + for (const part of normalized.split('/').filter(Boolean)) { + const anchoredParent = anchoredDirectoryPath(currentHandle, currentPath) + const childPath = resolve(anchoredParent, part) + let childHandle: FileHandle + try { + childHandle = await openDirectory(childPath) + } catch (error) { + if (!create || !isMissingFile(error)) throw unsafeDirectoryError(childPath, error) + let created = false + try { + await mkdir(childPath) + created = true + } catch (mkdirError) { + if ((mkdirError as NodeJS.ErrnoException | null)?.code !== 'EEXIST') throw mkdirError + } + childHandle = await openDirectory(childPath).catch((openError: unknown) => { + throw unsafeDirectoryError(childPath, openError) + }) + if (created) await syncDirectoryHandle(currentHandle) + } + await currentHandle.close() + currentHandle = childHandle + currentPath = resolve(currentPath, part) + } + return { handle: currentHandle, path: currentPath } + } catch (error) { + await currentHandle.close().catch(() => undefined) + throw error + } +} + +async function openDirectory(path: string): Promise { + const isKernelDescriptor = /^\/proc\/self\/fd\/\d+$/.test(path) + if (!isKernelDescriptor) { + const entry = await lstat(path) + if (entry.isSymbolicLink() || !entry.isDirectory()) { + throw new Error(`knowledge path has an unsafe directory: ${path}`) + } + } + const flags = isKernelDescriptor ? constants.O_RDONLY | constants.O_DIRECTORY : DIRECTORY_FLAGS + const handle = await open(path, flags) + const entry = await handle.stat() + if (!entry.isDirectory()) { + await handle.close() + throw new Error(`knowledge path has an unsafe directory: ${path}`) + } + return handle +} + +export function isKernelAnchoredPath(path: string): boolean { + return process.platform === 'linux' && /^\/proc\/self\/fd\/\d+(?:\/|$)/.test(path) +} + +function anchoredDirectoryPath(handle: FileHandle, fallback: string): string { + return process.platform === 'linux' ? `/proc/self/fd/${handle.fd}` : fallback +} + +async function regularFileModeOrUndefined(path: string): Promise { + try { + const entry = await lstat(path) + if (entry.isSymbolicLink() || !entry.isFile()) { + throw new Error(`knowledge path is not a regular file: ${path}`) + } + return entry.mode & 0o777 + } catch (error) { + if (isMissingFile(error)) return undefined + throw error + } +} + +async function syncDirectoryHandle(handle: FileHandle): Promise { + if (process.platform === 'win32') return + await handle.sync() +} + +function unsafeDirectoryError(path: string, cause: unknown): Error { + const code = (cause as NodeJS.ErrnoException | null)?.code + if (code === 'ELOOP' || code === 'ENOTDIR') { + return new Error(`knowledge path has an unsafe directory: ${path}`, { cause }) + } + return cause instanceof Error ? cause : new Error(`could not open knowledge directory: ${path}`) +} + +function normalizeRelativePath(path: string): string { + if (path.length === 0 || isAbsolute(path)) { + throw new Error(`knowledge path must be a non-empty relative path: ${path}`) + } + const normalized = path.replace(/\\/g, '/') + if (normalized.split('/').some((part) => part === '' || part === '.' || part === '..')) { + throw new Error(`knowledge path contains an unsafe segment: ${path}`) + } + return normalized +} diff --git a/src/file-transaction.ts b/src/file-transaction.ts new file mode 100644 index 0000000..361440c --- /dev/null +++ b/src/file-transaction.ts @@ -0,0 +1,690 @@ +import { createHash, randomUUID } from 'node:crypto' +import { mkdtemp, readdir, rm } from 'node:fs/promises' +import { join, relative, resolve, sep } from 'node:path' +import { contentHash } from '@tangle-network/agent-eval' +import { z } from 'zod' +import { + isKernelAnchoredPath, + isMissingFile, + readRegularFileNoFollow, + removeDurable, + renameDurable, + syncDirectory, + withSafeDescendant, + withSafeDirectory, + writeFileDurable, + writeJsonDurable, +} from './durable-fs' + +const digestSchema = z.string().regex(/^[a-f0-9]{64}$/) +const fileModeSchema = z.number().int().min(0).max(0o777) +const DEFAULT_FILE_MODE = 0o644 +const transactionEntrySchema = z + .object({ + index: z.number().int().nonnegative(), + path: z.string().min(1), + beforeHash: digestSchema.nullable(), + afterHash: digestSchema.nullable(), + beforeMode: fileModeSchema.optional(), + afterMode: fileModeSchema.optional(), + }) + .strict() +const transactionSchema = z + .object({ + schemaVersion: z.literal(1), + kind: z.literal('knowledge-file-transaction'), + transactionId: z.string().uuid(), + purpose: z.string().min(1), + createdAt: z.string().min(1), + entries: z.array(transactionEntrySchema).min(1), + }) + .strict() +const transactionDirectionSchema = z + .object({ + schemaVersion: z.literal(1), + transactionId: z.string().uuid(), + direction: z.literal('rollback'), + }) + .strict() + +export type KnowledgeFileTransaction = z.infer + +export interface KnowledgeFileTransactionState { + transaction: KnowledgeFileTransaction + direction: 'apply' | 'rollback' +} + +export interface KnowledgeFileTransactionPlanEntry { + path: string + beforeHash: string | null + afterHash: string | null + beforeMode?: number + afterMode?: number +} + +export function knowledgeFileTransactionPlanHash( + entries: readonly KnowledgeFileTransactionPlanEntry[], +): string { + const normalized = entries + .map((entry) => normalizePlanEntry(entry)) + .sort((left, right) => left.path.localeCompare(right.path)) + if (new Set(normalized.map((entry) => entry.path)).size !== normalized.length) { + throw new Error('knowledge transaction plan repeats a path') + } + return contentHash(normalized) +} + +export interface KnowledgeFileMutation { + path: string + content: string | Buffer | null + mode?: number +} + +export async function prepareKnowledgeFileTransaction(input: { + root: string + transactionRoot: string + purpose: string + mutations: readonly KnowledgeFileMutation[] + includeUnchanged?: boolean + now?: () => Date +}): Promise { + if (input.mutations.length === 0) throw new Error('knowledge file transaction has no mutations') + return withTransactionRoot(input.root, input.transactionRoot, true, async (transactionRoot) => { + const activeTransactions = await activeTransactionDirectoryNames(transactionRoot) + if (activeTransactions.length > 0) { + throw new Error( + `knowledge file transaction is already active: ${activeTransactions.join(', ')}`, + ) + } + + const paths = new Set() + const prepared = await Promise.all( + input.mutations.map(async (mutation, index) => { + const path = assertKnowledgeMutationPath(mutation.path) + if (paths.has(path)) throw new Error(`knowledge file transaction repeats path: ${path}`) + if (mutation.content === null && mutation.mode !== undefined) { + throw new Error(`deleted knowledge file cannot declare a mode: ${path}`) + } + paths.add(path) + const before = await withSafeDescendant(input.root, path, readRegularFile) + const after = + mutation.content === null + ? null + : Buffer.isBuffer(mutation.content) + ? mutation.content + : Buffer.from(mutation.content, 'utf8') + const afterMode = after + ? fileModeSchema.parse(mutation.mode ?? before?.mode ?? DEFAULT_FILE_MODE) + : undefined + if ( + !input.includeUnchanged && + sameBytes(before?.bytes ?? null, after) && + before?.mode === afterMode + ) { + return null + } + return { + entry: { + index, + path, + beforeHash: before ? hashBytes(before.bytes) : null, + afterHash: after ? hashBytes(after) : null, + ...(before ? { beforeMode: before.mode } : {}), + ...(afterMode === undefined ? {} : { afterMode }), + }, + before: before?.bytes ?? null, + after, + } + }), + ) + const changed = prepared.filter((entry): entry is NonNullable => entry !== null) + if (changed.length === 0) return null + + const preparationDir = await mkdtemp(join(transactionRoot, 'prepare-')) + let activated = false + try { + for (const item of changed) { + if (item.before) { + await writeFileDurable( + snapshotPath(preparationDir, 'before', item.entry.index), + item.before, + ) + } + if (item.after) { + await writeFileDurable( + snapshotPath(preparationDir, 'after', item.entry.index), + item.after, + ) + } + } + const transaction = transactionSchema.parse({ + schemaVersion: 1, + kind: 'knowledge-file-transaction', + transactionId: randomUUID(), + purpose: input.purpose, + createdAt: (input.now ?? (() => new Date()))().toISOString(), + entries: changed.map((item) => item.entry), + }) + await writeJsonDurable(join(preparationDir, 'transaction.json'), transaction) + const activeDir = join( + transactionRoot, + activeTransactionDirectoryName(transaction.transactionId), + ) + await renameDurable(preparationDir, activeDir) + activated = true + const competingTransactions = await activeTransactionDirectoryNames(transactionRoot) + if (competingTransactions.length > 1) { + await rm(activeDir, { recursive: true, force: true }) + await syncDirectory(transactionRoot) + throw new Error('multiple knowledge file transactions became active concurrently') + } + return transaction + } finally { + if (!activated) await rm(preparationDir, { recursive: true, force: true }) + } + }) +} + +export async function commitKnowledgeFileMutations(input: { + root: string + transactionRoot: string + purpose: string + mutations: readonly KnowledgeFileMutation[] + assertOwned?: () => void + now?: () => Date +}): Promise { + if ( + await loadKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + }) + ) { + throw new Error('recover the pending knowledge transaction before planning another write') + } + const transaction = await prepareKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + purpose: input.purpose, + mutations: input.mutations, + now: input.now, + }) + if (!transaction) return false + input.assertOwned?.() + await applyKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + transaction, + beforeCommit: input.assertOwned, + }) + await finishKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + transaction, + assertOwned: input.assertOwned, + }) + return true +} + +export async function recoverKnowledgeFileTransaction(input: { + root: string + transactionRoot: string + expectedPurpose: string + direction?: 'apply' | 'rollback' + validate?: (transaction: KnowledgeFileTransaction) => void + assertOwned?: () => void +}): Promise { + const pending = await inspectKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + }) + if (!pending) return false + const { transaction, direction } = pending + if (transaction.purpose !== input.expectedPurpose) { + throw new Error(`knowledge transaction '${transaction.purpose}' requires its owner to resume`) + } + input.validate?.(transaction) + input.assertOwned?.() + if (input.direction === 'apply' && direction === 'rollback') { + throw new Error(`knowledge transaction '${transaction.transactionId}' is already rolling back`) + } + const requestedDirection = input.direction ?? direction + if (requestedDirection === 'rollback') { + await rollbackKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + transaction, + beforeCommit: input.assertOwned, + }) + } else { + await applyKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + transaction, + beforeCommit: input.assertOwned, + }) + } + await finishKnowledgeFileTransaction({ + root: input.root, + transactionRoot: input.transactionRoot, + transaction, + assertOwned: input.assertOwned, + }) + return true +} + +export async function loadKnowledgeFileTransaction(input: { + root: string + transactionRoot: string +}): Promise { + return (await inspectKnowledgeFileTransaction(input))?.transaction ?? null +} + +export async function inspectKnowledgeFileTransaction(input: { + root: string + transactionRoot: string +}): Promise { + return loadKnowledgeFileTransactionState(input) +} + +export async function applyKnowledgeFileTransaction(input: { + root: string + transactionRoot: string + transaction: KnowledgeFileTransaction + beforeCommit?: (entry: KnowledgeFileTransaction['entries'][number]) => Promise | void +}): Promise { + const transaction = transactionSchema.parse(input.transaction) + assertTransactionEntries(transaction) + await withTransactionDirectory( + input.root, + input.transactionRoot, + transaction, + async (transactionDir) => { + await assertActiveTransaction(transactionDir, transaction) + if ((await readTransactionDirection(transactionDir, transaction)) === 'rollback') { + throw new Error(`knowledge transaction '${transaction.transactionId}' is rolling back`) + } + for (const entry of transaction.entries) { + await withSafeDescendant(input.root, entry.path, async (target) => { + const current = await readRegularFile(target) + if (fileMatches(current, entry.afterHash, entry.afterMode)) return + if (!fileMatches(current, entry.beforeHash, entry.beforeMode)) { + throw new Error(`knowledge file changed outside transaction: ${entry.path}`) + } + await input.beforeCommit?.(entry) + if (entry.afterHash === null) { + await removeDurable(target) + return + } + const after = ( + await readRegularFileNoFollow(snapshotPath(transactionDir, 'after', entry.index)) + ).bytes + if (hashBytes(after) !== entry.afterHash) { + throw new Error(`knowledge transaction snapshot changed: ${entry.path}`) + } + await writeFileDurable(target, after, { mode: entry.afterMode! }) + }) + } + await assertKnowledgeFileTransactionApplied(input.root, transaction) + }, + ) +} + +export async function assertKnowledgeFileTransactionApplied( + root: string, + transaction: KnowledgeFileTransaction, +): Promise { + for (const entry of transaction.entries) { + await withSafeDescendant(root, entry.path, async (target) => { + const current = await readRegularFile(target) + if (!fileMatches(current, entry.afterHash, entry.afterMode)) { + throw new Error(`knowledge file transaction did not commit: ${entry.path}`) + } + }) + } +} + +export async function finishKnowledgeFileTransaction(input: { + root: string + transactionRoot: string + transaction: KnowledgeFileTransaction + assertOwned?: () => void +}): Promise { + const transaction = transactionSchema.parse(input.transaction) + assertTransactionEntries(transaction) + await withTransactionRoot(input.root, input.transactionRoot, false, async (transactionRoot) => { + const activeName = activeTransactionDirectoryName(transaction.transactionId) + await withTransactionDirectory( + input.root, + input.transactionRoot, + transaction, + async (transactionDir) => { + await assertActiveTransaction(transactionDir, transaction) + input.assertOwned?.() + const direction = await readTransactionDirection(transactionDir, transaction) + await assertKnowledgeFileTransactionTerminal(input.root, transaction, direction) + await withSafeDirectory(transactionRoot, activeName, false, async (currentDir) => { + await assertActiveTransaction(currentDir, transaction) + const currentDirection = await readTransactionDirection(currentDir, transaction) + if (currentDirection !== direction) { + throw new Error( + `knowledge transaction '${transaction.transactionId}' changed direction`, + ) + } + await assertKnowledgeFileTransactionTerminal(input.root, transaction, currentDirection) + }) + }, + ) + await rm(join(transactionRoot, activeName), { recursive: true, force: false }) + await syncDirectory(transactionRoot) + }) +} + +export async function rollbackKnowledgeFileTransaction(input: { + root: string + transactionRoot: string + transaction: KnowledgeFileTransaction + beforeCommit?: (entry: KnowledgeFileTransaction['entries'][number]) => Promise | void +}): Promise { + await withTransactionDirectory( + input.root, + input.transactionRoot, + input.transaction, + async (transactionDir) => { + const transaction = transactionSchema.parse(input.transaction) + assertTransactionEntries(transaction) + await assertActiveTransaction(transactionDir, transaction) + await input.beforeCommit?.(transaction.entries.at(-1)!) + const direction = await readTransactionDirection(transactionDir, transaction) + if (direction !== 'rollback') { + await writeJsonDurable(join(transactionDir, 'direction.json'), { + schemaVersion: 1, + transactionId: transaction.transactionId, + direction: 'rollback', + }) + } + for (const entry of [...transaction.entries].reverse()) { + await withSafeDescendant(input.root, entry.path, async (target) => { + const current = await readRegularFile(target) + if (fileMatches(current, entry.beforeHash, entry.beforeMode)) return + if (!fileMatches(current, entry.afterHash, entry.afterMode)) { + throw new Error(`knowledge file changed outside rollback: ${entry.path}`) + } + await input.beforeCommit?.(entry) + if (entry.beforeHash === null) { + await removeDurable(target) + return + } + const before = ( + await readRegularFileNoFollow(snapshotPath(transactionDir, 'before', entry.index)) + ).bytes + if (hashBytes(before) !== entry.beforeHash) { + throw new Error(`knowledge rollback snapshot changed: ${entry.path}`) + } + await writeFileDurable(target, before, { mode: entry.beforeMode! }) + }) + } + await input.beforeCommit?.(transaction.entries[0]!) + for (const entry of transaction.entries) { + await withSafeDescendant(input.root, entry.path, async (target) => { + const current = await readRegularFile(target) + if (!fileMatches(current, entry.beforeHash, entry.beforeMode)) { + throw new Error(`knowledge file transaction did not roll back: ${entry.path}`) + } + }) + } + }, + ) +} + +async function loadKnowledgeFileTransactionState(input: { + root: string + transactionRoot: string +}): Promise { + try { + return await withTransactionRoot(input.root, input.transactionRoot, false, async (root) => { + const activeTransactions = await activeTransactionDirectoryNames(root) + if (activeTransactions.length === 0) return null + if (activeTransactions.length > 1) { + throw new Error('multiple knowledge file transactions are active') + } + return withSafeDirectory(root, activeTransactions[0]!, false, async (transactionDir) => { + const transaction = await readActiveTransaction(transactionDir) + return { + transaction, + direction: await readTransactionDirection(transactionDir, transaction), + } + }) + }) + } catch (error) { + if (isMissingFile(error)) return null + throw error + } +} + +async function readActiveTransaction(transactionDir: string): Promise { + const transaction = transactionSchema.parse( + JSON.parse( + (await readRegularFileNoFollow(join(transactionDir, 'transaction.json'))).bytes.toString( + 'utf8', + ), + ), + ) + assertTransactionEntries(transaction) + return transaction +} + +async function assertActiveTransaction( + transactionDir: string, + expected: KnowledgeFileTransaction, +): Promise { + const active = await readActiveTransaction(transactionDir) + if ( + active.transactionId !== expected.transactionId || + contentHash(active) !== contentHash(transactionSchema.parse(expected)) + ) { + throw new Error(`active knowledge transaction does not match '${expected.transactionId}'`) + } +} + +async function readTransactionDirection( + transactionDir: string, + transaction: KnowledgeFileTransaction, +): Promise<'apply' | 'rollback'> { + try { + const direction = transactionDirectionSchema.parse( + JSON.parse( + (await readRegularFileNoFollow(join(transactionDir, 'direction.json'))).bytes.toString( + 'utf8', + ), + ), + ) + if (direction.transactionId !== transaction.transactionId) { + throw new Error('knowledge transaction direction belongs to another transaction') + } + return direction.direction + } catch (error) { + if (isMissingFile(error)) return 'apply' + throw error + } +} + +function snapshotPath(root: string, side: 'before' | 'after', index: number): string { + return join(root, side, `${index}.bin`) +} + +async function withTransactionDirectory( + root: string, + transactionRoot: string, + transaction: KnowledgeFileTransaction, + use: (transactionDir: string) => Promise | T, +): Promise { + const parsed = transactionSchema.parse(transaction) + try { + return await withTransactionRoot(root, transactionRoot, false, (safeRoot) => + withSafeDirectory(safeRoot, activeTransactionDirectoryName(parsed.transactionId), false, use), + ) + } catch (error) { + if (isMissingFile(error)) { + throw new Error(`active knowledge transaction does not match '${parsed.transactionId}'`, { + cause: error, + }) + } + throw error + } +} + +async function activeTransactionDirectoryNames(transactionRoot: string): Promise { + const active: string[] = [] + for (const entry of await readdir(transactionRoot, { withFileTypes: true })) { + if (entry.name === 'active') { + throw new Error('knowledge transaction store contains an unsupported active journal') + } + if (!entry.name.startsWith('active-')) continue + if (!entry.isDirectory()) { + throw new Error(`knowledge transaction journal is not a directory: ${entry.name}`) + } + activeTransactionDirectoryName(entry.name.slice('active-'.length)) + active.push(entry.name) + } + return active.sort() +} + +function activeTransactionDirectoryName(transactionId: string): string { + return `active-${z.string().uuid().parse(transactionId)}` +} + +async function assertKnowledgeFileTransactionTerminal( + root: string, + transaction: KnowledgeFileTransaction, + direction: 'apply' | 'rollback', +): Promise { + if (direction === 'apply') { + await assertKnowledgeFileTransactionApplied(root, transaction) + return + } + for (const entry of transaction.entries) { + await withSafeDescendant(root, entry.path, async (target) => { + const current = await readRegularFile(target) + if (!fileMatches(current, entry.beforeHash, entry.beforeMode)) { + throw new Error(`knowledge file transaction did not roll back: ${entry.path}`) + } + }) + } +} + +async function withTransactionRoot( + root: string, + transactionRoot: string, + create: boolean, + use: (transactionRoot: string) => Promise | T, +): Promise { + if (isKernelAnchoredPath(transactionRoot)) { + const match = transactionRoot.match(/^(\/proc\/self\/fd\/\d+)(?:\/(.*))?$/) + if (!match?.[1]) throw new Error('knowledge transaction directory has an invalid anchor') + return withSafeDirectory(match[1], match[2] ?? '', create, use) + } + const resolvedRoot = resolve(root) + const resolvedTransactionRoot = resolve(transactionRoot) + if (!resolvedTransactionRoot.startsWith(`${resolvedRoot}${sep}`)) { + throw new Error('knowledge transaction directory escaped its root') + } + return withSafeDirectory( + root, + relative(resolvedRoot, resolvedTransactionRoot).replace(/\\/g, '/'), + create, + use, + ) +} + +function assertTransactionEntries(transaction: KnowledgeFileTransaction): void { + const indexes = new Set() + const paths = new Set() + for (const entry of transaction.entries) { + const normalized = assertKnowledgeMutationPath(entry.path) + if (entry.path !== normalized || indexes.has(entry.index) || paths.has(entry.path)) { + throw new Error('knowledge file transaction has duplicate or unsafe entries') + } + assertHashModePair(entry.path, 'before', entry.beforeHash, entry.beforeMode) + assertHashModePair(entry.path, 'after', entry.afterHash, entry.afterMode) + indexes.add(entry.index) + paths.add(entry.path) + } +} + +function normalizePlanEntry(entry: KnowledgeFileTransactionPlanEntry) { + const path = assertKnowledgeMutationPath(entry.path) + assertHashModePair(path, 'before', entry.beforeHash, entry.beforeMode) + assertHashModePair(path, 'after', entry.afterHash, entry.afterMode) + return { + path, + beforeHash: entry.beforeHash, + afterHash: entry.afterHash, + ...(entry.beforeMode === undefined + ? {} + : { beforeMode: fileModeSchema.parse(entry.beforeMode) }), + ...(entry.afterMode === undefined ? {} : { afterMode: fileModeSchema.parse(entry.afterMode) }), + } +} + +function assertHashModePair( + path: string, + side: 'before' | 'after', + hash: string | null, + mode: number | undefined, +): void { + if ((hash === null) !== (mode === undefined)) { + throw new Error(`knowledge transaction ${side} hash and mode disagree: ${path}`) + } + if (mode !== undefined) fileModeSchema.parse(mode) +} + +export function assertKnowledgeMutationPath(path: string): string { + const normalized = normalizeTransactionPath(path) + if ( + normalized === '.agent-knowledge/sources.json' || + normalized.startsWith('knowledge/') || + normalized.startsWith('raw/') + ) { + return normalized + } + throw new Error(`knowledge transaction contains an unsupported path: ${path}`) +} + +function normalizeTransactionPath(path: string): string { + const normalized = path.replace(/\\/g, '/') + if ( + normalized.length === 0 || + normalized.startsWith('/') || + /^[A-Za-z]:/.test(normalized) || + normalized.split('/').some((part) => part === '' || part === '.' || part === '..') + ) { + throw new Error(`knowledge file transaction has an unsafe path: ${path}`) + } + return normalized +} + +async function readRegularFile(path: string): Promise<{ bytes: Buffer; mode: number } | null> { + try { + return await readRegularFileNoFollow(path) + } catch (error) { + if (isMissingFile(error)) return null + throw error + } +} + +function hashBytes(bytes: Buffer): string { + return createHash('sha256').update(bytes).digest('hex') +} + +function sameBytes(left: Buffer | null, right: Buffer | null): boolean { + if (left === null || right === null) return left === right + return left.equals(right) +} + +function fileMatches( + file: { bytes: Buffer; mode: number } | null, + hash: string | null, + mode: number | undefined, +): boolean { + return (file ? hashBytes(file.bytes) : null) === hash && file?.mode === mode +} diff --git a/src/freshness.ts b/src/freshness.ts index b1d3580..9f6264c 100644 --- a/src/freshness.ts +++ b/src/freshness.ts @@ -1,5 +1,8 @@ -import { mkdir, readFile, writeFile } from 'node:fs/promises' -import { dirname, join } from 'node:path' +import { readFile } from 'node:fs/promises' +import { join } from 'node:path' +import { z } from 'zod' +import { isMissingFile, writeJsonDurableWithinRoot } from './durable-fs' +import { withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' /** * Knowledge freshness store: tracks when each `(workspaceId, sourceId)` pair @@ -81,34 +84,48 @@ export interface FileSystemFreshnessStoreOptions { root: string } +const freshnessRecordSchema = z + .object({ + workspaceId: z.string().min(1), + sourceId: z.string().min(1), + lastRefreshedAt: z.iso.datetime(), + contentHash: z.string().min(1).optional(), + }) + .strict() +const freshnessFileSchema = z + .object({ records: z.record(z.string(), freshnessRecordSchema) }) + .strict() + /** * Filesystem-backed implementation. Single JSON file per knowledge root, * indexed by `${workspaceId}::${sourceId}`. Reads parse on every call — * cron tick rate is well below the cost of one JSON parse. * - * Concurrent writes from a single process serialize through `writeQueue`. - * Cross-process concurrency is undefined; the consuming app should run the - * cron in a single worker. + * Writes share the package-wide filesystem lock, so multiple workers cannot + * overwrite one another or run through an interrupted knowledge promotion. */ export function createFileSystemFreshnessStore( options: FileSystemFreshnessStoreOptions, ): KnowledgeFreshnessStore { const path = join(options.root, '.agent-knowledge', 'freshness.json') - let writeQueue: Promise = Promise.resolve() const read = async (): Promise> => { - try { - const text = await readFile(path, 'utf8') - const parsed = JSON.parse(text) as { records?: Record } - return parsed.records ?? {} - } catch { - return {} - } + return withKnowledgeRead(options.root, async () => { + try { + return freshnessFileSchema.parse(JSON.parse(await readFile(path, 'utf8'))).records + } catch (error) { + if (isMissingFile(error)) return {} + throw error + } + }) } const write = async (records: Record): Promise => { - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, `${JSON.stringify({ records }, null, 2)}\n`, 'utf8') + await writeJsonDurableWithinRoot( + options.root, + '.agent-knowledge/freshness.json', + freshnessFileSchema.parse({ records }), + ) } return { @@ -118,17 +135,16 @@ export function createFileSystemFreshnessStore( return record ? new Date(record.lastRefreshedAt) : null }, async mark(input) { - writeQueue = writeQueue.then(async () => { + await withKnowledgeMutation(options.root, async () => { const records = await read() records[buildKey(input)] = { workspaceId: input.workspaceId, sourceId: input.sourceId, lastRefreshedAt: input.when.toISOString(), - contentHash: input.contentHash, + ...(input.contentHash === undefined ? {} : { contentHash: input.contentHash }), } await write(records) }) - await writeQueue }, async stale(input) { const last = await this.last(input) diff --git a/src/index.ts b/src/index.ts index 6e451ea..83fe913 100644 --- a/src/index.ts +++ b/src/index.ts @@ -21,6 +21,14 @@ export * from './kb-store' export * from './lint' export * from './material-facts-metric' export * from './memory/index' +export type { + PendingKnowledgeMutation, + RecoverPendingKnowledgeMutationOptions, +} from './mutation-lock' +export { + inspectPendingKnowledgeMutation, + recoverPendingKnowledgeMutation, +} from './mutation-lock' export * from './proposals' export * from './propose-from-finding' export * from './rag-eval' diff --git a/src/indexer.ts b/src/indexer.ts index 6f533aa..1a0e1ab 100644 --- a/src/indexer.ts +++ b/src/indexer.ts @@ -1,10 +1,15 @@ -import { join } from 'node:path' +import { writeJsonDurableWithinRoot } from './durable-fs' import { buildKnowledgeGraph } from './graph' +import { withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' import { loadSourceRegistry } from './sources' -import { layoutFor, loadKnowledgePages, writeJson } from './store' +import { loadKnowledgePages } from './store' import type { KnowledgeIndex } from './types' export async function buildKnowledgeIndex(root: string): Promise { + return withKnowledgeRead(root, () => buildKnowledgeIndexUnlocked(root)) +} + +async function buildKnowledgeIndexUnlocked(root: string): Promise { const [pages, sourceRegistry] = await Promise.all([ loadKnowledgePages(root), loadSourceRegistry(root), @@ -20,7 +25,9 @@ export async function buildKnowledgeIndex(root: string): Promise } export async function writeKnowledgeIndex(root: string): Promise { - const index = await buildKnowledgeIndex(root) - await writeJson(join(layoutFor(root).cacheDir, 'index.json'), index) - return index + return withKnowledgeMutation(root, async () => { + const index = await buildKnowledgeIndexUnlocked(root) + await writeJsonDurableWithinRoot(root, '.agent-knowledge/index.json', index) + return index + }) } diff --git a/src/investment-thesis-task.ts b/src/investment-thesis-task.ts index fe2fc57..ac7ab23 100644 --- a/src/investment-thesis-task.ts +++ b/src/investment-thesis-task.ts @@ -18,12 +18,13 @@ * is research depth, not teaching-to-the-test. */ -import { mkdir, writeFile } from 'node:fs/promises' import { join } from 'node:path' +import { writeFileDurableWithinRoot } from './durable-fs' import { defineReadinessSpec, type KnowledgeReadinessSpec } from './eval-readiness' +import { slugify } from './ids' import { buildKnowledgeIndex } from './indexer' import { kbIndexToText } from './material-facts-metric' -import { layoutFor } from './store' +import { withKnowledgeMutation } from './mutation-lock' import { type ResearchDriver, runVerifiedResearchLoop, @@ -156,23 +157,23 @@ async function writeThesisPage( input: ThesisTaskInput, thesis: string, ): Promise { - const { knowledgeDir } = layoutFor(root) - await mkdir(knowledgeDir, { recursive: true }) - const path = join(knowledgeDir, `thesis-${input.ticker.toLowerCase()}.md`) - const body = [ - '---', - `title: Investment thesis — ${input.company} (${input.ticker})`, - `ticker: ${input.ticker}`, - `cutoff: ${input.cutoff}`, - 'kind: investment-thesis', - '---', - `# Investment thesis — ${input.company} (${input.ticker}), as of ${input.cutoff}`, - '', - thesis.trim(), - '', - ].join('\n') - await writeFile(path, body, 'utf8') - return path + return withKnowledgeMutation(root, async () => { + const relativePath = `knowledge/thesis-${slugify(input.ticker)}.md` + const body = [ + '---', + `title: Investment thesis — ${input.company} (${input.ticker})`, + `ticker: ${input.ticker}`, + `cutoff: ${input.cutoff}`, + 'kind: investment-thesis', + '---', + `# Investment thesis — ${input.company} (${input.ticker}), as of ${input.cutoff}`, + '', + thesis.trim(), + '', + ].join('\n') + await writeFileDurableWithinRoot(root, relativePath, body, { encoding: 'utf8' }) + return join(root, relativePath) + }) } export interface ThesisRunOptions { diff --git a/src/kb-improvement.ts b/src/kb-improvement.ts index a531a18..af38885 100644 --- a/src/kb-improvement.ts +++ b/src/kb-improvement.ts @@ -1,33 +1,59 @@ -import { cp, mkdir, open, readdir, readFile, rename, rm, stat, writeFile } from 'node:fs/promises' -import { dirname, join, relative } from 'node:path' +import { createHash } from 'node:crypto' +import { cp, lstat, mkdir, mkdtemp, rm, stat } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path' +import { + canonicalJson, + contentHash, + type RunRecord, + validateRunRecord, +} from '@tangle-network/agent-eval' +import { z } from 'zod' +import { + isMissingFile, + listRegularFilesWithinRoot, + readRegularFileWithinRoot, + renameDurable, + withSafeDirectory, + writeJsonDurableWithinRoot, +} from './durable-fs' import type { BuildEvalKnowledgeBundleOptions, EvalKnowledgeBundleBuildResult, KnowledgeReadinessSpec, } from './eval-readiness' -import { sha256, stableId } from './ids' +import { + applyKnowledgeFileTransaction, + assertKnowledgeMutationPath, + finishKnowledgeFileTransaction, + type KnowledgeFileMutation, + type KnowledgeFileTransaction, + type KnowledgeFileTransactionPlanEntry, + knowledgeFileTransactionPlanHash, + prepareKnowledgeFileTransaction, + rollbackKnowledgeFileTransaction, +} from './file-transaction' +import { sha256, slugify, stableId } from './ids' import { buildKnowledgeIndex, writeKnowledgeIndex } from './indexer' +import { acquireDurableFileLock, withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' import { type KnowledgeBaseQualityOptions, type KnowledgeBaseQualityReport, scoreKnowledgeBaseIndex, } from './rag-eval' import { - type RagAnswerQualityResult, type RagKnowledgeImprovementPhase, - type RagKnowledgeImprovementPhaseResult, type RagKnowledgeResearchOptions, type RagKnowledgeUpdateInput, type RagKnowledgeUpdateResult, - type RagPromotionResult, type RunRagKnowledgeImprovementLoopOptions, type RunRagKnowledgeImprovementLoopResult, runRagKnowledgeImprovementLoop, } from './rag-improvement-loop' import { readinessFor } from './readiness-helpers' import type { RunKnowledgeResearchLoopOptions } from './research-loop' -import type { RetrievalConfig, RunRetrievalImprovementLoopOptions } from './retrieval-eval' -import { initKnowledgeBase, layoutFor } from './store' +import type { RunRetrievalImprovementLoopOptions } from './retrieval-eval' +import { layoutFor } from './store' import type { KnowledgeIndex } from './types' import { type ValidateKnowledgeOptions, @@ -42,11 +68,33 @@ export type KnowledgeImprovementStatus = | 'rejected' | 'blocked' +interface KnowledgeImprovementMetricProvenanceBase { + evaluator: string + version: string +} + +export type KnowledgeImprovementMetricProvenance = + | (KnowledgeImprovementMetricProvenanceBase & { + method: 'deterministic' + }) + | (KnowledgeImprovementMetricProvenanceBase & { + method: 'sampled' | 'composite' + corpusHash: string + runRecords: RunRecord[] + }) + | (KnowledgeImprovementMetricProvenanceBase & { + method: 'model' + model: string + corpusHash: string + runRecords: RunRecord[] + }) + export interface KnowledgeImprovementMetric { score: number passed: boolean dimensions?: Record notes?: string + provenance: KnowledgeImprovementMetricProvenance } export interface KnowledgeImprovementEvaluationInput { @@ -70,37 +118,20 @@ export type KnowledgeImprovementEvaluator = ( input: KnowledgeImprovementEvaluationInput, ) => Promise | KnowledgeImprovementMetric -export interface KnowledgeImprovementLifecycleRecord { - stage: 'candidate-update' | 'candidate-evaluation' - phases: readonly RagKnowledgeImprovementPhaseResult[] - findingCount: number - retrievalWinnerConfig?: RetrievalConfig - answerQuality?: RagAnswerQualityResult - promotionDecision?: RagPromotionResult -} - export interface KnowledgeImprovementCandidateRecord { iteration: number candidateId: string - candidateRoot: string baseHash: string candidateHash?: string + evidenceHash?: string + promotionPlanHash?: string status: KnowledgeImprovementStatus createdAt: string updatedAt: string - validation?: ValidateKnowledgeResult - kbQuality?: KnowledgeBaseQualityReport - readinessBlockingMissing?: number - evaluation?: KnowledgeImprovementMetric - lifecycle?: readonly KnowledgeImprovementLifecycleRecord[] - retrievalWinnerConfig?: RetrievalConfig - answerQuality?: RagAnswerQualityResult - promotionDecision?: RagPromotionResult - notes?: string } export interface KnowledgeImprovementRunState { - version: 1 + schemaVersion: 1 runId: string root: string goal: string @@ -116,14 +147,268 @@ export interface KnowledgeImprovementRunState { export interface KnowledgeImprovementResult { runId: string - runDir: string state: KnowledgeImprovementRunState candidate?: KnowledgeImprovementCandidateRecord + evaluation?: KnowledgeImprovementMetric lifecycle?: RunRagKnowledgeImprovementLoopResult promoted: boolean blocked: boolean } +const digestSchema = z.string().regex(/^[a-f0-9]{64}$/) +const runIdSchema = z.string().min(1).max(2_048) +const safePathSegmentSchema = z + .string() + .min(1) + .max(128) + .regex(/^[A-Za-z0-9][A-Za-z0-9._-]*$/) +const improvementStatusSchema = z.enum([ + 'running', + 'candidate-ready', + 'promoted', + 'rejected', + 'blocked', +]) +const runRecordSchema = z.custom((value) => { + try { + validateRunRecord(value) + return true + } catch { + return false + } +}, 'invalid agent-eval RunRecord') +const deterministicMetricProvenanceSchema = z + .object({ + evaluator: z.string().min(1), + version: z.string().min(1), + method: z.literal('deterministic'), + }) + .strict() +const measuredMetricProvenanceSchema = z + .object({ + evaluator: z.string().min(1), + version: z.string().min(1), + method: z.enum(['sampled', 'composite']), + corpusHash: digestSchema, + runRecords: z.array(runRecordSchema).min(1), + }) + .strict() +const modelMetricProvenanceSchema = z + .object({ + evaluator: z.string().min(1), + version: z.string().min(1), + method: z.literal('model'), + model: z.string().min(1), + corpusHash: digestSchema, + runRecords: z.array(runRecordSchema).min(1), + }) + .strict() +const improvementMetricSchema = z + .object({ + score: z.number().finite().min(0).max(1), + passed: z.boolean(), + dimensions: z.record(z.string(), z.number().finite()).optional(), + notes: z.string().optional(), + provenance: z.discriminatedUnion('method', [ + deterministicMetricProvenanceSchema, + measuredMetricProvenanceSchema, + modelMetricProvenanceSchema, + ]), + }) + .strict() + .superRefine((metric, context) => { + if (metric.provenance.method === 'deterministic') return + const actualCorpusHash = contentHash(metric.provenance.runRecords) + if (actualCorpusHash !== metric.provenance.corpusHash) { + context.addIssue({ + code: 'custom', + path: ['provenance', 'corpusHash'], + message: 'metric corpus hash must bind the complete RunRecord array', + }) + } + }) +const candidateRecordSchema = z + .object({ + iteration: z.number().int().positive(), + candidateId: safePathSegmentSchema, + baseHash: digestSchema, + candidateHash: digestSchema.optional(), + evidenceHash: digestSchema.optional(), + promotionPlanHash: digestSchema.optional(), + status: improvementStatusSchema, + createdAt: z.iso.datetime(), + updatedAt: z.iso.datetime(), + }) + .strict() + +export const KnowledgeImprovementRunStateSchema = z + .object({ + schemaVersion: z.literal(1), + runId: runIdSchema, + root: z.string().min(1), + goal: z.string().min(1), + status: improvementStatusSchema, + baseHash: digestSchema, + createdAt: z.iso.datetime(), + updatedAt: z.iso.datetime(), + ownerId: z.string().min(1).optional(), + candidates: z.array(candidateRecordSchema), + promotedCandidateId: safePathSegmentSchema.optional(), + blockedReason: z.string().min(1).optional(), + }) + .strict() + .superRefine((state, context) => { + const candidateIds = new Set() + for (const [index, candidate] of state.candidates.entries()) { + if (candidateIds.has(candidate.candidateId)) { + context.addIssue({ + code: 'custom', + path: ['candidates', index, 'candidateId'], + message: 'candidate ids must be unique within an improvement run', + }) + } + candidateIds.add(candidate.candidateId) + if (candidate.baseHash !== state.baseHash) { + context.addIssue({ + code: 'custom', + path: ['candidates', index, 'baseHash'], + message: 'candidate base hash must match its improvement run', + }) + } + if (candidate.status === 'candidate-ready') { + if (!candidate.candidateHash || !candidate.evidenceHash || !candidate.promotionPlanHash) { + context.addIssue({ + code: 'custom', + path: ['candidates', index], + message: 'ready candidates require content, evidence, and promotion-plan identities', + }) + } + } + if ( + candidate.status === 'promoted' && + (!candidate.candidateHash || !candidate.evidenceHash || !candidate.promotionPlanHash) + ) { + context.addIssue({ + code: 'custom', + path: ['candidates', index], + message: 'promoted candidates require content, evidence, and promotion-plan identities', + }) + } + if ( + candidate.status === 'promoted' && + Boolean(candidate.evidenceHash) !== Boolean(candidate.promotionPlanHash) + ) { + context.addIssue({ + code: 'custom', + path: ['candidates', index], + message: 'promoted candidate evidence and promotion-plan identities must appear together', + }) + } + if (candidate.status === 'promoted' && state.status !== 'promoted') { + context.addIssue({ + code: 'custom', + path: ['candidates', index, 'status'], + message: 'only a promoted run may contain a promoted candidate', + }) + } + } + if (state.status === 'promoted') { + const promoted = state.candidates.filter( + (candidate) => candidate.candidateId === state.promotedCandidateId, + ) + if (promoted.length !== 1 || promoted[0]?.status !== 'promoted') { + context.addIssue({ + code: 'custom', + path: ['promotedCandidateId'], + message: 'promoted state must identify exactly one promoted candidate', + }) + } + } else if (state.promotedCandidateId !== undefined) { + context.addIssue({ + code: 'custom', + path: ['promotedCandidateId'], + message: 'only a promoted run may identify a promoted candidate', + }) + } + if ( + state.status === 'candidate-ready' && + state.candidates.filter((candidate) => candidate.status === 'candidate-ready').length !== 1 + ) { + context.addIssue({ + code: 'custom', + path: ['status'], + message: 'candidate-ready state must contain exactly one ready candidate', + }) + } + if (state.status === 'blocked' && state.blockedReason === undefined) { + context.addIssue({ + code: 'custom', + path: ['blockedReason'], + message: 'blocked state must include a reason', + }) + } + }) + +export const KnowledgeImprovementEvidenceSchema = z + .object({ + schemaVersion: z.literal(1), + kind: z.literal('knowledge-improvement-evidence'), + runId: runIdSchema, + candidateId: safePathSegmentSchema, + iteration: z.number().int().positive(), + goalHash: digestSchema, + baseHash: digestSchema, + candidateHash: digestSchema, + promotionPlanHash: digestSchema, + validation: z.unknown(), + readiness: z.unknown().nullable(), + kbQuality: z.unknown(), + evaluation: improvementMetricSchema, + lifecycle: z.unknown().nullable(), + }) + .strict() + +export type KnowledgeImprovementEvidence = z.infer + +/** Portable identity of one measured candidate. Paths and mutable run state are deliberately excluded. */ +export const KnowledgeImprovementCandidateRefSchema = z + .object({ + schemaVersion: z.literal(1), + kind: z.literal('knowledge-improvement-candidate'), + runId: runIdSchema, + candidateId: safePathSegmentSchema, + goalHash: digestSchema, + baseHash: digestSchema, + candidateHash: digestSchema, + evidenceHash: digestSchema, + promotionPlanHash: digestSchema, + }) + .strict() + +export type KnowledgeImprovementCandidateRef = z.infer< + typeof KnowledgeImprovementCandidateRefSchema +> + +export interface PromoteKnowledgeCandidateOptions { + root: string + candidate: KnowledgeImprovementCandidateRef + ownerId?: string + leaseTtlMs?: number + now?: () => Date + onState?: (state: KnowledgeImprovementRunState) => Promise | void +} + +export interface UseKnowledgeImprovementCandidateOptions { + root: string + candidate: KnowledgeImprovementCandidateRef +} + +export interface ResolvedKnowledgeImprovementCandidate { + root: string + candidate: KnowledgeImprovementCandidateRef + evaluation: KnowledgeImprovementMetric +} + export interface KnowledgeImprovementRetrievalOptions extends Omit { runDir?: RunRetrievalImprovementLoopOptions['runDir'] @@ -147,11 +432,9 @@ export interface KnowledgeImprovementOptions { root: string goal: string runId?: string - runDir?: string ownerId?: string leaseTtlMs?: number resume?: boolean - promote?: boolean maxCandidates?: number candidateResearchIterations?: number strict?: ValidateKnowledgeOptions['strict'] @@ -177,17 +460,10 @@ export interface KnowledgeImprovementOptions { interface LeaseHandle { ownerId: string - path: string + assertOwned(): void release(): Promise } -interface LeaseFile { - ownerId: string - acquiredAt: string - expiresAt: string - pid: number -} - const DEFAULT_LEASE_TTL_MS = 15 * 60 * 1000 const UPDATE_PHASES: readonly RagKnowledgeImprovementPhase[] = [ 'knowledge-acquisition', @@ -205,47 +481,187 @@ export function knowledgeImprovementRunId(root: string, goal: string): string { } export function knowledgeImprovementRunDir(root: string, runId: string): string { - return join(layoutFor(root).cacheDir, 'improvements', runId) + const parsedRunId = runIdSchema.parse(runId) + const safeRunId = safePathSegmentSchema.safeParse(parsedRunId) + const runSegment = safeRunId.success + ? safeRunId.data + : `${slugify(parsedRunId).slice(0, 72)}-${sha256(parsedRunId).slice(0, 16)}` + const improvementsDir = join(layoutFor(root).cacheDir, 'improvements') + const runDir = join(improvementsDir, runSegment) + const resolvedImprovementsDir = resolve(improvementsDir) + const resolvedRunDir = resolve(runDir) + if (!resolvedRunDir.startsWith(`${resolvedImprovementsDir}${sep}`)) { + throw new Error('knowledge improvement run directory escaped its root') + } + return runDir +} + +async function withKnowledgeImprovementRun( + root: string, + runId: string, + create: boolean, + use: (runDir: string) => Promise | T, +): Promise { + const runDir = knowledgeImprovementRunDir(root, runId) + const relativePath = descendantPath(root, runDir) + if (!relativePath) throw new Error('knowledge improvement run directory escaped its root') + return withSafeDirectory(root, relativePath, create, async (openedRunDir) => { + const result = await use(openedRunDir) + const openedIdentity = await stat(openedRunDir) + const currentIdentity = await withSafeDirectory(root, relativePath, false, (currentRunDir) => + stat(currentRunDir), + ) + if (openedIdentity.dev !== currentIdentity.dev || openedIdentity.ino !== currentIdentity.ino) { + throw new Error('knowledge improvement run directory changed during use') + } + return result + }) } export async function loadKnowledgeImprovementState( root: string, runId: string, - runDir = knowledgeImprovementRunDir(root, runId), ): Promise { + const expectedRunId = runIdSchema.parse(runId) try { - return JSON.parse(await readFile(statePath(runDir), 'utf8')) as KnowledgeImprovementRunState - } catch { - return null + return await withKnowledgeImprovementRun(root, expectedRunId, false, (runDir) => + loadKnowledgeImprovementStateFromRun(root, expectedRunId, runDir), + ) + } catch (error) { + if (isMissingFile(error)) return null + throw error } } +async function loadKnowledgeImprovementStateFromRun( + root: string, + runId: string, + runDir: string, +): Promise { + const stateFile = await readRegularFileWithinRoot(runDir, 'state.json') + const raw = JSON.parse(stateFile.bytes.toString('utf8')) as unknown + const state = KnowledgeImprovementRunStateSchema.parse(raw) as KnowledgeImprovementRunState + if (state.runId !== runId) { + throw new Error('knowledge improvement state does not match the requested run') + } + if (resolve(state.root) !== resolve(root)) { + throw new Error('knowledge improvement state does not match the requested root') + } + for (const candidate of state.candidates) { + if (candidate.status === 'running') await assertCandidateWorkspace(runDir, candidate) + } + return state +} + +/** Freeze the exact knowledge bytes and measured evidence a later approval may promote. */ +export function knowledgeImprovementCandidateRef( + result: Pick, +): KnowledgeImprovementCandidateRef { + if (!result.candidate) throw new Error('knowledge improvement result has no candidate') + return candidateRefFor(result.runId, result.state, result.candidate) +} + +/** Use one measured snapshot while its directory identity remains open and stable. */ +export async function withKnowledgeImprovementCandidate( + options: UseKnowledgeImprovementCandidateOptions, + use: (candidate: ResolvedKnowledgeImprovementCandidate) => Promise | T, +): Promise { + assertExactCandidatePlatform() + const candidateRef = KnowledgeImprovementCandidateRefSchema.parse(options.candidate) + return withKnowledgeImprovementRun(options.root, candidateRef.runId, false, async (runDir) => { + const state = await loadKnowledgeImprovementStateFromRun( + options.root, + candidateRef.runId, + runDir, + ) + return withMeasuredCandidateSnapshot(options.root, runDir, state, candidateRef, (resolved) => + withIsolatedCandidateCopy(resolved.root, candidateRef.candidateHash, (root) => + use({ + root, + candidate: candidateRef, + evaluation: resolved.evidence.evaluation, + }), + ), + ) + }) +} + +/** Promote one previously measured candidate without rerunning research or evaluation. */ +export async function promoteKnowledgeCandidate( + options: PromoteKnowledgeCandidateOptions, +): Promise { + assertExactCandidatePlatform() + const candidateRef = Object.freeze( + KnowledgeImprovementCandidateRefSchema.parse(options.candidate), + ) + const now = options.now ?? (() => new Date()) + return withKnowledgeImprovementRun(options.root, candidateRef.runId, false, async (runDir) => { + const lease = await acquireRunLease(runDir, { + ownerId: options.ownerId ?? `pid-${process.pid}`, + ttlMs: options.leaseTtlMs ?? DEFAULT_LEASE_TTL_MS, + }) + try { + lease.assertOwned() + const state = await loadKnowledgeImprovementStateFromRun( + options.root, + candidateRef.runId, + runDir, + ) + return await promoteReadyCandidate({ + root: options.root, + runDir, + state, + candidateRef, + leaseTtlMs: options.leaseTtlMs ?? DEFAULT_LEASE_TTL_MS, + assertRunOwned: lease.assertOwned, + now, + onState: options.onState, + }) + } finally { + await lease.release() + } + }) +} + export async function improveKnowledgeBase( options: KnowledgeImprovementOptions, ): Promise { + assertExactCandidatePlatform() assertKnowledgeImprovementOptions(options) - const now = options.now ?? (() => new Date()) - const runId = options.runId ?? knowledgeImprovementRunId(options.root, options.goal) - const runDir = options.runDir ?? knowledgeImprovementRunDir(options.root, runId) - await initKnowledgeBase(options.root) - await mkdir(runDir, { recursive: true }) + const runId = runIdSchema.parse( + options.runId ?? knowledgeImprovementRunId(options.root, options.goal), + ) + return withKnowledgeImprovementRun(options.root, runId, true, (runDir) => + improveKnowledgeBaseInRun(options, runId, runDir, now), + ) +} +async function improveKnowledgeBaseInRun( + options: KnowledgeImprovementOptions, + runId: string, + runDir: string, + now: () => Date, +): Promise { const lease = await acquireRunLease(runDir, { ownerId: options.ownerId ?? `pid-${process.pid}`, ttlMs: options.leaseTtlMs ?? DEFAULT_LEASE_TTL_MS, - now, }) try { + lease.assertOwned() let state = options.resume === false ? null - : await loadKnowledgeImprovementState(options.root, runId, runDir) + : await loadKnowledgeImprovementStateFromRun(options.root, runId, runDir).catch((error) => { + if (isMissingFile(error)) return null + throw error + }) if (!state) { const baseHash = await hashKnowledgeBase(options.root) + await createBaselineSnapshot(runDir, options.root, baseHash) state = { - version: 1, + schemaVersion: 1, runId, root: options.root, goal: options.goal, @@ -259,22 +675,61 @@ export async function improveKnowledgeBase( await saveState(runDir, state, options.onState) await appendLedger(runDir, { type: 'run.created', runId, baseHash }) } + if (state.goal !== options.goal) { + throw new Error('knowledge improvement state does not match the requested goal') + } + const promotedCandidateId = state.promotedCandidateId + const promotedCandidate = + state.status === 'promoted' + ? state.candidates.find((candidate) => candidate.candidateId === promotedCandidateId) + : undefined + if (state.status === 'promoted' && !promotedCandidate) { + throw new Error('promoted knowledge state has no promoted candidate') + } + const resumablePromotion = state.candidates.find( + (candidate) => + (candidate.status === 'candidate-ready' || candidate.status === 'promoted') && + candidate.evidenceHash !== undefined && + candidate.promotionPlanHash !== undefined, + ) + const resumableCandidateRef = resumablePromotion + ? candidateRefFor(runId, state, resumablePromotion) + : undefined + await withKnowledgeMutation(options.root, () => undefined, { + resumeTransaction: resumableCandidateRef + ? { + purpose: promotionTransactionPurpose(resumableCandidateRef), + validate: (transaction) => + assertPromotionTransaction(transaction, resumableCandidateRef), + } + : undefined, + }) + await ensureBaselineSnapshot(runDir, options.root, state.baseHash) if (state.status === 'promoted') { - return { runId, runDir, state, promoted: true, blocked: false } + const promoted = promotedCandidate! + return await promoteReadyCandidate({ + root: options.root, + runDir, + state, + candidateRef: candidateRefFor(runId, state, promoted), + leaseTtlMs: options.leaseTtlMs ?? DEFAULT_LEASE_TTL_MS, + assertRunOwned: lease.assertOwned, + now, + onState: options.onState, + }) } if (state.status === 'blocked') { - return { runId, runDir, state, promoted: false, blocked: true } + return { runId, state, promoted: false, blocked: true } } const maxCandidates = Math.max(1, options.maxCandidates ?? 1) let candidate = findActiveCandidate(state) + let lastRejectedCandidate: KnowledgeImprovementCandidateRecord | undefined + let lastRejectedEvaluation: KnowledgeImprovementMetric | undefined let lifecycle: RunRagKnowledgeImprovementLoopResult | undefined - while ( - candidate?.status === 'running' || - (!candidate && state.candidates.length < maxCandidates) - ) { + while (candidate || state.candidates.length < maxCandidates) { if (!candidate) { const currentHash = await hashKnowledgeBase(options.root) if (currentHash !== state.baseHash) { @@ -285,9 +740,12 @@ export async function improveKnowledgeBase( options.onState, now, ) - return { runId, runDir, state, promoted: false, blocked: true } + return { runId, state, promoted: false, blocked: true } } - candidate = await createCandidateWorkspace(runDir, state, options.root, now) + const activeState = state + candidate = await withBaselineSnapshot(runDir, activeState.baseHash, (baselineRoot) => + createCandidateWorkspace(runDir, activeState, baselineRoot, now), + ) state.candidates.push(candidate) state.status = 'running' state.updatedAt = now().toISOString() @@ -300,33 +758,27 @@ export async function improveKnowledgeBase( }) } - lifecycle = await runCandidateLifecycle(runDir, runId, candidate, options, now) - candidate.status = 'candidate-ready' - candidate.updatedAt = now().toISOString() - state.status = 'candidate-ready' - state.updatedAt = now().toISOString() - await saveState(runDir, state, options.onState) - await appendLedger(runDir, { - type: 'candidate.ready', - runId, - candidateId: candidate.candidateId, - }) - } - - if (!candidate) { - state.status = 'rejected' - state.updatedAt = now().toISOString() - await saveState(runDir, state, options.onState) - return { runId, runDir, state, promoted: false, blocked: false } - } + const measured = await measureCandidate(runId, runDir, state, candidate, options, now) + candidate = measured.candidate + const evaluation = measured.evaluation + lifecycle = measured.lifecycle - candidate = await evaluateCandidate(runDir, state, candidate, lifecycle, options, now) - state.updatedAt = now().toISOString() - await saveState(runDir, state, options.onState) + if (evaluation.passed) { + candidate.status = 'candidate-ready' + candidate.updatedAt = now().toISOString() + state.status = 'candidate-ready' + state.updatedAt = now().toISOString() + await saveState(runDir, state, options.onState) + await appendLedger(runDir, { + type: 'candidate.ready', + runId, + candidateId: candidate.candidateId, + }) + break + } - if (!candidate.evaluation?.passed) { candidate.status = 'rejected' - state.status = 'rejected' + state.status = 'running' state.updatedAt = now().toISOString() await saveState(runDir, state, options.onState) await appendLedger(runDir, { @@ -334,52 +786,436 @@ export async function improveKnowledgeBase( runId, candidateId: candidate.candidateId, }) - return { runId, runDir, state, candidate, lifecycle, promoted: false, blocked: false } + lastRejectedCandidate = candidate + lastRejectedEvaluation = evaluation + candidate = undefined } - if (options.promote === false) { - state.status = 'candidate-ready' + if (!candidate) { + state.status = 'rejected' state.updatedAt = now().toISOString() await saveState(runDir, state, options.onState) - return { runId, runDir, state, candidate, lifecycle, promoted: false, blocked: false } - } - - const currentHash = await hashKnowledgeBase(options.root) - if (currentHash !== state.baseHash) { - state = await blockRun( - runDir, - state, - `base changed before promotion: expected ${state.baseHash}, got ${currentHash}`, - options.onState, - now, - ) - await appendLedger(runDir, { - type: 'promotion.blocked', + return { runId, - candidateId: candidate.candidateId, - reason: state.blockedReason, - }) - return { runId, runDir, state, candidate, lifecycle, promoted: false, blocked: true } + state, + candidate: lastRejectedCandidate, + ...(lastRejectedEvaluation ? { evaluation: lastRejectedEvaluation } : {}), + lifecycle, + promoted: false, + blocked: false, + } } - await promoteCandidate(options.root, candidate.candidateRoot) - candidate.status = 'promoted' - candidate.updatedAt = now().toISOString() - state.status = 'promoted' - state.promotedCandidateId = candidate.candidateId - state.updatedAt = now().toISOString() - await saveState(runDir, state, options.onState) - await appendLedger(runDir, { - type: 'candidate.promoted', + const evidence = await assertCandidateEvidence(runDir, candidateRefFor(runId, state, candidate)) + return { runId, - candidateId: candidate.candidateId, - }) - return { runId, runDir, state, candidate, lifecycle, promoted: true, blocked: false } + state, + candidate, + evaluation: evidence.evaluation, + lifecycle, + promoted: false, + blocked: false, + } } finally { await lease.release() } } +interface PromoteReadyCandidateInput { + root: string + runDir: string + state: KnowledgeImprovementRunState + candidateRef: KnowledgeImprovementCandidateRef + leaseTtlMs: number + assertRunOwned(): void + now: () => Date + onState?: KnowledgeImprovementOptions['onState'] + lifecycle?: RunRagKnowledgeImprovementLoopResult +} + +async function promoteReadyCandidate( + input: PromoteReadyCandidateInput, +): Promise { + const { candidateRef, runDir, state } = input + assertStateIdentity(input.root, candidateRef, state) + const candidate = state.candidates.find((entry) => entry.candidateId === candidateRef.candidateId) + if ( + !candidate || + canonicalJson(candidateRefFor(candidateRef.runId, state, candidate)) !== + canonicalJson(candidateRef) + ) { + throw new Error('knowledge candidate approval does not match the measured candidate') + } + + const purpose = promotionTransactionPurpose(candidateRef) + return withKnowledgeMutation( + input.root, + async (mutationLock) => { + input.assertRunOwned() + const transactionRoot = mutationLock.transactionRoot + let pending: KnowledgeFileTransaction | null = null + const currentHash = await hashKnowledgeBase(input.root) + if (state.status === 'promoted' && state.promotedCandidateId !== candidate.candidateId) { + throw new Error( + `knowledge run already promoted '${state.promotedCandidateId ?? 'unknown'}'`, + ) + } + if (state.status === 'promoted' && currentHash !== candidateRef.candidateHash) { + throw new Error( + `promoted knowledge base changed: expected ${candidateRef.candidateHash}, got ${currentHash}`, + ) + } + if (state.status !== 'promoted' && state.status !== 'candidate-ready') { + throw new Error( + `knowledge candidate is not ready for promotion: run=${state.status}, candidate=${candidate.status}`, + ) + } + if (currentHash !== state.baseHash && currentHash !== candidateRef.candidateHash) { + return await blockPromotion( + input, + candidate, + `base changed before promotion: expected ${state.baseHash}, got ${currentHash}`, + ) + } + + if (currentHash !== candidateRef.candidateHash) { + pending = await withMeasuredCandidateSnapshot( + input.root, + runDir, + state, + candidateRef, + (resolved) => + withBaselineSnapshot(runDir, state.baseHash, async (baselineRoot) => { + const plan = await promotionPlanEntries(baselineRoot, resolved.root) + if (knowledgeFileTransactionPlanHash(plan) !== candidateRef.promotionPlanHash) { + throw new Error('knowledge candidate promotion plan changed after approval') + } + return prepareKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + purpose, + mutations: await promotionMutations(resolved.root, plan), + includeUnchanged: true, + now: input.now, + }) + }), + ) + if (!pending) { + throw new Error('knowledge promotion plan unexpectedly contained no file changes') + } + try { + assertPromotionTransaction(pending, candidateRef) + } catch (error) { + try { + await rollbackKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + beforeCommit() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + await finishKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + assertOwned() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + } catch (cleanupError) { + throw new AggregateError( + [error, cleanupError], + 'invalid knowledge promotion transaction could not be removed', + ) + } + throw error + } + } + try { + if (pending) { + await applyKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + beforeCommit() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + } + mutationLock.assertOwned() + input.assertRunOwned() + if ((await hashKnowledgeBase(input.root)) !== candidateRef.candidateHash) { + throw new Error('promoted knowledge content does not match the approved candidate') + } + await writeKnowledgeIndex(input.root) + } catch (error) { + if (!pending) throw error + try { + mutationLock.assertOwned() + input.assertRunOwned() + } catch (ownershipError) { + throw new AggregateError( + [error, ownershipError], + 'knowledge promotion lost its lock and left the transaction pending', + ) + } + try { + await rollbackKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + beforeCommit() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + await finishKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + assertOwned() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + await writeKnowledgeIndex(input.root) + } catch (rollbackError) { + throw new AggregateError( + [error, rollbackError], + 'knowledge promotion failed and could not restore the previous files', + ) + } + throw error + } + candidate.status = 'promoted' + candidate.updatedAt = input.now().toISOString() + state.status = 'promoted' + state.promotedCandidateId = candidate.candidateId + state.updatedAt = input.now().toISOString() + await saveState(runDir, state, input.onState) + await ensurePromotionEvent(runDir, candidateRef) + if (pending) { + await finishKnowledgeFileTransaction({ + root: input.root, + transactionRoot, + transaction: pending, + assertOwned() { + mutationLock.assertOwned() + input.assertRunOwned() + }, + }) + } + return promotionResult(input, candidate, true, false) + }, + { + staleMs: input.leaseTtlMs, + resumeTransaction: { + purpose, + validate: (transaction) => assertPromotionTransaction(transaction, candidateRef), + }, + }, + ) +} + +async function blockPromotion( + input: PromoteReadyCandidateInput, + candidate: KnowledgeImprovementCandidateRecord, + reason: string, +): Promise { + await blockRun(input.runDir, input.state, reason, input.onState, input.now) + await appendLedger(input.runDir, { + type: 'promotion.blocked', + runId: input.candidateRef.runId, + candidateId: candidate.candidateId, + reason, + }) + return promotionResult(input, candidate, false, true) +} + +function promotionResult( + input: PromoteReadyCandidateInput, + candidate: KnowledgeImprovementCandidateRecord, + promoted: boolean, + blocked: boolean, +): KnowledgeImprovementResult { + return { + runId: input.candidateRef.runId, + state: input.state, + candidate, + ...(input.lifecycle ? { lifecycle: input.lifecycle } : {}), + promoted, + blocked, + } +} + +function candidateRefFor( + runId: string, + state: KnowledgeImprovementRunState, + candidate: KnowledgeImprovementCandidateRecord, +): KnowledgeImprovementCandidateRef { + if (!candidate.candidateHash) { + throw new Error(`knowledge candidate '${candidate.candidateId}' has no content hash`) + } + if (!candidate.evidenceHash) { + throw new Error(`knowledge candidate '${candidate.candidateId}' has no evidence hash`) + } + if (!candidate.promotionPlanHash) { + throw new Error(`knowledge candidate '${candidate.candidateId}' has no promotion plan hash`) + } + if (candidate.status !== 'candidate-ready' && candidate.status !== 'promoted') { + throw new Error(`knowledge candidate '${candidate.candidateId}' is not ready`) + } + return Object.freeze({ + schemaVersion: 1, + kind: 'knowledge-improvement-candidate', + runId, + candidateId: candidate.candidateId, + goalHash: sha256(state.goal), + baseHash: candidate.baseHash, + candidateHash: candidate.candidateHash, + evidenceHash: candidate.evidenceHash, + promotionPlanHash: candidate.promotionPlanHash, + }) +} + +async function withMeasuredCandidateSnapshot( + liveRoot: string, + runDir: string, + state: KnowledgeImprovementRunState, + candidateRef: KnowledgeImprovementCandidateRef, + use: (snapshot: { + root: string + candidate: KnowledgeImprovementCandidateRecord + evidence: KnowledgeImprovementEvidence + }) => Promise | T, +): Promise { + assertStateIdentity(liveRoot, candidateRef, state) + const candidate = state.candidates.find((entry) => entry.candidateId === candidateRef.candidateId) + if (!candidate) { + throw new Error(`knowledge candidate '${candidateRef.candidateId}' does not exist`) + } + const expectedRef = candidateRefFor(candidateRef.runId, state, candidate) + if (canonicalJson(expectedRef) !== canonicalJson(candidateRef)) { + throw new Error('knowledge candidate approval does not match the measured candidate') + } + const evidence = await assertCandidateEvidence(runDir, candidateRef) + const relativePath = join( + 'candidates', + candidate.candidateId, + 'snapshots', + candidateRef.candidateHash, + ) + return withSafeDirectory(runDir, relativePath, false, async (root) => { + if ((await hashKnowledgeBase(root)) !== candidateRef.candidateHash) { + throw new Error('knowledge candidate snapshot changed after approval') + } + const result = await use({ root, candidate, evidence }) + if ((await hashKnowledgeBase(root)) !== candidateRef.candidateHash) { + throw new Error('knowledge candidate snapshot changed during use') + } + return result + }) +} + +async function withIsolatedCandidateCopy( + sourceRoot: string, + expectedHash: string, + use: (root: string) => Promise | T, +): Promise { + const isolationRoot = await mkdtemp(join(tmpdir(), 'agent-knowledge-candidate-')) + const candidateRoot = join(isolationRoot, 'candidate') + try { + await copyKnowledgeWorkspace(sourceRoot, candidateRoot) + if ((await hashKnowledgeBase(candidateRoot)) !== expectedHash) { + throw new Error('isolated knowledge candidate does not match its approved content') + } + const result = await use(candidateRoot) + if ((await hashKnowledgeBase(candidateRoot)) !== expectedHash) { + throw new Error('knowledge candidate snapshot changed during use') + } + return result + } finally { + await rm(isolationRoot, { recursive: true, force: true }) + } +} + +async function assertCandidateEvidence( + runDir: string, + candidate: KnowledgeImprovementCandidateRef, +): Promise { + const evidence = KnowledgeImprovementEvidenceSchema.parse( + JSON.parse( + ( + await readRegularFileWithinRoot( + runDir, + candidateEvidenceRelativePath(candidate.candidateId), + ) + ).bytes.toString('utf8'), + ), + ) + const actualHash = contentHash(evidence) + if (actualHash !== candidate.evidenceHash) { + throw new Error( + `knowledge candidate evidence changed after approval: expected ${candidate.evidenceHash}, got ${actualHash}`, + ) + } + if ( + evidence.runId !== candidate.runId || + evidence.candidateId !== candidate.candidateId || + evidence.goalHash !== candidate.goalHash || + evidence.baseHash !== candidate.baseHash || + evidence.candidateHash !== candidate.candidateHash || + evidence.promotionPlanHash !== candidate.promotionPlanHash || + evidence.evaluation.passed !== true + ) { + throw new Error('knowledge candidate evidence does not match the approved candidate') + } + return evidence +} + +function assertStateIdentity( + root: string, + candidateRef: KnowledgeImprovementCandidateRef, + state: KnowledgeImprovementRunState, +): void { + if (state.runId !== candidateRef.runId) { + throw new Error('knowledge candidate run identity does not match persisted state') + } + if (resolve(state.root) !== resolve(root)) { + throw new Error('knowledge candidate root does not match persisted state') + } + if (sha256(state.goal) !== candidateRef.goalHash) { + throw new Error('knowledge candidate goal does not match persisted state') + } + if (state.baseHash !== candidateRef.baseHash) { + throw new Error('knowledge candidate base does not match persisted state') + } +} + +async function assertCandidateWorkspace( + runDir: string, + candidate: Pick, +): Promise { + await withCandidateWorkspace(runDir, candidate, () => undefined) +} + +async function withCandidateWorkspace( + runDir: string, + candidate: Pick, + use: (candidateRoot: string) => Promise | T, +): Promise { + return withSafeDirectory( + runDir, + join('candidates', safePathSegmentSchema.parse(candidate.candidateId), 'workspace'), + false, + use, + ) +} + function assertKnowledgeImprovementOptions(options: KnowledgeImprovementOptions): void { if (options.step && options.knowledgeResearch?.step) { throw new Error('improveKnowledgeBase accepts either step or knowledgeResearch.step, not both') @@ -395,54 +1231,117 @@ function assertKnowledgeImprovementOptions(options: KnowledgeImprovementOptions) } } -async function runCandidateLifecycle( - runDir: string, +async function measureCandidate( runId: string, + runDir: string, + state: KnowledgeImprovementRunState, candidate: KnowledgeImprovementCandidateRecord, options: KnowledgeImprovementOptions, now: () => Date, -): Promise { - const lifecycles: RunRagKnowledgeImprovementLoopResult[] = [] - - if (shouldRunUpdateStage(options)) { - const updateLifecycle = await runRagKnowledgeImprovementLoop({ - goal: options.goal, - acquireKnowledge: options.acquireKnowledge, - knowledgeResearch: candidateKnowledgeResearchOptions(candidate.candidateRoot, options), - updateKnowledge: candidateUpdateHook(runId, candidate, options), - enabledPhases: selectedStagePhases(options, UPDATE_PHASES), - requiredPhases: selectedStageRequiredPhases(options, UPDATE_PHASES), - signal: options.signal, - now, - }) - lifecycles.push(updateLifecycle) - recordLifecycle(candidate, 'candidate-update', updateLifecycle) - } +): Promise<{ + candidate: KnowledgeImprovementCandidateRecord + evaluation: KnowledgeImprovementMetric + lifecycle?: RunRagKnowledgeImprovementLoopResult +}> { + return withCandidateWorkspace(runDir, candidate, async (candidateRoot) => { + const currentCandidateHash = await hashKnowledgeBase(candidateRoot) + if ( + candidate.status === 'candidate-ready' && + candidate.candidateHash === currentCandidateHash && + candidate.evidenceHash !== undefined && + candidate.promotionPlanHash !== undefined + ) { + const evidence = await assertCandidateEvidence( + runDir, + candidateRefFor(runId, state, candidate), + ) + return { candidate, evaluation: evidence.evaluation } + } - if (shouldRunEvaluationStage(options)) { - const candidateIndex = await buildKnowledgeIndex(candidate.candidateRoot) - const evaluationLifecycle = await runRagKnowledgeImprovementLoop({ - goal: options.goal, - retrieval: options.retrieval - ? { - ...options.retrieval, - index: candidateIndex, - runDir: options.retrieval.runDir ?? join(runDir, 'retrieval', candidate.candidateId), - } - : undefined, - diagnose: options.diagnose, - evaluateAnswers: options.evaluateAnswers, - promote: options.decidePromotion, - enabledPhases: selectedStagePhases(options, EVALUATION_PHASES), - requiredPhases: selectedStageRequiredPhases(options, EVALUATION_PHASES), - signal: options.signal, - now, + clearCandidateMeasurement(candidate) + const lifecycles: RunRagKnowledgeImprovementLoopResult[] = [] + if (candidate.status === 'running') { + const updateLifecycle = await runCandidateUpdateLifecycle( + runId, + candidate, + candidateRoot, + options, + now, + ) + if (updateLifecycle) lifecycles.push(updateLifecycle) + } + return withFrozenCandidateWorkspace(runDir, candidate, candidateRoot, async (snapshot) => { + const evaluationLifecycle = await runCandidateEvaluationLifecycle( + runDir, + candidate, + snapshot.root, + options, + now, + ) + if (evaluationLifecycle) lifecycles.push(evaluationLifecycle) + const lifecycle = mergeLifecycleResults(options.goal, lifecycles) + const measured = await evaluateCandidate( + runDir, + state, + candidate, + snapshot, + lifecycle, + options, + now, + ) + return { ...measured, ...(lifecycle ? { lifecycle } : {}) } }) - lifecycles.push(evaluationLifecycle) - recordLifecycle(candidate, 'candidate-evaluation', evaluationLifecycle) - } + }) +} + +async function runCandidateUpdateLifecycle( + runId: string, + candidate: KnowledgeImprovementCandidateRecord, + candidateRoot: string, + options: KnowledgeImprovementOptions, + now: () => Date, +): Promise { + if (!shouldRunUpdateStage(options)) return undefined + const lifecycle = await runRagKnowledgeImprovementLoop({ + goal: options.goal, + acquireKnowledge: options.acquireKnowledge, + knowledgeResearch: candidateKnowledgeResearchOptions(candidateRoot, options), + updateKnowledge: candidateUpdateHook(runId, candidate, candidateRoot, options), + enabledPhases: selectedStagePhases(options, UPDATE_PHASES), + requiredPhases: selectedStageRequiredPhases(options, UPDATE_PHASES), + signal: options.signal, + now, + }) + return lifecycle +} - return mergeLifecycleResults(options.goal, lifecycles) +async function runCandidateEvaluationLifecycle( + runDir: string, + candidate: KnowledgeImprovementCandidateRecord, + candidateRoot: string, + options: KnowledgeImprovementOptions, + now: () => Date, +): Promise { + if (!shouldRunEvaluationStage(options)) return undefined + const candidateIndex = await buildKnowledgeIndex(candidateRoot) + const lifecycle = await runRagKnowledgeImprovementLoop({ + goal: options.goal, + retrieval: options.retrieval + ? { + ...options.retrieval, + index: candidateIndex, + runDir: options.retrieval.runDir ?? join(runDir, 'retrieval', candidate.candidateId), + } + : undefined, + diagnose: options.diagnose, + evaluateAnswers: options.evaluateAnswers, + promote: options.decidePromotion, + enabledPhases: selectedStagePhases(options, EVALUATION_PHASES), + requiredPhases: selectedStageRequiredPhases(options, EVALUATION_PHASES), + signal: options.signal, + now, + }) + return lifecycle } function candidateKnowledgeResearchOptions( @@ -468,6 +1367,7 @@ function candidateKnowledgeResearchOptions( function candidateUpdateHook( runId: string, candidate: KnowledgeImprovementCandidateRecord, + candidateRoot: string, options: KnowledgeImprovementOptions, ): RunRagKnowledgeImprovementLoopOptions['updateKnowledge'] { if (!options.updateKnowledge) return undefined @@ -477,9 +1377,9 @@ function candidateUpdateHook( runId, iteration: candidate.iteration, candidateId: candidate.candidateId, - root: candidate.candidateRoot, + root: candidateRoot, baselineRoot: options.root, - candidateRoot: candidate.candidateRoot, + candidateRoot, baseHash: candidate.baseHash, }) } @@ -523,26 +1423,6 @@ function selectedStageRequiredPhases( return (options.requiredPhases ?? []).filter((phase) => stagePhases.includes(phase)) } -function recordLifecycle( - candidate: KnowledgeImprovementCandidateRecord, - stage: KnowledgeImprovementLifecycleRecord['stage'], - lifecycle: RunRagKnowledgeImprovementLoopResult, -): void { - const record: KnowledgeImprovementLifecycleRecord = { - stage, - phases: lifecycle.phases, - findingCount: lifecycle.findings.length, - retrievalWinnerConfig: lifecycle.retrieval?.winnerConfig, - answerQuality: lifecycle.answerQuality, - promotionDecision: lifecycle.promotion, - } - candidate.lifecycle = [...(candidate.lifecycle ?? []), record] - candidate.retrievalWinnerConfig = - lifecycle.retrieval?.winnerConfig ?? candidate.retrievalWinnerConfig - candidate.answerQuality = lifecycle.answerQuality ?? candidate.answerQuality - candidate.promotionDecision = lifecycle.promotion ?? candidate.promotionDecision -} - function mergeLifecycleResults( goal: string, lifecycles: readonly RunRagKnowledgeImprovementLoopResult[], @@ -571,59 +1451,97 @@ async function evaluateCandidate( runDir: string, state: KnowledgeImprovementRunState, candidate: KnowledgeImprovementCandidateRecord, + snapshot: { root: string; hash: string }, lifecycle: RunRagKnowledgeImprovementLoopResult | undefined, options: KnowledgeImprovementOptions, now: () => Date, -): Promise { - const [baselineIndex, candidateIndex] = await Promise.all([ - buildKnowledgeIndex(options.root), - buildKnowledgeIndex(candidate.candidateRoot), - ]) - const validation = validateKnowledgeIndex(candidateIndex, { strict: options.strict }) - const readiness = readinessFor(options, candidateIndex) - const kbQuality = scoreKnowledgeBaseIndex(candidateIndex, { - strict: options.strict, - ...options.kbQuality, - }) - const candidateHash = await hashKnowledgeBase(candidate.candidateRoot) - const metric = - options.evaluate?.({ - runId: state.runId, - iteration: candidate.iteration, - root: options.root, - baselineRoot: options.root, - candidateRoot: candidate.candidateRoot, - baselineIndex, - candidateIndex, - baseHash: state.baseHash, - candidateHash, - validation, - readiness, - kbQuality, - lifecycle, - signal: options.signal, - }) ?? - defaultKnowledgeImprovementMetric( - validation, - readiness, - options.readinessSpecs, - kbQuality, - lifecycle, +): Promise<{ + candidate: KnowledgeImprovementCandidateRecord + evaluation: KnowledgeImprovementMetric +}> { + return withBaselineSnapshot(runDir, state.baseHash, async (baselineRoot) => { + const [baselineIndex, candidateIndex] = await Promise.all([ + buildKnowledgeIndex(baselineRoot), + buildKnowledgeIndex(snapshot.root), + ]) + const validation = validateKnowledgeIndex(candidateIndex, { strict: options.strict }) + const readiness = readinessFor(options, candidateIndex) + const kbQuality = scoreKnowledgeBaseIndex(candidateIndex, { + strict: options.strict, + ...options.kbQuality, + }) + const candidateHash = snapshot.hash + const metric = + options.evaluate?.({ + runId: state.runId, + iteration: candidate.iteration, + root: options.root, + baselineRoot, + candidateRoot: snapshot.root, + baselineIndex, + candidateIndex, + baseHash: state.baseHash, + candidateHash, + validation, + readiness, + kbQuality, + lifecycle, + signal: options.signal, + }) ?? + defaultKnowledgeImprovementMetric( + validation, + readiness, + options.readinessSpecs, + kbQuality, + lifecycle, + ) + const evaluation = applyLifecycleFailures(normalizeMetric(await metric), lifecycle) + const measuredHash = await hashKnowledgeBase(snapshot.root) + if (measuredHash !== candidateHash) { + throw new Error( + `knowledge candidate changed during evaluation: expected ${candidateHash}, got ${measuredHash}`, + ) + } + candidate.candidateHash = candidateHash + candidate.promotionPlanHash = knowledgeFileTransactionPlanHash( + await promotionPlanEntries(baselineRoot, snapshot.root), ) - candidate.validation = validation - candidate.kbQuality = kbQuality - candidate.readinessBlockingMissing = readiness?.report.blockingMissingRequirements.length - candidate.evaluation = applyLifecycleFailures(normalizeMetric(await metric), lifecycle) - candidate.candidateHash = candidateHash - candidate.updatedAt = now().toISOString() - await appendLedger(runDir, { - type: 'candidate.evaluated', - runId: state.runId, - candidateId: candidate.candidateId, - score: candidate.evaluation.score, - passed: candidate.evaluation.passed, + const evidence = KnowledgeImprovementEvidenceSchema.parse( + JSON.parse( + JSON.stringify({ + schemaVersion: 1, + kind: 'knowledge-improvement-evidence', + runId: state.runId, + candidateId: candidate.candidateId, + iteration: candidate.iteration, + goalHash: sha256(state.goal), + baseHash: candidate.baseHash, + candidateHash, + promotionPlanHash: candidate.promotionPlanHash, + validation, + readiness: readiness ?? null, + kbQuality, + evaluation, + lifecycle: lifecycle ?? null, + }), + ), + ) + candidate.evidenceHash = contentHash(evidence) + candidate.updatedAt = now().toISOString() + await writeJsonDurableWithinRoot( + runDir, + candidateEvidenceRelativePath(candidate.candidateId), + evidence, + ) + await appendLedger(runDir, { + type: 'candidate.evaluated', + runId: state.runId, + candidateId: candidate.candidateId, + score: evaluation.score, + passed: evaluation.passed, + }) + return { candidate, evaluation } }) - return candidate } function defaultKnowledgeImprovementMetric( @@ -654,12 +1572,6 @@ function defaultKnowledgeImprovementMetric( blockingMissing === 0 ? undefined : `${blockingMissing}/${blockingTotal} blocking knowledge requirements still missing`, - lifecycle?.answerQuality && !lifecycle.answerQuality.passed - ? 'answer quality failed' - : undefined, - lifecycle?.promotion && !lifecycle.promotion.promoted - ? `promotion decision held: ${lifecycle.promotion.reason}` - : undefined, ].filter((reason): reason is string => Boolean(reason)) return { score: average(Object.values(dimensions)), @@ -667,6 +1579,11 @@ function defaultKnowledgeImprovementMetric( dimensions, notes: failedReasons.length === 0 ? 'candidate passed configured checks' : failedReasons.join('; '), + provenance: { + evaluator: '@tangle-network/agent-knowledge/default-knowledge-improvement-metric', + version: '1', + method: 'deterministic', + }, } } @@ -694,10 +1611,7 @@ function applyLifecycleFailures( } function normalizeMetric(metric: KnowledgeImprovementMetric): KnowledgeImprovementMetric { - if (!Number.isFinite(metric.score) || metric.score < 0 || metric.score > 1) { - throw new Error(`knowledge improvement score must be in [0, 1], got ${String(metric.score)}`) - } - return { ...metric, passed: Boolean(metric.passed) } + return improvementMetricSchema.parse(metric) } async function createCandidateWorkspace( @@ -708,13 +1622,12 @@ async function createCandidateWorkspace( ): Promise { const iteration = state.candidates.length + 1 const candidateId = stableId('kcand', `${state.runId}:${iteration}:${now().toISOString()}`) - const candidateRoot = join(runDir, 'candidates', candidateId, 'workspace') + const candidateRoot = candidateWorkspacePath(runDir, candidateId) await copyKnowledgeWorkspace(root, candidateRoot) const createdAt = now().toISOString() return { iteration, candidateId, - candidateRoot, baseHash: state.baseHash, status: 'running', createdAt, @@ -722,6 +1635,127 @@ async function createCandidateWorkspace( } } +function candidateWorkspacePath(runDir: string, candidateId: string): string { + return join(runDir, 'candidates', safePathSegmentSchema.parse(candidateId), 'workspace') +} + +function baselineSnapshotPath(runDir: string): string { + return join(runDir, 'baseline') +} + +async function createBaselineSnapshot( + runDir: string, + root: string, + expectedHash: string, +): Promise { + const target = baselineSnapshotPath(runDir) + try { + await assertBaselineSnapshot(runDir, expectedHash) + return + } catch (error) { + if (!isMissingFile(error)) throw error + } + const preparation = await mkdtemp(join(runDir, 'baseline-prepare-')) + let activated = false + try { + await copyKnowledgeWorkspace(root, preparation) + const actualHash = await hashKnowledgeBase(preparation) + if (actualHash !== expectedHash) { + throw new Error( + `knowledge base changed while baseline was frozen: expected ${expectedHash}, got ${actualHash}`, + ) + } + await renameDurable(preparation, target) + activated = true + } finally { + if (!activated) await rm(preparation, { recursive: true, force: true }) + } +} + +async function ensureBaselineSnapshot( + runDir: string, + root: string, + expectedHash: string, +): Promise { + try { + await assertBaselineSnapshot(runDir, expectedHash) + } catch (error) { + if (!isMissingFile(error)) throw error + const liveHash = await hashKnowledgeBase(root) + if (liveHash !== expectedHash) { + throw new Error( + 'knowledge improvement baseline snapshot is missing and cannot be reconstructed', + ) + } + await createBaselineSnapshot(runDir, root, expectedHash) + } +} + +async function assertBaselineSnapshot(runDir: string, expectedHash: string): Promise { + await withBaselineSnapshot(runDir, expectedHash, () => undefined) +} + +async function withBaselineSnapshot( + runDir: string, + expectedHash: string, + use: (baselineRoot: string) => Promise | T, +): Promise { + return withSafeDirectory(runDir, 'baseline', false, async (baselineRoot) => { + const actualHash = await hashKnowledgeBase(baselineRoot) + if (actualHash !== expectedHash) { + throw new Error( + `knowledge improvement baseline changed: expected ${expectedHash}, got ${actualHash}`, + ) + } + return use(baselineRoot) + }) +} + +async function withFrozenCandidateWorkspace( + runDir: string, + candidate: KnowledgeImprovementCandidateRecord, + candidateRoot: string, + use: (snapshot: { root: string; hash: string }) => Promise | T, +): Promise { + const snapshotsPath = join( + 'candidates', + safePathSegmentSchema.parse(candidate.candidateId), + 'snapshots', + ) + return withSafeDirectory(runDir, snapshotsPath, true, async (snapshotsDir) => { + const preparation = await mkdtemp(join(snapshotsDir, 'prepare-')) + let activated = false + try { + await copyKnowledgeWorkspace(candidateRoot, preparation) + const hash = await hashKnowledgeBase(preparation) + try { + const result = await withSafeDirectory(snapshotsDir, hash, false, async (existing) => { + if ((await hashKnowledgeBase(existing)) !== hash) { + throw new Error('knowledge candidate snapshot does not match its content identity') + } + return use({ root: existing, hash }) + }) + await rm(preparation, { recursive: true, force: true }) + activated = true + return result + } catch (error) { + if (!isMissingFile(error)) throw error + } + await renameDurable(preparation, join(snapshotsDir, hash)) + activated = true + return withSafeDirectory(snapshotsDir, hash, false, (root) => use({ root, hash })) + } finally { + if (!activated) await rm(preparation, { recursive: true, force: true }) + } + }) +} + +function clearCandidateMeasurement(candidate: KnowledgeImprovementCandidateRecord): void { + delete candidate.candidateHash + delete candidate.evidenceHash + delete candidate.promotionPlanHash +} + function findActiveCandidate( state: KnowledgeImprovementRunState, ): KnowledgeImprovementCandidateRecord | undefined { @@ -732,7 +1766,8 @@ function findActiveCandidate( async function copyKnowledgeWorkspace(sourceRoot: string, targetRoot: string): Promise { await rm(targetRoot, { recursive: true, force: true }) - await initKnowledgeBase(targetRoot) + await mkdir(join(targetRoot, 'knowledge'), { recursive: true }) + await mkdir(join(targetRoot, 'raw', 'sources'), { recursive: true }) await copyIfExists(join(sourceRoot, 'knowledge'), join(targetRoot, 'knowledge')) await copyIfExists(join(sourceRoot, 'raw'), join(targetRoot, 'raw')) await copyIfExists( @@ -742,123 +1777,231 @@ async function copyKnowledgeWorkspace(sourceRoot: string, targetRoot: string): P await writeKnowledgeIndex(targetRoot) } -async function promoteCandidate(root: string, candidateRoot: string): Promise { - await copyIfExists(join(candidateRoot, 'knowledge'), join(root, 'knowledge'), { replace: true }) - await copyIfExists(join(candidateRoot, 'raw'), join(root, 'raw'), { replace: true }) - await copyIfExists( - join(layoutFor(candidateRoot).cacheDir, 'sources.json'), - join(layoutFor(root).cacheDir, 'sources.json'), - { replace: true }, +function promotionTransactionPurpose(candidate: KnowledgeImprovementCandidateRef): string { + return `knowledge-promotion:${contentHash(candidate)}` +} + +async function promotionPlanEntries( + baselineRoot: string, + candidateRoot: string, +): Promise { + const [before, after] = await Promise.all([ + knowledgeHashEntries(baselineRoot), + knowledgeHashEntries(candidateRoot), + ]) + const beforeByPath = new Map(before.map((entry) => [entry.path, entry])) + const afterByPath = new Map(after.map((entry) => [entry.path, entry])) + const paths = [ + ...new Set([...before.map((entry) => entry.path), ...after.map((entry) => entry.path)]), + ].sort((left, right) => left.localeCompare(right)) + return paths.map((path) => { + assertKnowledgeMutationPath(path) + const beforeEntry = beforeByPath.get(path) + const afterEntry = afterByPath.get(path) + return { + path, + beforeHash: beforeEntry?.transactionHash ?? null, + afterHash: afterEntry?.transactionHash ?? null, + ...(beforeEntry ? { beforeMode: beforeEntry.mode } : {}), + ...(afterEntry ? { afterMode: afterEntry.mode } : {}), + } + }) +} + +async function promotionMutations( + candidateRoot: string, + plan: readonly KnowledgeFileTransactionPlanEntry[], +): Promise { + return Promise.all( + plan.map(async (entry) => { + if (entry.afterHash === null) return { path: entry.path, content: null } + const file = await readRegularFileWithinRoot(candidateRoot, entry.path) + const actualHash = createHash('sha256').update(file.bytes).digest('hex') + if (actualHash !== entry.afterHash || file.mode !== entry.afterMode) { + throw new Error(`knowledge candidate file changed before promotion: ${entry.path}`) + } + return { path: entry.path, content: file.bytes, mode: file.mode } + }), ) - await writeKnowledgeIndex(root) } -async function copyIfExists( - source: string, - target: string, - options: { replace?: boolean } = {}, -): Promise { - try { - await stat(source) - } catch { - return +function assertPromotionTransaction( + transaction: KnowledgeFileTransaction, + candidate: KnowledgeImprovementCandidateRef, +): void { + const actualPlanHash = knowledgeFileTransactionPlanHash(transaction.entries) + if (actualPlanHash !== candidate.promotionPlanHash) { + throw new Error( + `knowledge promotion plan changed after approval: expected ${candidate.promotionPlanHash}, got ${actualPlanHash}`, + ) } - if (options.replace) await rm(target, { recursive: true, force: true }) - await mkdir(dirname(target), { recursive: true }) - await cp(source, target, { recursive: true }) } -export async function hashKnowledgeBase(root: string): Promise { - const entries: Array<{ path: string; hash: string }> = [] - for (const rel of ['knowledge', 'raw']) { - entries.push(...(await hashTreeEntries(root, rel))) +async function ensurePromotionEvent( + runDir: string, + candidateRef: KnowledgeImprovementCandidateRef, +): Promise { + if (await hasPromotionEvent(runDir, candidateRef)) return + await appendLedger(runDir, { + type: 'candidate.promoted', + runId: candidateRef.runId, + candidateId: candidateRef.candidateId, + candidateHash: candidateRef.candidateHash, + evidenceHash: candidateRef.evidenceHash, + promotionPlanHash: candidateRef.promotionPlanHash, + }) +} + +async function hasPromotionEvent( + runDir: string, + candidateRef: KnowledgeImprovementCandidateRef, +): Promise { + let matched = false + for (const row of await loadKnowledgeImprovementEventsFromRun(runDir)) { + if (row.type !== 'candidate.promoted' || row.candidateId !== candidateRef.candidateId) continue + if ( + row.runId !== candidateRef.runId || + row.candidateHash !== candidateRef.candidateHash || + row.evidenceHash !== candidateRef.evidenceHash || + row.promotionPlanHash !== candidateRef.promotionPlanHash + ) { + throw new Error('persisted knowledge promotion event conflicts with the approved candidate') + } + matched = true } - const sourceRegistry = relative(root, layoutFor(root).sourceRegistryPath).replace(/\\/g, '/') - entries.push(...(await hashFileEntry(root, sourceRegistry))) - entries.sort((a, b) => a.path.localeCompare(b.path)) - return sha256(JSON.stringify(entries)) + return matched +} + +export interface KnowledgeImprovementEvent extends Record { + at: string + type: string } -async function hashTreeEntries( +export async function loadKnowledgeImprovementEvents( root: string, - relDir: string, -): Promise> { - const abs = join(root, relDir) + runId: string, +): Promise { + const parsedRunId = runIdSchema.parse(runId) try { - const s = await stat(abs) - if (!s.isDirectory()) return [] - } catch { - return [] - } - const out: Array<{ path: string; hash: string }> = [] - const entries = await readdir(abs, { withFileTypes: true }) - for (const entry of entries) { - const rel = join(relDir, entry.name).replace(/\\/g, '/') - if (entry.isDirectory()) out.push(...(await hashTreeEntries(root, rel))) - else if (entry.isFile()) out.push(...(await hashFileEntry(root, rel))) + return await withKnowledgeImprovementRun(root, parsedRunId, false, (runDir) => + loadKnowledgeImprovementEventsFromRun(runDir), + ) + } catch (error) { + if (isMissingFile(error)) return [] + throw error } - return out } -async function hashFileEntry( - root: string, - rel: string, -): Promise> { +async function loadKnowledgeImprovementEventsFromRun( + runDir: string, +): Promise { + const events: KnowledgeImprovementEvent[] = [] try { - const bytes = await readFile(join(root, rel)) - return [{ path: rel, hash: sha256(bytes.toString('base64')) }] - } catch { - return [] + for (const file of await listRegularFilesWithinRoot(runDir, 'events')) { + const name = file.path.slice('events/'.length) + if (name.includes('/') || !name.endsWith('.json')) { + throw new Error(`knowledge event store contains an unsupported entry: ${name}`) + } + events.push(parseKnowledgeImprovementEvent(JSON.parse(file.bytes.toString('utf8')))) + } + } catch (error) { + if (!isMissingFile(error)) throw error + } + + const unique = new Map() + for (const event of events) { + const { at: _at, ...semantic } = event + unique.set(contentHash(semantic), event) } + return [...unique.values()].sort((left, right) => left.at.localeCompare(right.at)) } -async function acquireRunLease( - runDir: string, - options: { ownerId: string; ttlMs: number; now: () => Date }, -): Promise { - const path = join(runDir, 'run.lock') - const now = options.now() - const expiresAt = new Date(now.getTime() + options.ttlMs) - const payload: LeaseFile = { - ownerId: options.ownerId, - acquiredAt: now.toISOString(), - expiresAt: expiresAt.toISOString(), - pid: process.pid, +function parseKnowledgeImprovementEvent(value: unknown): KnowledgeImprovementEvent { + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new Error('knowledge improvement event is not an object') + } + const event = value as Record + if (typeof event.at !== 'string' || typeof event.type !== 'string') { + throw new Error('knowledge improvement event is missing at or type') } + return event as KnowledgeImprovementEvent +} + +async function copyIfExists(source: string, target: string): Promise { + let sourceStat: Awaited> try { - const handle = await open(path, 'wx') + sourceStat = await lstat(source) + } catch (error) { + if (isMissingFile(error)) return + throw error + } + if (!sourceStat.isDirectory() && !sourceStat.isFile()) { + throw new Error(`knowledge surface contains an unsupported filesystem entry: ${source}`) + } + await mkdir(dirname(target), { recursive: true }) + await cp(source, target, { recursive: sourceStat.isDirectory(), dereference: false }) +} + +export async function hashKnowledgeBase(root: string): Promise { + return withKnowledgeRead(root, () => hashKnowledgeBaseUnlocked(root)) +} + +async function hashKnowledgeBaseUnlocked(root: string): Promise { + const entries = await knowledgeHashEntries(root) + return sha256(JSON.stringify(entries.map(({ path, hash, mode }) => ({ path, hash, mode })))) +} + +interface KnowledgeFileIdentity { + path: string + hash: string + transactionHash: string + mode: number +} + +async function knowledgeHashEntries(root: string): Promise { + const entries: KnowledgeFileIdentity[] = [] + for (const rel of ['knowledge', 'raw']) { try { - await handle.writeFile(`${JSON.stringify(payload, null, 2)}\n`, 'utf8') - } finally { - await handle.close() + for (const file of await listRegularFilesWithinRoot(root, rel)) { + entries.push(knowledgeFileIdentity(file.path, file.bytes, file.mode)) + } + } catch (error) { + if (!isMissingFile(error)) throw error } + } + const sourceRegistry = relative(root, layoutFor(root).sourceRegistryPath).replace(/\\/g, '/') + try { + const file = await readRegularFileWithinRoot(root, sourceRegistry) + entries.push(knowledgeFileIdentity(sourceRegistry, file.bytes, file.mode)) } catch (error) { - const code = (error as NodeJS.ErrnoException).code - if (code !== 'EEXIST') throw error - const existing = await readLease(path) - if (existing && new Date(existing.expiresAt).getTime() > now.getTime()) { - throw new Error( - `knowledge improvement run is locked by ${existing.ownerId} until ${existing.expiresAt}`, - ) - } - await rm(path, { force: true }) - return acquireRunLease(runDir, options) + if (!isMissingFile(error)) throw error } + entries.sort((a, b) => a.path.localeCompare(b.path)) + return entries +} + +function knowledgeFileIdentity(path: string, bytes: Buffer, mode: number): KnowledgeFileIdentity { return { - ownerId: options.ownerId, path, - async release() { - const current = await readLease(path) - if (!current || current.ownerId === options.ownerId) await rm(path, { force: true }) - }, + hash: sha256(bytes.toString('base64')), + transactionHash: createHash('sha256').update(bytes).digest('hex'), + mode, } } -async function readLease(path: string): Promise { - try { - return JSON.parse(await readFile(path, 'utf8')) as LeaseFile - } catch { - return null +async function acquireRunLease( + runDir: string, + options: { ownerId: string; ttlMs: number }, +): Promise { + const path = join(runDir, 'run.lock.durable') + const acquired = await acquireDurableFileLock(runDir, { + lockfilePath: path, + staleMs: options.ttlMs, + }) + return { + ownerId: options.ownerId, + assertOwned: acquired.assertOwned, + release: acquired.release, } } @@ -881,25 +2024,55 @@ async function saveState( state: KnowledgeImprovementRunState, onState?: KnowledgeImprovementOptions['onState'], ): Promise { - await writeJsonAtomic(statePath(runDir), state) + await writeJsonDurableWithinRoot( + runDir, + 'state.json', + KnowledgeImprovementRunStateSchema.parse(state), + ) await onState?.(state) } async function appendLedger(runDir: string, value: Record): Promise { - const row = { at: new Date().toISOString(), ...value } - await mkdir(runDir, { recursive: true }) - await writeFile(join(runDir, 'events.jsonl'), `${JSON.stringify(row)}\n`, { flag: 'a' }) + const type = value.type + if (typeof type !== 'string' || type.length === 0) { + throw new Error('knowledge improvement event requires a type') + } + const relativePath = join('events', `${contentHash(value)}.json`).replace(/\\/g, '/') + try { + const file = await readRegularFileWithinRoot(runDir, relativePath) + const existing = parseKnowledgeImprovementEvent(JSON.parse(file.bytes.toString('utf8'))) + const { at: _at, ...semantic } = existing + if (canonicalJson(semantic) !== canonicalJson(value)) { + throw new Error('knowledge improvement event identity conflicts with durable content') + } + return + } catch (error) { + if (!isMissingFile(error)) throw error + } + await writeJsonDurableWithinRoot(runDir, relativePath, { + at: new Date().toISOString(), + ...value, + }) } -async function writeJsonAtomic(path: string, value: unknown): Promise { - await mkdir(dirname(path), { recursive: true }) - const tmp = `${path}.${process.pid}.${Date.now()}.tmp` - await writeFile(tmp, `${JSON.stringify(value, null, 2)}\n`, 'utf8') - await rename(tmp, path) +function candidateEvidenceRelativePath(candidateId: string): string { + return join('candidates', safePathSegmentSchema.parse(candidateId), 'evidence.json').replace( + /\\/g, + '/', + ) +} + +function descendantPath(root: string, path: string): string | undefined { + const value = relative(resolve(root), resolve(path)).replace(/\\/g, '/') + if (value === '' || value === '..' || value.startsWith('../') || isAbsolute(value)) + return undefined + return value } -function statePath(runDir: string): string { - return join(runDir, 'state.json') +function assertExactCandidatePlatform(): void { + if (process.platform !== 'linux') { + throw new Error('exact knowledge candidate workflows require Linux directory descriptors') + } } function average(values: readonly number[]): number { diff --git a/src/kb-store.ts b/src/kb-store.ts index e3135ed..0ae813d 100644 --- a/src/kb-store.ts +++ b/src/kb-store.ts @@ -1,7 +1,14 @@ -import { mkdir, readFile, writeFile } from 'node:fs/promises' -import { dirname, join } from 'node:path' +import { z } from 'zod' +import { readRegularFileWithinRoot, writeJsonDurableWithinRoot } from './durable-fs' import type { KnowledgeEventQuery } from './events' import { buildKnowledgeGraph } from './graph' +import { withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' +import { + KnowledgeEventSchema, + KnowledgeIndexSchema, + KnowledgePageSchema, + SourceRecordSchema, +} from './schemas' import type { KnowledgeEvent, KnowledgeIndex, KnowledgePage, SourceRecord } from './types' export interface KbStore { @@ -81,34 +88,132 @@ export class MemoryKbStore implements KbStore { } } -export class FileSystemKbStore extends MemoryKbStore { - constructor(private readonly dir: string) { - super() +const knowledgeEventsSchema = z.array(KnowledgeEventSchema) + +export class FileSystemKbStore implements KbStore { + constructor(private readonly dir: string) {} + + async putSource(source: SourceRecord): Promise { + const parsed = SourceRecordSchema.parse(source) as SourceRecord + await this.updateIndex((index) => ({ + ...index, + generatedAt: new Date().toISOString(), + sources: [parsed, ...index.sources.filter((entry) => entry.id !== parsed.id)], + })) + } + + async getSource(id: string): Promise { + return withKnowledgeRead(this.dir, async () => { + const index = await this.readIndex() + return clone(index?.sources.find((source) => source.id === id) ?? null) + }) + } + + async listSources(): Promise { + return withKnowledgeRead(this.dir, async () => clone((await this.readIndex())?.sources ?? [])) + } + + async putPage(page: KnowledgePage): Promise { + const parsed = KnowledgePageSchema.parse(page) as KnowledgePage + await this.updateIndex((index) => { + const pages = [parsed, ...index.pages.filter((entry) => entry.id !== parsed.id)] + return { + ...index, + generatedAt: new Date().toISOString(), + pages, + graph: buildKnowledgeGraph(pages), + } + }) + } + + async getPage(idOrPath: string): Promise { + return withKnowledgeRead(this.dir, async () => { + const index = await this.readIndex() + return clone( + index?.pages.find((page) => page.id === idOrPath || page.path === idOrPath) ?? null, + ) + }) + } + + async listPages(): Promise { + return withKnowledgeRead(this.dir, async () => clone((await this.readIndex())?.pages ?? [])) } async putIndex(index: KnowledgeIndex): Promise { - await super.putIndex(index) - await writeJson(join(this.dir, 'index.json'), index) + const parsed = KnowledgeIndexSchema.parse(index) as KnowledgeIndex + await withKnowledgeMutation(this.dir, () => + writeJsonDurableWithinRoot(this.dir, 'index.json', parsed), + ) } async getIndex(): Promise { - try { - return JSON.parse(await readFile(join(this.dir, 'index.json'), 'utf8')) as KnowledgeIndex - } catch { - return await super.getIndex() - } + return withKnowledgeRead(this.dir, () => this.readIndex()) } async putEvent(event: KnowledgeEvent): Promise { - await super.putEvent(event) - const events = await this.listEvents() - await writeJson(join(this.dir, 'events.json'), events) + const parsed = KnowledgeEventSchema.parse(event) as KnowledgeEvent + await withKnowledgeMutation(this.dir, async () => { + const current = await this.readEvents() + const next = [...current.filter((entry) => entry.id !== parsed.id), parsed].sort((a, b) => + a.createdAt.localeCompare(b.createdAt), + ) + await writeJsonDurableWithinRoot(this.dir, 'events.json', next) + }) + } + + async listEvents(query: KnowledgeEventQuery = {}): Promise { + return withKnowledgeRead(this.dir, async () => { + let events = await this.readEvents() + if (query.type) events = events.filter((event) => event.type === query.type) + if (query.target) events = events.filter((event) => event.target === query.target) + return clone(events.slice(-(query.limit ?? events.length))) + }) + } + + private async updateIndex(change: (index: KnowledgeIndex) => KnowledgeIndex): Promise { + await withKnowledgeMutation(this.dir, async () => { + const current = (await this.readIndex()) ?? emptyIndex(this.dir) + const next = KnowledgeIndexSchema.parse(change(current)) as KnowledgeIndex + await writeJsonDurableWithinRoot(this.dir, 'index.json', next) + }) + } + + private async readIndex(): Promise { + return readJsonFile( + this.dir, + 'index.json', + KnowledgeIndexSchema, + ) as Promise + } + + private async readEvents(): Promise { + return ((await readJsonFile(this.dir, 'events.json', knowledgeEventsSchema)) ?? + []) as KnowledgeEvent[] } } -async function writeJson(path: string, value: unknown): Promise { - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, 'utf8') +function emptyIndex(root: string): KnowledgeIndex { + return { + root, + generatedAt: new Date(0).toISOString(), + sources: [], + pages: [], + graph: { nodes: [], edges: [] }, + } +} + +async function readJsonFile( + root: string, + relativePath: string, + schema: z.ZodType, +): Promise { + try { + const file = await readRegularFileWithinRoot(root, relativePath) + return schema.parse(JSON.parse(file.bytes.toString('utf8'))) + } catch (error) { + if ((error as NodeJS.ErrnoException | null)?.code === 'ENOENT') return null + throw error + } } function clone(value: T): T { diff --git a/src/mutation-lock.ts b/src/mutation-lock.ts new file mode 100644 index 0000000..115e037 --- /dev/null +++ b/src/mutation-lock.ts @@ -0,0 +1,361 @@ +import { AsyncLocalStorage } from 'node:async_hooks' +import { mkdir, readFile } from 'node:fs/promises' +import { dirname, join, resolve } from 'node:path' +import { check, type LockOptions, lock } from 'proper-lockfile' +import { + isMissingFile, + withSafeDescendant, + withSafeDirectory, + writeJsonDurableWithinRoot, +} from './durable-fs' +import { + inspectKnowledgeFileTransaction, + type KnowledgeFileTransaction, + loadKnowledgeFileTransaction, + recoverKnowledgeFileTransaction, +} from './file-transaction' + +const DEFAULT_STALE_MS = 15 * 60 * 1000 +const DEFAULT_READ_RETRIES = 100 +const DEFAULT_READ_WAIT_MS = 25 + +interface KnowledgeMutationScope { + active: boolean + lock: KnowledgeMutationLock +} + +const activeRoots = new AsyncLocalStorage>() +const activeReadRoots = new AsyncLocalStorage>() + +export interface KnowledgeMutationLock { + readonly transactionRoot: string + assertOwned(): void +} + +export class KnowledgeLockLostError extends Error { + constructor(message: string, options?: { cause?: unknown }) { + super(message, options) + this.name = 'KnowledgeLockLostError' + } +} + +interface DurableFileLock { + assertOwned(): void + release(): Promise +} + +export interface KnowledgeMutationOptions { + staleMs?: number + resumeTransaction?: { + purpose: string + validate?: (transaction: KnowledgeFileTransaction) => void + direction?: 'apply' | 'rollback' + } + retries?: LockOptions['retries'] +} + +export interface PendingKnowledgeMutation { + transactionId: string + purpose: string + createdAt: string + direction: 'apply' | 'rollback' + paths: string[] +} + +export interface RecoverPendingKnowledgeMutationOptions { + transactionId: string + action: 'apply' | 'rollback' +} + +export interface KnowledgeReadOptions { + staleMs?: number + retries?: number + waitMs?: number +} + +export async function withKnowledgeMutation( + root: string, + mutate: (lock: KnowledgeMutationLock) => Promise | T, + options: KnowledgeMutationOptions = {}, +): Promise { + const resolvedRoot = resolve(root) + const active = activeRoots.getStore() + const existing = active?.get(resolvedRoot) + if (existing?.active) { + existing.lock.assertOwned() + const result = await mutate(existing.lock) + existing.lock.assertOwned() + return result + } + + return withSafeDirectory(resolvedRoot, '.agent-knowledge', true, async (cacheDir) => { + const acquired = await acquireDurableFileLock(resolvedRoot, { + lockfilePath: join(cacheDir, 'mutation.lock.durable'), + staleMs: options.staleMs, + retries: + options.retries ?? + ({ retries: 100, factor: 1.1, minTimeout: 10, maxTimeout: 200, randomize: true } as const), + }) + const mutationLock: KnowledgeMutationLock = { + transactionRoot: join(cacheDir, 'file-transactions'), + assertOwned: acquired.assertOwned, + } + const scope: KnowledgeMutationScope = { active: true, lock: mutationLock } + try { + const pending = await loadKnowledgeFileTransaction({ + root: resolvedRoot, + transactionRoot: mutationLock.transactionRoot, + }) + if (pending) { + const resume = options.resumeTransaction + if (!resume || pending.purpose !== resume.purpose) { + throw new Error(`knowledge transaction '${pending.purpose}' requires its owner to resume`) + } + } + const locks = new Map(active) + locks.set(resolvedRoot, scope) + return await activeRoots.run(locks, async () => { + const epoch = await beginMutationEpoch(cacheDir) + let completed = false + try { + if (pending) { + const resume = options.resumeTransaction + if (!resume) + throw new Error( + `knowledge transaction '${pending.purpose}' requires its owner to resume`, + ) + await recoverKnowledgeFileTransaction({ + root: resolvedRoot, + transactionRoot: mutationLock.transactionRoot, + expectedPurpose: resume.purpose, + direction: resume.direction, + validate: resume.validate, + assertOwned: acquired.assertOwned, + }) + } + mutationLock.assertOwned() + const result = await mutate(mutationLock) + mutationLock.assertOwned() + completed = true + return result + } finally { + mutationLock.assertOwned() + const stillPending = await loadKnowledgeFileTransaction({ + root: resolvedRoot, + transactionRoot: mutationLock.transactionRoot, + }) + if (completed || !stillPending) await finishMutationEpoch(cacheDir, epoch) + } + }) + } finally { + scope.active = false + await acquired.release() + } + }) +} + +export async function inspectPendingKnowledgeMutation( + root: string, +): Promise { + const resolvedRoot = resolve(root) + const pending = await inspectKnowledgeFileTransaction({ + root: resolvedRoot, + transactionRoot: join(resolvedRoot, '.agent-knowledge', 'file-transactions'), + }) + if (!pending) return null + const { transaction, direction } = pending + return { + transactionId: transaction.transactionId, + purpose: transaction.purpose, + createdAt: transaction.createdAt, + direction, + paths: transaction.entries.map((entry) => entry.path), + } +} + +export async function recoverPendingKnowledgeMutation( + root: string, + options: RecoverPendingKnowledgeMutationOptions, +): Promise { + const pending = await inspectPendingKnowledgeMutation(root) + if (!pending) throw new Error('knowledge base has no pending mutation') + + let matched = false + await withKnowledgeMutation( + root, + () => { + if (!matched) throw new Error('pending knowledge mutation changed before recovery') + }, + { + resumeTransaction: { + purpose: pending.purpose, + direction: options.action, + validate(transaction) { + if (transaction.transactionId !== options.transactionId) { + throw new Error(`pending knowledge mutation does not match '${options.transactionId}'`) + } + matched = true + }, + }, + }, + ) +} + +export async function withKnowledgeRead( + root: string, + read: () => Promise | T, + options: KnowledgeReadOptions = {}, +): Promise { + const resolvedRoot = resolve(root) + if (activeRoots.getStore()?.get(resolvedRoot)?.active) return await read() + if (activeReadRoots.getStore()?.has(resolvedRoot)) return await read() + + const readRoots = new Set(activeReadRoots.getStore()) + readRoots.add(resolvedRoot) + return activeReadRoots.run(readRoots, async () => { + const retries = options.retries ?? DEFAULT_READ_RETRIES + for (let attempt = 0; attempt <= retries; attempt += 1) { + const before = await readMutationEpoch(resolvedRoot) + if (isOdd(before)) { + await waitForActiveMutationEpoch(resolvedRoot, before, options) + continue + } + + const result = await read() + const after = await readMutationEpoch(resolvedRoot) + if (before === after && !isOdd(after)) return result + if (isOdd(after)) await waitForActiveMutationEpoch(resolvedRoot, after, options) + } + throw new Error('knowledge read could not observe a stable mutation epoch') + }) +} + +export async function acquireDurableFileLock( + target: string, + options: { lockfilePath: string; staleMs?: number; retries?: LockOptions['retries'] }, +): Promise { + const stale = Math.max(5_000, options.staleMs ?? DEFAULT_STALE_MS) + const update = Math.max(1_000, Math.floor(stale / 3)) + let compromised: Error | undefined + let released = false + await mkdir(dirname(options.lockfilePath), { recursive: true }) + const release = await lock(target, { + lockfilePath: options.lockfilePath, + realpath: false, + stale, + update, + retries: options.retries, + onCompromised(error) { + compromised = error + }, + }) + + return { + assertOwned() { + if (released) throw new KnowledgeLockLostError('knowledge filesystem lock is no longer owned') + if (compromised) + throw new KnowledgeLockLostError('knowledge filesystem lock was compromised', { + cause: compromised, + }) + }, + async release() { + if (released) return + released = true + let releaseError: unknown + try { + await release() + } catch (error) { + releaseError = error + } + if (releaseError && compromised) { + throw new AggregateError( + [releaseError, compromised], + 'knowledge filesystem lock was compromised and could not be released', + ) + } + if (releaseError) throw releaseError + if (compromised) { + throw new KnowledgeLockLostError('knowledge filesystem lock was compromised', { + cause: compromised, + }) + } + }, + } +} + +async function beginMutationEpoch(cacheDir: string): Promise { + const current = await readMutationEpochFromCache(cacheDir) + const odd = isOdd(current) ? current : current + 1 + await writeMutationEpoch(cacheDir, odd) + return odd +} + +async function finishMutationEpoch(cacheDir: string, odd: number): Promise { + await writeMutationEpoch(cacheDir, isOdd(odd) ? odd + 1 : odd) +} + +async function readMutationEpoch(root: string): Promise { + try { + return await withSafeDirectory(root, '.agent-knowledge', false, readMutationEpochFromCache) + } catch (error) { + if (isMissingFile(error)) return 0 + throw error + } +} + +async function readMutationEpochFromCache(cacheDir: string): Promise { + try { + const text = await withSafeDescendant(cacheDir, 'mutation-epoch.json', (path) => + readFile(path, 'utf8'), + ) + const parsed = JSON.parse(text) as { epoch?: unknown } + const epoch = parsed.epoch + if (typeof epoch !== 'number' || !Number.isSafeInteger(epoch) || epoch < 0) { + throw new Error('knowledge mutation epoch is invalid') + } + return epoch + } catch (error) { + if (isMissingFile(error)) return 0 + throw error + } +} + +async function writeMutationEpoch(cacheDir: string, epoch: number): Promise { + await writeJsonDurableWithinRoot(cacheDir, 'mutation-epoch.json', { + epoch, + updatedAt: new Date().toISOString(), + }) +} + +async function waitForActiveMutationEpoch( + root: string, + epoch: number, + options: KnowledgeReadOptions, +): Promise { + if (!(await hasActiveMutationLock(root, options))) { + throw new Error(`knowledge mutation epoch ${epoch} is odd with no active writer`) + } + await new Promise((resolve) => setTimeout(resolve, options.waitMs ?? DEFAULT_READ_WAIT_MS)) +} + +async function hasActiveMutationLock( + root: string, + options: Pick, +): Promise { + try { + return await withSafeDirectory(root, '.agent-knowledge', false, (cacheDir) => + check(root, { + lockfilePath: join(cacheDir, 'mutation.lock.durable'), + realpath: false, + stale: Math.max(5_000, options.staleMs ?? DEFAULT_STALE_MS), + }), + ) + } catch (error) { + if (isMissingFile(error)) return false + throw error + } +} + +function isOdd(value: number): boolean { + return value % 2 === 1 +} diff --git a/src/proposals.ts b/src/proposals.ts index 1fdac61..bfec395 100644 --- a/src/proposals.ts +++ b/src/proposals.ts @@ -1,5 +1,7 @@ -import { mkdir, readFile, writeFile } from 'node:fs/promises' -import { dirname, join } from 'node:path' +import { readFile } from 'node:fs/promises' +import { contentHash } from '@tangle-network/agent-eval' +import { commitKnowledgeFileMutations } from './file-transaction' +import { withKnowledgeMutation } from './mutation-lock' import { parseKnowledgeWriteBlocks } from './write-protocol' export interface ApplyWriteBlocksResult { @@ -12,18 +14,26 @@ export async function applyKnowledgeWriteBlocks( proposalText: string, ): Promise { const parsed = parseKnowledgeWriteBlocks(proposalText) - const written: string[] = [] - for (const block of parsed.blocks) { - const path = join(root, block.path) - await mkdir(dirname(path), { recursive: true }) - await writeFile( - path, - block.content.endsWith('\n') ? block.content : `${block.content}\n`, - 'utf8', - ) - written.push(block.path) - } - return { written, warnings: parsed.warnings } + const purpose = `knowledge-proposal:${contentHash(parsed.blocks)}` + return withKnowledgeMutation( + root, + async (lock) => { + if (parsed.blocks.length > 0) { + await commitKnowledgeFileMutations({ + root, + transactionRoot: lock.transactionRoot, + purpose, + mutations: parsed.blocks.map((block) => ({ + path: block.path, + content: block.content.endsWith('\n') ? block.content : `${block.content}\n`, + })), + assertOwned: lock.assertOwned, + }) + } + return { written: parsed.blocks.map((block) => block.path), warnings: parsed.warnings } + }, + { resumeTransaction: { purpose } }, + ) } export async function applyKnowledgeWriteBlocksFile( diff --git a/src/retrieval-eval.ts b/src/retrieval-eval.ts index babfa04..14da2c9 100644 --- a/src/retrieval-eval.ts +++ b/src/retrieval-eval.ts @@ -60,6 +60,7 @@ export interface RetrievalEvalArtifact { requestedK: number hits: readonly RetrievedKnowledgeHit[] durationMs: number + /** Informational copy. Billable retrievers account through context.cost.runPaidCall. */ costUsd?: number metadata?: Record } @@ -87,6 +88,7 @@ export interface RetrievalEvalRetrieverInput { export interface RetrievalEvalRetrieverResult { hits: readonly RetrievedKnowledgeHit[] + /** Informational copy. Billable retrievers account through context.cost.runPaidCall. */ costUsd?: number metadata?: Record } @@ -219,7 +221,6 @@ export function buildRetrievalEvalDispatch(options: BuildRetrievalEvalDispatchOp context, }) const normalized = normalizeRetrieverResult(result) - observeRetrievalCost(context, normalized.costUsd) return { config, query: scenario.query, @@ -540,18 +541,6 @@ async function defaultSearchRetriever( return { hits: results.map(hitFromSearchResult) } } -function observeRetrievalCost(context: DispatchContext, costUsd: number | undefined): void { - if (costUsd === undefined) { - return - } - if (!Number.isFinite(costUsd) || costUsd < 0) { - throw new Error( - `retrieval costUsd must be a non-negative finite number, got ${String(costUsd)}`, - ) - } - context.cost.observe(costUsd, 'agent-knowledge:retrieval') -} - function hitFromSearchResult(result: KnowledgeSearchResult): RetrievedKnowledgeHit { return { pageId: result.page.id, diff --git a/src/schemas.ts b/src/schemas.ts index e370ee2..fbb3660 100644 --- a/src/schemas.ts +++ b/src/schemas.ts @@ -21,6 +21,8 @@ export const SourceRecordSchema = z.object({ contentHash: z.string().min(16), text: z.string().optional(), anchors: z.array(SourceAnchorSchema).optional(), + validUntil: z.iso.datetime().optional(), + lastVerifiedAt: z.iso.datetime().optional(), metadata: z.record(z.string(), z.unknown()).optional(), createdAt: z.string().min(1), }) diff --git a/src/sources.ts b/src/sources.ts index 736a926..e7eff1a 100644 --- a/src/sources.ts +++ b/src/sources.ts @@ -1,10 +1,22 @@ -import { copyFile, mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises' -import { basename, dirname, join, relative } from 'node:path' +import { lstat, readdir, readFile } from 'node:fs/promises' +import { basename, join, relative } from 'node:path' +import { z } from 'zod' import { type SourceAdapter, textSourceAdapter } from './adapters' +import { readRegularFileWithinRoot, writeJsonDurableWithinRoot } from './durable-fs' +import { commitKnowledgeFileMutations, type KnowledgeFileMutation } from './file-transaction' import { sha256, slugify, stableId } from './ids' +import { withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' +import { SourceRecordSchema } from './schemas' import { layoutFor } from './store' import type { SourceRecord, SourceRegistry } from './types' +const sourceRegistrySchema = z + .object({ + generatedAt: z.string().min(1), + sources: z.array(SourceRecordSchema.passthrough()), + }) + .strict() + export interface AddSourceOptions { copyIntoRaw?: boolean adapters?: SourceAdapter[] @@ -22,23 +34,29 @@ export interface AddSourceTextInput { } export async function loadSourceRegistry(root: string): Promise { - const path = sourceRegistryPath(root) + return withKnowledgeRead(root, () => loadSourceRegistryUnlocked(root)) +} + +async function loadSourceRegistryUnlocked(root: string): Promise { try { - const parsed = JSON.parse(await readFile(path, 'utf8')) as SourceRegistry - return { - generatedAt: - typeof parsed.generatedAt === 'string' ? parsed.generatedAt : new Date(0).toISOString(), - sources: Array.isArray(parsed.sources) ? parsed.sources : [], + const file = await readRegularFileWithinRoot(root, '.agent-knowledge/sources.json') + return sourceRegistrySchema.parse(JSON.parse(file.bytes.toString('utf8'))) as SourceRegistry + } catch (error) { + if ((error as NodeJS.ErrnoException | null)?.code === 'ENOENT') { + return { generatedAt: new Date(0).toISOString(), sources: [] } } - } catch { - return { generatedAt: new Date(0).toISOString(), sources: [] } + throw error } } export async function writeSourceRegistry(root: string, registry: SourceRegistry): Promise { - const path = sourceRegistryPath(root) - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, `${JSON.stringify(registry, null, 2)}\n`, 'utf8') + await withKnowledgeMutation(root, async () => { + await writeJsonDurableWithinRoot( + root, + '.agent-knowledge/sources.json', + sourceRegistrySchema.parse(registry), + ) + }) } export async function addSourcePath( @@ -46,17 +64,29 @@ export async function addSourcePath( sourcePath: string, options: AddSourceOptions = {}, ): Promise { - const s = await stat(sourcePath) - if (s.isDirectory()) { - const out: SourceRecord[] = [] - for (const file of await listFiles(sourcePath)) { - out.push(...(await addSourcePath(root, file, options))) - } - return out + const sourceStat = await lstat(sourcePath) + if (sourceStat.isDirectory()) { + const files = await listSourceFiles(sourcePath) + const prepared = await Promise.all( + files.map((file) => preparePathSource(root, file.path, file.mode, options)), + ) + return commitSourceBatch(root, prepared, options) } + if (!sourceStat.isFile()) throw new Error(`unsupported source path: ${sourcePath}`) - const layout = layoutFor(root) - await mkdir(layout.rawSourcesDir, { recursive: true }) + return commitSourceBatch( + root, + [await preparePathSource(root, sourcePath, sourceStat.mode & 0o777, options)], + options, + ) +} + +async function preparePathSource( + root: string, + sourcePath: string, + mode: number, + options: AddSourceOptions, +): Promise { const bytes = await readFile(sourcePath) const contentHash = sha256(bytes.toString('base64')) const fileName = basename(sourcePath) @@ -64,41 +94,29 @@ export async function addSourcePath( const adapter = adapters.find((candidate) => candidate.canLoad({ uri: sourcePath, bytes })) const loaded = adapter ? await adapter.load({ uri: sourcePath, bytes }) : {} const id = stableId('src', `${contentHash}:${fileName}`) - const targetRel = join( - 'raw', - 'sources', - `${slugify(fileName.replace(/\.[^.]+$/, ''))}-${contentHash.slice(0, 8)}${ext(fileName)}`, - ).replace(/\\/g, '/') - const targetAbs = join(root, targetRel) - - if (options.copyIntoRaw ?? true) { - await mkdir(dirname(targetAbs), { recursive: true }) - await copyFile(sourcePath, targetAbs) - } + const targetRel = rawSourcePath(fileName, contentHash, ext(fileName)) + const copyIntoRaw = options.copyIntoRaw ?? true - const existing = await loadSourceRegistry(root) - const record: SourceRecord = { - id, - uri: targetRel, - title: loaded.title ?? fileName, - mediaType: loaded.mediaType ?? mediaTypeFor(fileName), - contentHash, - text: loaded.text ?? textPreview(fileName, bytes), - anchors: (loaded.anchors ?? []).map((anchor) => ({ ...anchor, sourceId: id })), - createdAt: (options.now ?? (() => new Date()))().toISOString(), - metadata: { - ...(loaded.metadata ?? {}), - originalPath: sourcePath, - sizeBytes: bytes.length, - projectRelativePath: relative(root, sourcePath).replace(/\\/g, '/'), + return { + record: { + id, + uri: copyIntoRaw ? targetRel : sourcePath, + title: loaded.title ?? fileName, + mediaType: loaded.mediaType ?? mediaTypeFor(fileName), + contentHash, + text: loaded.text ?? textPreview(fileName, bytes), + anchors: (loaded.anchors ?? []).map((anchor) => ({ ...anchor, sourceId: id })), + createdAt: (options.now ?? (() => new Date()))().toISOString(), + metadata: { + ...(loaded.metadata ?? {}), + originalPath: sourcePath, + sizeBytes: bytes.length, + projectRelativePath: relative(root, sourcePath).replace(/\\/g, '/'), + }, }, + mutation: copyIntoRaw ? { path: targetRel, content: bytes, mode } : null, + purposeParts: [sourcePath, contentHash, copyIntoRaw, mode], } - const next: SourceRegistry = { - generatedAt: new Date().toISOString(), - sources: [record, ...existing.sources.filter((source) => source.id !== id)], - } - await writeSourceRegistry(root, next) - return [record] } export async function addSourceText( @@ -106,6 +124,14 @@ export async function addSourceText( input: AddSourceTextInput, options: Pick = {}, ): Promise { + const [record] = await commitSourceBatch(root, [await prepareTextSource(input, options)], options) + return record! +} + +async function prepareTextSource( + input: AddSourceTextInput, + options: Pick, +): Promise { const text = input.text const contentHash = sha256(text) const fileName = basename(input.uri) || `${slugify(input.title ?? input.uri)}.txt` @@ -115,56 +141,130 @@ export async function addSourceText( ) const loaded = adapter ? await adapter.load(adapterInput) : {} const id = stableId('src', `${contentHash}:${input.uri}`) - const targetRel = join( - 'raw', - 'sources', - `${slugify(fileName.replace(/\.[^.]+$/, ''))}-${contentHash.slice(0, 8)}.txt`, - ).replace(/\\/g, '/') - const targetAbs = join(root, targetRel) - await mkdir(dirname(targetAbs), { recursive: true }) - await writeFile(targetAbs, text.endsWith('\n') ? text : `${text}\n`, 'utf8') - - const existing = await loadSourceRegistry(root) - const record: SourceRecord = { - id, - uri: targetRel, - title: input.title ?? loaded.title ?? fileName, - mediaType: input.mediaType ?? loaded.mediaType ?? 'text/plain', - contentHash, - text: loaded.text ?? text.slice(0, 200_000), - anchors: (loaded.anchors ?? []).map((anchor) => ({ ...anchor, sourceId: id })), - validUntil: input.validUntil, - lastVerifiedAt: input.lastVerifiedAt, - createdAt: (options.now ?? (() => new Date()))().toISOString(), - metadata: { - ...(loaded.metadata ?? {}), - ...(input.metadata ?? {}), - originalUri: input.uri, - sizeBytes: Buffer.byteLength(text, 'utf8'), + const targetRel = rawSourcePath(fileName, contentHash, '.txt') + const rawContent = text.endsWith('\n') ? text : `${text}\n` + + return { + record: { + id, + uri: targetRel, + title: input.title ?? loaded.title ?? fileName, + mediaType: input.mediaType ?? loaded.mediaType ?? 'text/plain', + contentHash, + text: loaded.text ?? text.slice(0, 200_000), + anchors: (loaded.anchors ?? []).map((anchor) => ({ ...anchor, sourceId: id })), + ...(input.validUntil === undefined ? {} : { validUntil: input.validUntil }), + ...(input.lastVerifiedAt === undefined ? {} : { lastVerifiedAt: input.lastVerifiedAt }), + createdAt: (options.now ?? (() => new Date()))().toISOString(), + metadata: { + ...(loaded.metadata ?? {}), + ...(input.metadata ?? {}), + originalUri: input.uri, + sizeBytes: Buffer.byteLength(text, 'utf8'), + }, }, + mutation: { path: targetRel, content: rawContent }, + purposeParts: [input.uri, contentHash], } - await writeSourceRegistry(root, { - generatedAt: new Date().toISOString(), - sources: [record, ...existing.sources.filter((source) => source.id !== id)], - }) - return record +} + +interface PreparedSourceImport { + record: SourceRecord + mutation: KnowledgeFileMutation | null + purposeParts: readonly unknown[] +} + +async function commitSourceBatch( + root: string, + prepared: readonly PreparedSourceImport[], + options: Pick, +): Promise { + if (prepared.length === 0) return [] + const purpose = `source-import:${sha256(JSON.stringify(prepared.map((item) => item.purposeParts)))}` + return withKnowledgeMutation( + root, + async (lock) => { + const existing = await loadSourceRegistry(root) + const records = uniqueSources(prepared.map((item) => item.record)) + const next = sourceRegistrySchema.parse({ + generatedAt: (options.now ?? (() => new Date()))().toISOString(), + sources: [ + ...records, + ...existing.sources.filter( + (source) => !records.some((record) => record.id === source.id), + ), + ], + }) as SourceRegistry + await commitKnowledgeFileMutations({ + root, + transactionRoot: lock.transactionRoot, + purpose, + mutations: [ + ...uniqueMutations(prepared.flatMap((item) => (item.mutation ? [item.mutation] : []))), + { + path: relative(root, sourceRegistryPath(root)).replace(/\\/g, '/'), + content: `${JSON.stringify(next, null, 2)}\n`, + }, + ], + assertOwned: lock.assertOwned, + }) + return records + }, + { resumeTransaction: { purpose } }, + ) } export function sourceRegistryPath(root: string): string { return join(layoutFor(root).cacheDir, 'sources.json') } -async function listFiles(root: string): Promise { +function uniqueSources(records: readonly SourceRecord[]): SourceRecord[] { + const byId = new Map() + for (const record of records) byId.set(record.id, record) + return [...byId.values()] +} + +function uniqueMutations(mutations: readonly KnowledgeFileMutation[]): KnowledgeFileMutation[] { + const byPath = new Map() + for (const mutation of mutations) { + const existing = byPath.get(mutation.path) + if (existing && !sameMutation(existing, mutation)) { + throw new Error(`source import produced conflicting raw path: ${mutation.path}`) + } + byPath.set(mutation.path, mutation) + } + return [...byPath.values()] +} + +function sameMutation(left: KnowledgeFileMutation, right: KnowledgeFileMutation): boolean { + if (left.mode !== right.mode) return false + if (Buffer.isBuffer(left.content) && Buffer.isBuffer(right.content)) { + return left.content.equals(right.content) + } + return left.content === right.content +} + +async function listSourceFiles(root: string): Promise> { const entries = await readdir(root, { withFileTypes: true }) - const out: string[] = [] + const out: Array<{ path: string; mode: number }> = [] for (const entry of entries) { const full = join(root, entry.name) - if (entry.isDirectory()) out.push(...(await listFiles(full))) - else if (entry.isFile()) out.push(full) + const entryStat = await lstat(full) + if (entryStat.isDirectory()) out.push(...(await listSourceFiles(full))) + else if (entryStat.isFile()) out.push({ path: full, mode: entryStat.mode & 0o777 }) + else throw new Error(`unsupported directory entry in source import: ${full}`) } return out } +function rawSourcePath(fileName: string, contentHash: string, extension: string): string { + return join( + 'raw', + 'sources', + `${slugify(fileName.replace(/\.[^.]+$/, ''))}-${contentHash.slice(0, 8)}${extension}`, + ).replace(/\\/g, '/') +} + function ext(fileName: string): string { const idx = fileName.lastIndexOf('.') return idx >= 0 ? fileName.slice(idx) : '' diff --git a/src/store.ts b/src/store.ts index b9273b3..43b033c 100644 --- a/src/store.ts +++ b/src/store.ts @@ -1,7 +1,15 @@ -import { mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises' -import { dirname, join, relative } from 'node:path' +import { join } from 'node:path' +import { + isMissingFile, + listRegularFilesWithinRoot, + readRegularFileWithinRoot, + withSafeDirectory, + writeFileDurableWithinRoot, + writeJsonDurable, +} from './durable-fs' import { parseFrontmatter } from './frontmatter' import { slugify } from './ids' +import { withKnowledgeMutation, withKnowledgeRead } from './mutation-lock' import type { KnowledgePage } from './types' import { extractWikilinks, normalizeLinkTarget } from './wikilinks' @@ -55,26 +63,43 @@ export function isScaffoldPath(path: string): boolean { export async function initKnowledgeBase(root: string): Promise { const layout = layoutFor(root) - await mkdir(layout.knowledgeDir, { recursive: true }) - await mkdir(layout.rawSourcesDir, { recursive: true }) - await mkdir(layout.cacheDir, { recursive: true }) - await writeIfMissing(layout.indexPath, '# Knowledge Index\n\n') - await writeIfMissing(layout.logPath, '# Knowledge Log\n\n') + if (await knowledgeBaseInitialized(layout)) return layout + return withKnowledgeMutation(root, () => initKnowledgeBaseUnlocked(root)) +} + +async function initKnowledgeBaseUnlocked(root: string): Promise { + const layout = layoutFor(root) + await withSafeDirectory(root, 'knowledge', true, () => undefined) + await withSafeDirectory(root, 'raw/sources', true, () => undefined) + await withSafeDirectory(root, '.agent-knowledge', true, () => undefined) + await writeIfMissing(root, 'knowledge/index.md', '# Knowledge Index\n\n') + await writeIfMissing(root, 'knowledge/log.md', '# Knowledge Log\n\n') await writeIfMissing( - layout.sourceRegistryPath, + root, + '.agent-knowledge/sources.json', '{\n "generatedAt": "1970-01-01T00:00:00.000Z",\n "sources": []\n}\n', ) return layout } export async function loadKnowledgePages(root: string): Promise { - const layout = layoutFor(root) - const files = await listMarkdownFiles(layout.knowledgeDir) + return withKnowledgeRead(root, () => loadKnowledgePagesUnlocked(root)) +} + +async function loadKnowledgePagesUnlocked(root: string): Promise { + let files: Awaited> + try { + files = await listRegularFilesWithinRoot(root, 'knowledge') + } catch (error) { + if (isMissingFile(error)) return [] + throw error + } const pages: KnowledgePage[] = [] for (const file of files) { - const rel = relative(root, file).replace(/\\/g, '/') + const rel = file.path + if (!rel.endsWith('.md')) continue if (isScaffoldPath(rel)) continue - const content = await readFile(file, 'utf8') + const content = file.bytes.toString('utf8') const { frontmatter, body } = parseFrontmatter(content) const title = stringField(frontmatter.title) ?? @@ -100,31 +125,30 @@ export async function loadKnowledgePages(root: string): Promise } export async function writeJson(path: string, value: unknown): Promise { - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, 'utf8') + await writeJsonDurable(path, value) } -async function writeIfMissing(path: string, content: string): Promise { +async function writeIfMissing(root: string, relativePath: string, content: string): Promise { try { - await stat(path) - } catch { - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, content, 'utf8') + await readRegularFileWithinRoot(root, relativePath) + } catch (error) { + if (!isMissingFile(error)) throw error + await writeFileDurableWithinRoot(root, relativePath, content, { encoding: 'utf8' }) } } -async function listMarkdownFiles(root: string): Promise { +async function knowledgeBaseInitialized(layout: KnowledgeLayout): Promise { try { - const entries = await readdir(root, { withFileTypes: true }) - const out: string[] = [] - for (const entry of entries) { - const full = join(root, entry.name) - if (entry.isDirectory()) out.push(...(await listMarkdownFiles(full))) - else if (entry.isFile() && entry.name.endsWith('.md')) out.push(full) - } - return out - } catch { - return [] + await withSafeDirectory(layout.root, 'knowledge', false, () => undefined) + await withSafeDirectory(layout.root, 'raw/sources', false, () => undefined) + await withSafeDirectory(layout.root, '.agent-knowledge', false, () => undefined) + await readRegularFileWithinRoot(layout.root, 'knowledge/index.md') + await readRegularFileWithinRoot(layout.root, 'knowledge/log.md') + await readRegularFileWithinRoot(layout.root, '.agent-knowledge/sources.json') + return true + } catch (error) { + if (isMissingFile(error)) return false + throw error } } diff --git a/tests/benchmarks.test.ts b/tests/benchmarks.test.ts index 4f573cd..6df91a8 100644 --- a/tests/benchmarks.test.ts +++ b/tests/benchmarks.test.ts @@ -76,14 +76,30 @@ describe('knowledge benchmark adapters', () => { cases, runDir: '/runs/knowledge-benchmark-smoke', storage, - respond: ({ case: testCase }) => ({ - costUsd: 0.01, - hits: [ - testCase.id.endsWith('q1') - ? { pageId: 'page-1', path: 'knowledge/page-1.md', rank: 1 } - : { pageId: 'miss', path: 'knowledge/miss.md', rank: 1 }, - ], - }), + respond: async ({ case: testCase, context }) => { + const paid = await context.cost.runPaidCall({ + actor: 'benchmark-retriever', + model: 'retriever-fixture', + maximumCharge: { externallyEnforcedMaximumUsd: 0.01 }, + execute: async () => ({ + costUsd: 0.01, + hits: [ + testCase.id.endsWith('q1') + ? { pageId: 'page-1', path: 'knowledge/page-1.md', rank: 1 } + : { pageId: 'miss', path: 'knowledge/miss.md', rank: 1 }, + ], + }), + receipt: () => ({ + model: 'retriever-fixture', + inputTokens: 0, + outputTokens: 0, + usageUnknown: true, + actualCostUsd: 0.01, + }), + }) + if (!paid.succeeded) throw paid.error + return paid.value + }, }) expect(result.report.totalCases).toBe(2) @@ -166,7 +182,7 @@ describe('knowledge benchmark adapters', () => { expect(result.report.byTaskKind['rag-answer']?.n).toBe(3) expect(result.report.byTaskKind.hallucination?.n).toBe(2) expect(result.report.byTaskKind['kb-improvement']?.n).toBe(1) - expect(result.report.totalCostUsd).toBeCloseTo(INDUSTRY_RAG_BENCHMARKS.length * 0.001) + expect(result.report.totalCostUsd).toBe(0) for (const benchmark of INDUSTRY_RAG_BENCHMARKS) { expect(result.report.byFamily[benchmark.family]?.n).toBeGreaterThanOrEqual(1) } @@ -251,7 +267,7 @@ describe('knowledge benchmark adapters', () => { expect(result.report.byTaskKind['memory-summarization']?.n).toBe(1) expect(result.report.byTaskKind['memory-recommendation']?.n).toBe(1) expect(result.report.byTaskKind['memory-multiparty']?.n).toBe(1) - expect(result.report.totalCostUsd).toBeCloseTo(INDUSTRY_MEMORY_BENCHMARKS.length * 0.001) + expect(result.report.totalCostUsd).toBe(0) for (const benchmark of INDUSTRY_MEMORY_BENCHMARKS) { expect(result.report.byFamily[benchmark.family]?.n).toBeGreaterThanOrEqual(1) } diff --git a/tests/core.test.ts b/tests/core.test.ts index a65af1e..46d9fbb 100644 --- a/tests/core.test.ts +++ b/tests/core.test.ts @@ -1,4 +1,4 @@ -import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises' +import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { runAgentControlLoop } from '@tangle-network/agent-eval' @@ -6,6 +6,7 @@ import { describe, expect, it } from 'vitest' import { addSourcePath, + addSourceText, applyKnowledgeWriteBlocks, buildEvalKnowledgeBundle, buildKnowledgeIndex, @@ -15,6 +16,8 @@ import { createLocalDiscoveryDispatcher, defineReadinessSpec, explainKnowledgeTarget, + FileSystemKbStore, + hashKnowledgeBase, initKnowledgeBase, inspectKnowledgeIndex, KnowledgeIndexSchema, @@ -67,6 +70,109 @@ describe('knowledge write protocol', () => { }) }) +describe('source registry integrity', () => { + it('does not overwrite a malformed registry while adding a source', async () => { + await withProject(async (root) => { + const registryPath = join(root, '.agent-knowledge', 'sources.json') + await writeFile(registryPath, '{broken') + + await expect( + addSourceText(root, { uri: 'memory://new', text: 'new source' }), + ).rejects.toThrow() + await expect(readFile(registryPath, 'utf8')).resolves.toBe('{broken') + }) + }) + + it('rejects an invalid registry without replacing the durable copy', async () => { + await withProject(async (root) => { + const registryPath = join(root, '.agent-knowledge', 'sources.json') + const original = await readFile(registryPath, 'utf8') + + await expect(writeSourceRegistry(root, { generatedAt: '', sources: [] })).rejects.toThrow() + await expect(readFile(registryPath, 'utf8')).resolves.toBe(original) + }) + }) + + it('does not replace a malformed filesystem index with generated data', async () => { + await withProject(async (root) => { + const storeDir = join(root, '.store') + await mkdir(storeDir, { recursive: true }) + await writeFile(join(storeDir, 'index.json'), '{broken') + + await expect(new FileSystemKbStore(storeDir).getIndex()).rejects.toThrow() + await expect(readFile(join(storeDir, 'index.json'), 'utf8')).resolves.toBe('{broken') + + await writeFile(join(storeDir, 'index.json'), '{}') + await expect(new FileSystemKbStore(storeDir).getIndex()).rejects.toThrow() + await expect(readFile(join(storeDir, 'index.json'), 'utf8')).resolves.toBe('{}') + }) + }) + + it('reports a missing filesystem index instead of fabricating an empty one', async () => { + await withProject(async (root) => { + const storeDir = join(root, '.missing-store') + await expect(new FileSystemKbStore(storeDir).getIndex()).resolves.toBeNull() + }) + }) + + it('rejects a knowledge tree redirected through a symbolic link', async () => { + await withProject(async (root) => { + const outside = join(root, 'outside') + await rm(join(root, 'knowledge'), { recursive: true, force: true }) + await mkdir(outside) + await writeFile(join(outside, 'secret.md'), '# Outside Secret\n') + await symlink(outside, join(root, 'knowledge')) + + await expect(buildKnowledgeIndex(root)).rejects.toThrow(/unsafe directory/) + await expect(hashKnowledgeBase(root)).rejects.toThrow(/unsafe directory/) + }) + }) + + it('persists every filesystem store method across instances without lost events', async () => { + await withProject(async (root) => { + const storeDir = join(root, '.store') + const first = new FileSystemKbStore(storeDir) + const second = new FileSystemKbStore(storeDir) + const source = { + id: 'source-one', + uri: 'memory://source-one', + contentHash: '0123456789abcdef', + createdAt: '2026-01-01T00:00:00.000Z', + } + const page = { + id: 'page-one', + path: 'knowledge/page-one.md', + title: 'Page One', + text: 'Persisted page', + frontmatter: {}, + sourceIds: [source.id], + tags: [], + outLinks: [], + } + const eventOne = createKnowledgeEvent({ + type: 'source.added', + target: source.id, + now: () => new Date('2026-01-01T00:00:00.000Z'), + }) + const eventTwo = createKnowledgeEvent({ + type: 'index.built', + target: page.id, + now: () => new Date('2026-01-01T00:00:01.000Z'), + }) + + await first.putSource(source) + await first.putPage(page) + await Promise.all([first.putEvent(eventOne), second.putEvent(eventTwo)]) + + await expect(second.getSource(source.id)).resolves.toMatchObject(source) + await expect(second.getPage(page.path)).resolves.toMatchObject(page) + await expect(second.listSources()).resolves.toHaveLength(1) + await expect(second.listPages()).resolves.toHaveLength(1) + await expect(first.listEvents()).resolves.toEqual([eventOne, eventTwo]) + }) + }) +}) + describe('chunkMarkdown', () => { it('preserves heading breadcrumbs and strips frontmatter', () => { const chunks = chunkMarkdown('---\ntitle: Test\n---\n# Alpha\n\none\n\n## Beta\n\ntwo', { diff --git a/tests/file-transaction.test.ts b/tests/file-transaction.test.ts new file mode 100644 index 0000000..c2a2052 --- /dev/null +++ b/tests/file-transaction.test.ts @@ -0,0 +1,749 @@ +import { createHash } from 'node:crypto' +import { renameSync } from 'node:fs' +import { + chmod, + mkdir, + mkdtemp, + readFile, + rename, + rm, + stat, + symlink, + unlink, + writeFile, +} from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as delay } from 'node:timers/promises' +import { describe, expect, it } from 'vitest' +import { + applyKnowledgeFileTransaction, + finishKnowledgeFileTransaction, + loadKnowledgeFileTransaction, + prepareKnowledgeFileTransaction, + recoverKnowledgeFileTransaction, + rollbackKnowledgeFileTransaction, +} from '../src/file-transaction' +import { buildKnowledgeIndex } from '../src/indexer' +import { KnowledgeLockLostError, withKnowledgeMutation } from '../src/mutation-lock' +import { addSourceText, loadSourceRegistry } from '../src/sources' + +async function withRoot(fn: (root: string) => Promise): Promise { + const root = await mkdtemp(join(tmpdir(), 'agent-knowledge-transaction-')) + try { + await fn(root) + } finally { + await rm(root, { recursive: true, force: true }) + } +} + +describe('knowledge file transactions', () => { + it('resumes an interrupted multi-file commit without mixing versions', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), 'one-before\n') + await writeFile(join(root, 'knowledge', 'two.md'), 'two-before\n') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'interrupted-commit', + mutations: [ + { path: 'knowledge/one.md', content: 'one-after\n' }, + { path: 'knowledge/two.md', content: null }, + { path: 'knowledge/three.md', content: 'three-after\n' }, + ], + }) + expect(prepared).not.toBeNull() + + await expect( + applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + beforeCommit(entry) { + if (entry.path === 'knowledge/two.md') throw new Error('simulated interruption') + }, + }), + ).rejects.toThrow(/simulated interruption/) + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe('one-after\n') + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).resolves.toBe( + 'two-before\n', + ) + + const pending = await loadKnowledgeFileTransaction({ root, transactionRoot }) + expect(pending).not.toBeNull() + await applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: pending!, + }) + await finishKnowledgeFileTransaction({ root, transactionRoot, transaction: pending! }) + + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe('one-after\n') + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).rejects.toMatchObject({ + code: 'ENOENT', + }) + await expect(readFile(join(root, 'knowledge', 'three.md'), 'utf8')).resolves.toBe( + 'three-after\n', + ) + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toBeNull() + }) + }) + + it('resumes an interrupted rollback instead of reapplying the candidate', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), 'one-before\n') + await writeFile(join(root, 'knowledge', 'two.md'), 'two-before\n') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'interrupted-rollback', + mutations: [ + { path: 'knowledge/one.md', content: 'one-after\n' }, + { path: 'knowledge/two.md', content: 'two-after\n' }, + ], + }) + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: prepared! }) + + await expect( + rollbackKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + beforeCommit(entry) { + if (entry.path === 'knowledge/one.md') throw new Error('rollback interrupted') + }, + }), + ).rejects.toThrow(/rollback interrupted/) + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe('one-after\n') + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).resolves.toBe( + 'two-before\n', + ) + await expect( + finishKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }), + ).rejects.toThrow(/did not roll back/) + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toEqual( + prepared, + ) + + await recoverKnowledgeFileTransaction({ + root, + transactionRoot, + expectedPurpose: 'interrupted-rollback', + }) + + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe( + 'one-before\n', + ) + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).resolves.toBe( + 'two-before\n', + ) + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toBeNull() + }) + }) + + it('rejects stale apply, rollback, and finish calls without deleting the active transaction', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + const first = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'first', + mutations: [{ path: 'knowledge/page.md', content: 'first\n' }], + }) + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }) + await finishKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }) + const second = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'second', + mutations: [{ path: 'knowledge/page.md', content: 'second\n' }], + }) + + await expect( + applyKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }), + ).rejects.toThrow(/does not match/) + await expect( + rollbackKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }), + ).rejects.toThrow(/does not match/) + await expect( + finishKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }), + ).rejects.toThrow(/does not match/) + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toEqual(second) + + await expect( + finishKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: second!, + assertOwned() { + throw new KnowledgeLockLostError('stale owner') + }, + }), + ).rejects.toThrow(/stale owner/) + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toEqual(second) + }) + }) + + it('does not delete a successor installed after the finishing ownership check', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + const stagedRoot = join(root, '.staged-transactions') + const first = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'first', + mutations: [{ path: 'knowledge/page.md', content: 'first\n' }], + }) + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: first! }) + const second = await prepareKnowledgeFileTransaction({ + root, + transactionRoot: stagedRoot, + purpose: 'second', + mutations: [{ path: 'knowledge/page.md', content: 'second\n' }], + }) + + await expect( + finishKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: first!, + assertOwned() { + renameSync( + join(transactionRoot, `active-${first!.transactionId}`), + join(transactionRoot, `retired-${first!.transactionId}`), + ) + renameSync( + join(stagedRoot, `active-${second!.transactionId}`), + join(transactionRoot, `active-${second!.transactionId}`), + ) + }, + }), + ).rejects.toThrow(/does not match/) + + await expect(loadKnowledgeFileTransaction({ root, transactionRoot })).resolves.toEqual(second) + await expect(readFile(join(root, 'knowledge', 'page.md'), 'utf8')).resolves.toBe('first\n') + }) + }) + + it('fails loudly on an old active journal instead of starting another transaction', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + await mkdir(join(transactionRoot, 'active'), { recursive: true }) + + await expect( + prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'new', + mutations: [{ path: 'knowledge/page.md', content: 'new\n' }], + }), + ).rejects.toThrow(/unsupported active journal/) + }) + }) + + it('recovers an interrupted aggregate write before the next writer reads it', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.agent-knowledge', 'file-transactions') + await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'interrupted-source', + mutations: [ + { path: 'raw/sources/first.txt', content: 'first source\n' }, + { + path: '.agent-knowledge/sources.json', + content: `${JSON.stringify( + { + generatedAt: '2026-07-12T00:00:00.000Z', + sources: [ + { + id: 'src_first', + uri: 'raw/sources/first.txt', + contentHash: 'a'.repeat(64), + createdAt: '2026-07-12T00:00:00.000Z', + }, + ], + }, + null, + 2, + )}\n`, + }, + ], + }) + + await expect( + addSourceText(root, { + uri: 'test://second', + text: 'second source', + }), + ).rejects.toThrow(/requires its owner to resume/) + + await recoverKnowledgeFileTransaction({ + root, + transactionRoot, + expectedPurpose: 'interrupted-source', + }) + await addSourceText(root, { uri: 'test://second', text: 'second source' }) + + const registry = await loadSourceRegistry(root) + expect(registry.sources.map((source) => source.id)).toContain('src_first') + expect(registry.sources).toHaveLength(2) + await expect(readFile(join(root, 'raw', 'sources', 'first.txt'), 'utf8')).resolves.toBe( + 'first source\n', + ) + }) + }) + + it('rejects a forged path in an ordinary recovery journal', async () => { + await withRoot(async (root) => { + const packagePath = join(root, 'package.json') + const original = '{"private":true}\n' + const forged = '{"scripts":{"postinstall":"malicious"}}\n' + await writeFile(packagePath, original) + const transactionRoot = join(root, '.agent-knowledge', 'file-transactions') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'ordinary-recovery', + mutations: [{ path: 'knowledge/page.md', content: '# Approved\n' }], + }) + expect(prepared).not.toBeNull() + const transactionDir = join(transactionRoot, `active-${prepared!.transactionId}`) + const transactionPath = join(transactionDir, 'transaction.json') + const transaction = JSON.parse(await readFile(transactionPath, 'utf8')) as { + entries: Array<{ + index: number + path: string + beforeHash: string | null + afterHash: string | null + beforeMode?: number + afterMode?: number + }> + } + const index = Math.max(...transaction.entries.map((entry) => entry.index)) + 1 + transaction.entries.push({ + index, + path: 'package.json', + beforeHash: createHash('sha256').update(original).digest('hex'), + afterHash: createHash('sha256').update(forged).digest('hex'), + }) + await mkdir(join(transactionDir, 'after'), { recursive: true }) + await writeFile(join(transactionDir, 'after', `${index}.bin`), forged) + await writeFile(transactionPath, `${JSON.stringify(transaction, null, 2)}\n`) + + await expect(withKnowledgeMutation(root, async () => undefined)).rejects.toThrow( + /unsupported path: package.json/, + ) + await expect(readFile(packagePath, 'utf8')).resolves.toBe(original) + }) + }) + + it('holds an external reader until every file in a transaction is committed', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), '# One before\n') + await writeFile(join(root, 'knowledge', 'two.md'), '# Two before\n') + let startReader!: () => void + const readerStart = new Promise((resolve) => { + startReader = resolve + }) + let readerSettled = false + const reader = readerStart + .then(() => buildKnowledgeIndex(root)) + .finally(() => { + readerSettled = true + }) + + await withKnowledgeMutation(root, async (lock) => { + const transactionRoot = lock.transactionRoot + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'reader-consistency', + mutations: [ + { path: 'knowledge/one.md', content: '# One after\n' }, + { path: 'knowledge/two.md', content: '# Two after\n' }, + ], + }) + expect(prepared).not.toBeNull() + await applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + async beforeCommit(entry) { + if (entry.path !== 'knowledge/two.md') return + startReader() + await delay(50) + expect(readerSettled).toBe(false) + }, + }) + await finishKnowledgeFileTransaction({ root, transactionRoot, transaction: prepared! }) + }) + + const index = await reader + expect(index.pages.map((page) => page.title).sort()).toEqual(['One after', 'Two after']) + }) + }) + + it('rejects a missing or truncated live file instead of overwriting it', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + const target = join(root, 'knowledge', 'page.md') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(target, 'measured-before\n') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'external-change', + mutations: [{ path: 'knowledge/page.md', content: 'approved-after\n' }], + }) + expect(prepared).not.toBeNull() + + await writeFile(target, 'truncated') + await expect( + applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }), + ).rejects.toThrow(/changed outside transaction/) + await unlink(target) + await expect( + applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }), + ).rejects.toThrow(/changed outside transaction/) + }) + }) + + it.skipIf(process.platform === 'win32')('preserves an existing file mode exactly', async () => { + await withRoot(async (root) => { + const target = join(root, 'knowledge', 'page.md') + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(target, 'before\n') + await chmod(target, 0o764) + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'mode-preservation', + mutations: [{ path: 'knowledge/page.md', content: 'after\n' }], + }) + expect(prepared).not.toBeNull() + + await applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }) + expect((await stat(target)).mode & 0o777).toBe(0o764) + }) + }) + + it.skipIf(process.platform === 'win32')( + 'commits a mode-only change when file bytes are unchanged', + async () => { + await withRoot(async (root) => { + const target = join(root, 'knowledge', 'page.md') + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(target, 'same bytes\n') + await chmod(target, 0o600) + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'mode-only-change', + mutations: [{ path: 'knowledge/page.md', content: 'same bytes\n', mode: 0o755 }], + }) + + expect(prepared).not.toBeNull() + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: prepared! }) + expect(await readFile(target, 'utf8')).toBe('same bytes\n') + expect((await stat(target)).mode & 0o777).toBe(0o755) + }) + }, + ) + + it.skipIf(process.platform === 'win32')( + 'restores the original mode when a mode-changing transaction rolls back', + async () => { + await withRoot(async (root) => { + const target = join(root, 'knowledge', 'page.md') + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(target, 'before\n') + await chmod(target, 0o600) + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'mode-rollback', + mutations: [{ path: 'knowledge/page.md', content: 'after\n', mode: 0o777 }], + }) + + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: prepared! }) + expect((await stat(target)).mode & 0o777).toBe(0o777) + await rollbackKnowledgeFileTransaction({ root, transactionRoot, transaction: prepared! }) + expect(await readFile(target, 'utf8')).toBe('before\n') + expect((await stat(target)).mode & 0o777).toBe(0o600) + }) + }, + ) + + it('rejects symbolic links in transaction paths', async () => { + await withRoot(async (root) => { + const outside = join(root, 'outside.md') + await writeFile(outside, 'outside\n') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await symlink(outside, join(root, 'knowledge', 'linked.md')) + + await expect( + prepareKnowledgeFileTransaction({ + root, + transactionRoot: join(root, '.transactions'), + purpose: 'symlink-attack', + mutations: [{ path: 'knowledge/linked.md', content: 'overwrite\n' }], + }), + ).rejects.toThrow(/not a regular file/) + await expect(readFile(outside, 'utf8')).resolves.toBe('outside\n') + }) + }) + + it('rejects a symbolic-link ancestor without writing outside the knowledge root', async () => { + await withRoot(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-outside-')) + try { + await writeFile(join(outside, 'page.md'), 'outside\n') + await symlink(outside, join(root, 'knowledge')) + + await expect( + prepareKnowledgeFileTransaction({ + root, + transactionRoot: join(root, '.transactions'), + purpose: 'ancestor-symlink-attack', + mutations: [{ path: 'knowledge/page.md', content: 'overwrite\n' }], + }), + ).rejects.toThrow(/unsafe directory/) + await expect(readFile(join(outside, 'page.md'), 'utf8')).resolves.toBe('outside\n') + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }) + + it('rejects a symbolic-link metadata directory before acquiring the package lock', async () => { + await withRoot(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-lock-outside-')) + try { + await symlink(outside, join(root, '.agent-knowledge')) + await expect(withKnowledgeMutation(root, async () => undefined)).rejects.toThrow( + /unsafe directory/, + ) + await expect(stat(join(outside, 'mutation.lock.durable.lock'))).rejects.toMatchObject({ + code: 'ENOENT', + }) + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }) + + it.skipIf(process.platform !== 'linux')( + 'keeps an in-flight write on its opened directory when an ancestor is swapped', + async () => { + await withRoot(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-outside-')) + try { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'page.md'), 'before\n') + await writeFile(join(outside, 'page.md'), 'outside\n') + const transactionRoot = join(root, '.transactions') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'ancestor-swap-attack', + mutations: [{ path: 'knowledge/page.md', content: 'after\n' }], + }) + expect(prepared).not.toBeNull() + + let swapped = false + await expect( + applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + async beforeCommit() { + if (swapped) return + swapped = true + await rename(join(root, 'knowledge'), join(root, 'knowledge-original')) + await symlink(outside, join(root, 'knowledge')) + }, + }), + ).rejects.toThrow(/unsafe directory/) + + await expect(readFile(join(outside, 'page.md'), 'utf8')).resolves.toBe('outside\n') + await expect(readFile(join(root, 'knowledge-original', 'page.md'), 'utf8')).resolves.toBe( + 'after\n', + ) + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }, + ) + + it.skipIf(process.platform !== 'linux')( + 'keeps transaction state in its opened metadata directory when that path is swapped', + async () => { + await withRoot(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-metadata-outside-')) + try { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'page.md'), 'before\n') + + await withKnowledgeMutation(root, async (lock) => { + const transaction = await prepareKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + purpose: 'metadata-swap', + mutations: [{ path: 'knowledge/page.md', content: 'after\n' }], + }) + expect(transaction).not.toBeNull() + + await rename(join(root, '.agent-knowledge'), join(root, '.agent-knowledge-original')) + await symlink(outside, join(root, '.agent-knowledge')) + + await applyKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + transaction: transaction!, + }) + await finishKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + transaction: transaction!, + }) + }) + + await expect(readFile(join(root, 'knowledge', 'page.md'), 'utf8')).resolves.toBe( + 'after\n', + ) + await expect(stat(join(outside, 'file-transactions'))).rejects.toMatchObject({ + code: 'ENOENT', + }) + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }, + ) + + it('rolls an applied transaction back to the exact previous files', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), 'one-before\n') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'rollback', + mutations: [ + { path: 'knowledge/one.md', content: 'one-after\n' }, + { path: 'knowledge/two.md', content: 'two-after\n' }, + ], + }) + expect(prepared).not.toBeNull() + await applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }) + + await rollbackKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }) + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe( + 'one-before\n', + ) + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).rejects.toMatchObject({ + code: 'ENOENT', + }) + }) + }) + + it('does not start rollback after the writer loses ownership', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), 'one-before\n') + await writeFile(join(root, 'knowledge', 'two.md'), 'two-before\n') + const transactionRoot = join(root, '.transactions') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'lock-loss', + mutations: [ + { path: 'knowledge/one.md', content: 'one-after\n' }, + { path: 'knowledge/two.md', content: 'two-after\n' }, + ], + }) + expect(prepared).not.toBeNull() + await applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }) + + await expect( + rollbackKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + beforeCommit() { + throw new KnowledgeLockLostError('simulated ownership loss') + }, + }), + ).rejects.toThrow(/ownership loss/) + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe('one-after\n') + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).resolves.toBe('two-after\n') + }) + }) + + it('rejects a changed transaction snapshot', async () => { + await withRoot(async (root) => { + const transactionRoot = join(root, '.transactions') + const prepared = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'snapshot-integrity', + mutations: [{ path: 'knowledge/page.md', content: 'approved\n' }], + }) + expect(prepared).not.toBeNull() + await writeFile( + join(transactionRoot, `active-${prepared!.transactionId}`, 'after', '0.bin'), + 'tampered\n', + ) + + await expect( + applyKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: prepared!, + }), + ).rejects.toThrow(/snapshot changed/) + await expect(readFile(join(root, 'knowledge', 'page.md'), 'utf8')).rejects.toMatchObject({ + code: 'ENOENT', + }) + }) + }) +}) diff --git a/tests/freshness.test.ts b/tests/freshness.test.ts index da032be..e061173 100644 --- a/tests/freshness.test.ts +++ b/tests/freshness.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp, rm } from 'node:fs/promises' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { describe, expect, it } from 'vitest' @@ -105,19 +105,43 @@ describe('createFileSystemFreshnessStore', () => { it('serializes concurrent marks without losing writes', async () => { await withTempRoot(async (root) => { - const store = createFileSystemFreshnessStore({ root }) + const stores = [ + createFileSystemFreshnessStore({ root }), + createFileSystemFreshnessStore({ root }), + createFileSystemFreshnessStore({ root }), + ] const t = new Date('2026-05-14T12:00:00.000Z') - // Two stores opened on the same root cannot serialize across processes, - // but a single store instance must. await Promise.all([ - store.mark({ workspaceId: 'w1', sourceId: 'a', when: t }), - store.mark({ workspaceId: 'w1', sourceId: 'b', when: t }), - store.mark({ workspaceId: 'w1', sourceId: 'c', when: t }), + stores[0]!.mark({ workspaceId: 'w1', sourceId: 'a', when: t }), + stores[1]!.mark({ workspaceId: 'w1', sourceId: 'b', when: t }), + stores[2]!.mark({ workspaceId: 'w1', sourceId: 'c', when: t }), ]) - const list = await store.list('w1') + const list = await stores[0]!.list('w1') expect(list.map((r) => r.sourceId).sort()).toEqual(['a', 'b', 'c']) }) }) + + it('does not replace malformed freshness data', async () => { + await withTempRoot(async (root) => { + const store = createFileSystemFreshnessStore({ root }) + await store.mark({ + workspaceId: 'w1', + sourceId: 'first', + when: new Date('2026-05-14T12:00:00.000Z'), + }) + const path = join(root, '.agent-knowledge', 'freshness.json') + await writeFile(path, '{broken') + + await expect( + store.mark({ + workspaceId: 'w1', + sourceId: 'second', + when: new Date('2026-05-14T12:01:00.000Z'), + }), + ).rejects.toThrow() + await expect(readFile(path, 'utf8')).resolves.toBe('{broken') + }) + }) }) describe('createD1FreshnessStoreStub', () => { diff --git a/tests/kb-improvement.test.ts b/tests/kb-improvement.test.ts index d26f27a..ba80094 100644 --- a/tests/kb-improvement.test.ts +++ b/tests/kb-improvement.test.ts @@ -1,8 +1,21 @@ -import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { createHash } from 'node:crypto' +import { + chmod, + mkdir, + mkdtemp, + readdir, + readFile, + rename, + rm, + stat, + symlink, + writeFile, +} from 'node:fs/promises' import { tmpdir } from 'node:os' -import { dirname, join } from 'node:path' +import { dirname, join, relative } from 'node:path' import { describe, expect, it } from 'vitest' import { + applyKnowledgeWriteBlocks, buildEvalKnowledgeBundle, buildKnowledgeIndex, defineReadinessSpec, @@ -10,10 +23,16 @@ import { hashKnowledgeBase, improveKnowledgeBase, initKnowledgeBase, + knowledgeImprovementCandidateRef, knowledgeImprovementRunDir, + loadKnowledgeImprovementEvents, + loadKnowledgeImprovementState, + promoteKnowledgeCandidate, sha256, stableId, + withKnowledgeImprovementCandidate, } from '../src/index' +import { withKnowledgeMutation } from '../src/mutation-lock' async function withKb(fn: (root: string) => Promise): Promise { const root = await mkdtemp(join(tmpdir(), 'agent-knowledge-improve-')) @@ -25,6 +44,28 @@ async function withKb(fn: (root: string) => Promise): Promise { } } +async function withEmptyRoot(fn: (root: string) => Promise): Promise { + const root = await mkdtemp(join(tmpdir(), 'agent-knowledge-improve-empty-')) + try { + await fn(root) + } finally { + await rm(root, { recursive: true, force: true }) + } +} + +function mutableCandidateRoot( + root: string, + result: { runId: string; candidate?: { candidateId: string } }, +): string { + if (!result.candidate) throw new Error('knowledge improvement result has no candidate') + return join( + knowledgeImprovementRunDir(root, result.runId), + 'candidates', + result.candidate.candidateId, + 'workspace', + ) +} + const refundSpec = defineReadinessSpec({ id: 'refund-policy', description: 'Refund policy support knowledge', @@ -62,7 +103,128 @@ function refundProposal(sourceId: string, extra = ''): string { ].join('\n') } +function passingMetric() { + return { + score: 1, + passed: true, + provenance: { + evaluator: 'agent-knowledge-test', + version: '1', + method: 'deterministic' as const, + }, + } +} + +async function improveAndPromote(options: Parameters[0]) { + const staged = await improveKnowledgeBase(options) + const promoted = await promoteKnowledgeCandidate({ + root: options.root, + candidate: knowledgeImprovementCandidateRef(staged), + }) + return { ...promoted, evaluation: staged.evaluation, lifecycle: staged.lifecycle } +} + describe('improveKnowledgeBase', () => { + it('leaves the live knowledge base unchanged unless promotion is explicit', async () => { + await withEmptyRoot(async (root) => { + const before = await hashKnowledgeBase(root) + await expect(stat(join(root, '.agent-knowledge'))).rejects.toMatchObject({ code: 'ENOENT' }) + const result = await improveKnowledgeBase({ + root, + goal: 'Create a measured candidate without applying it', + runId: 'propose-only-default', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'candidate.md'), '# Candidate\n') + return { applied: true, summary: 'created candidate' } + }, + evaluate: passingMetric, + }) + + expect(result.promoted).toBe(false) + expect(result.state.status).toBe('candidate-ready') + expect(result.candidate?.candidateHash).not.toBe(before) + await expect(hashKnowledgeBase(root)).resolves.toBe(before) + await expect(readFile(join(root, 'knowledge', 'candidate.md'), 'utf8')).rejects.toMatchObject( + { + code: 'ENOENT', + }, + ) + await expect(stat(join(root, 'knowledge'))).rejects.toMatchObject({ code: 'ENOENT' }) + await expect(stat(join(root, 'raw'))).rejects.toMatchObject({ code: 'ENOENT' }) + await expect( + readFile(join(root, '.agent-knowledge', 'sources.json'), 'utf8'), + ).rejects.toMatchObject({ code: 'ENOENT' }) + }) + }) + + it('rejects evaluator results without provenance', async () => { + await withKb(async (root) => { + await expect( + improveKnowledgeBase({ + root, + goal: 'Reject an anonymous evaluation', + runId: 'anonymous-evaluator', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'candidate.md'), '# Candidate\n') + return { applied: true, summary: 'created candidate' } + }, + evaluate: (() => ({ score: 1, passed: true })) as never, + }), + ).rejects.toThrow(/provenance/) + }) + }) + + it('does not create improvement state through a symbolic-link metadata directory', async () => { + await withEmptyRoot(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-improvement-outside-')) + try { + await symlink(outside, join(root, '.agent-knowledge')) + await expect( + improveKnowledgeBase({ + root, + goal: 'Reject redirected improvement state', + runId: 'redirected-state', + }), + ).rejects.toThrow(/unsafe directory/) + await expect(stat(join(outside, 'improvements'))).rejects.toMatchObject({ code: 'ENOENT' }) + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }) + + it.skipIf(process.platform !== 'linux')( + 'rejects a run directory swapped while candidate work is active', + async () => { + await withKb(async (root) => { + const outside = await mkdtemp(join(tmpdir(), 'agent-knowledge-run-outside-')) + const runId = 'run-directory-swap' + const runDir = knowledgeImprovementRunDir(root, runId) + try { + await expect( + improveKnowledgeBase({ + root, + goal: 'Reject redirected run state', + runId, + async updateKnowledge({ candidateRoot }) { + await writeFile(join(candidateRoot, 'knowledge', 'candidate.md'), '# Candidate\n') + await rename(runDir, `${runDir}-original`) + await symlink(outside, runDir) + return { applied: true, summary: 'candidate written before redirect' } + }, + evaluate: passingMetric, + }), + ).rejects.toThrow(/unsafe directory|changed during use/) + await expect(readFile(join(outside, 'state.json'), 'utf8')).rejects.toMatchObject({ + code: 'ENOENT', + }) + } finally { + await rm(outside, { recursive: true, force: true }) + } + }) + }, + ) + it('evaluates root-level KB readiness without running an improvement loop', async () => { await withKb(async (root) => { const empty = await evaluateKnowledgeBaseReadiness({ @@ -77,7 +239,7 @@ describe('improveKnowledgeBase', () => { expect(empty.summary).toContain('blocking readiness requirement') const source = refundSource() - const improved = await improveKnowledgeBase({ + const improved = await improveAndPromote({ root, goal: 'Build billing support refund-policy knowledge', runId: 'readiness-evaluator', @@ -119,7 +281,7 @@ describe('improveKnowledgeBase', () => { expect(emptyReadiness.report.blockingMissingRequirements).toHaveLength(1) const source = refundSource() - const result = await improveKnowledgeBase({ + const result = await improveAndPromote({ root, goal: 'Build billing support refund-policy knowledge', runId: 'refund-improvement', @@ -134,12 +296,10 @@ describe('improveKnowledgeBase', () => { expect(result.promoted).toBe(true) expect(result.state.status).toBe('promoted') - expect(result.candidate?.evaluation?.score).toBe(1) + expect(result.evaluation?.score).toBe(1) expect( - result.candidate?.lifecycle?.some((record) => - record.phases.some( - (phase) => phase.phase === 'knowledge-update' && phase.status === 'completed', - ), + result.lifecycle?.phases.some( + (phase) => phase.phase === 'knowledge-update' && phase.status === 'completed', ), ).toBe(true) @@ -163,7 +323,6 @@ describe('improveKnowledgeBase', () => { runId: 'resume-edit', readinessSpecs: [refundSpec], strict: true, - promote: false, step: () => ({ done: true, sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], @@ -173,8 +332,7 @@ describe('improveKnowledgeBase', () => { expect(first.promoted).toBe(false) expect(first.state.status).toBe('candidate-ready') - const candidateRoot = first.candidate?.candidateRoot - expect(candidateRoot).toBeDefined() + const candidateRoot = mutableCandidateRoot(root, first) const editedPage = join(candidateRoot!, 'knowledge', 'support', 'refund-escalation.md') await mkdir(dirname(editedPage), { recursive: true }) @@ -192,7 +350,7 @@ describe('improveKnowledgeBase', () => { ].join('\n'), ) - const resumed = await improveKnowledgeBase({ + const resumed = await improveAndPromote({ root, goal: 'Build billing support refund-policy knowledge', runId: 'resume-edit', @@ -209,6 +367,505 @@ describe('improveKnowledgeBase', () => { }) }) + it('promotes one exact approved candidate without rerunning candidate work', async () => { + await withKb(async (root) => { + const source = refundSource() + let updateCalls = 0 + const staged = await improveKnowledgeBase({ + root, + goal: 'Build billing support refund-policy knowledge', + runId: 'approved-candidate', + readinessSpecs: [refundSpec], + strict: true, + step: () => { + updateCalls += 1 + return { + done: true, + sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], + proposalText: refundProposal(source.id), + } + }, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + + const promoted = await promoteKnowledgeCandidate({ root, candidate }) + expect(promoted.promoted).toBe(true) + expect(promoted.state.promotedCandidateId).toBe(candidate.candidateId) + expect(updateCalls).toBe(1) + await expect( + readFile(join(root, 'knowledge', 'support', 'refund-policy.md'), 'utf8'), + ).resolves.toContain('refund within 30 days') + + const repeated = await promoteKnowledgeCandidate({ root, candidate }) + expect(repeated.promoted).toBe(true) + expect(repeated.state.promotedCandidateId).toBe(candidate.candidateId) + expect(updateCalls).toBe(1) + }) + }) + + it('finishes an interrupted approved promotion without rerunning candidate work', async () => { + await withKb(async (root) => { + const source = refundSource() + let updateCalls = 0 + const staged = await improveKnowledgeBase({ + root, + goal: 'Build billing support refund-policy knowledge', + runId: 'interrupted-promotion', + readinessSpecs: [refundSpec], + strict: true, + step: () => { + updateCalls += 1 + return { + done: true, + sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], + proposalText: refundProposal(source.id), + } + }, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + + await expect( + promoteKnowledgeCandidate({ + root, + candidate, + onState() { + throw new Error('operator stopped after state persistence') + }, + }), + ).rejects.toThrow(/operator stopped/) + + const runDir = knowledgeImprovementRunDir(root, candidate.runId) + await rm(join(runDir, 'candidates', candidate.candidateId, 'snapshots'), { + recursive: true, + force: true, + }) + await rm(join(runDir, 'candidates', candidate.candidateId, 'evidence.json'), { + force: true, + }) + await expect( + applyKnowledgeWriteBlocks( + root, + ['---FILE: knowledge/unapproved.md---', '# Unapproved', '---END FILE---'].join('\n'), + ), + ).rejects.toThrow(/requires its owner to resume/) + const recovered = await promoteKnowledgeCandidate({ root, candidate }) + + expect(recovered.promoted).toBe(true) + expect(updateCalls).toBe(1) + const events = await loadKnowledgeImprovementEvents(root, candidate.runId) + expect(events.filter((event) => event.type === 'candidate.promoted')).toHaveLength(1) + const transactionEntries = await readdir(join(root, '.agent-knowledge', 'file-transactions')) + expect(transactionEntries.filter((entry) => entry.startsWith('active-'))).toEqual([]) + }) + }) + + it('rejects a forged promotion journal entry outside the measured knowledge files', async () => { + await withKb(async (root) => { + const packagePath = join(root, 'package.json') + const originalPackage = '{"private":true}\n' + const forgedPackage = '{"scripts":{"postinstall":"malicious"}}\n' + await writeFile(packagePath, originalPackage) + const staged = await improveKnowledgeBase({ + root, + goal: 'Prepare an exact knowledge-only candidate', + runId: 'forged-promotion-journal', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'candidate.md'), '# Candidate\n') + return { applied: true, summary: 'created candidate' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + await expect( + promoteKnowledgeCandidate({ + root, + candidate, + onState() { + throw new Error('leave promotion pending') + }, + }), + ).rejects.toThrow(/leave promotion pending/) + + const transactionRoot = join(root, '.agent-knowledge', 'file-transactions') + const activeTransactions = (await readdir(transactionRoot)).filter((entry) => + entry.startsWith('active-'), + ) + expect(activeTransactions).toHaveLength(1) + const transactionDir = join(transactionRoot, activeTransactions[0]!) + const transactionPath = join(transactionDir, 'transaction.json') + const transaction = JSON.parse(await readFile(transactionPath, 'utf8')) as { + entries: Array<{ + index: number + path: string + beforeHash: string | null + afterHash: string | null + beforeMode?: number + afterMode?: number + }> + } + const index = Math.max(...transaction.entries.map((entry) => entry.index)) + 1 + transaction.entries.push({ + index, + path: 'package.json', + beforeHash: createHash('sha256').update(originalPackage).digest('hex'), + afterHash: createHash('sha256').update(forgedPackage).digest('hex'), + beforeMode: 0o644, + afterMode: 0o644, + }) + await mkdir(join(transactionDir, 'after'), { recursive: true }) + await writeFile(join(transactionDir, 'after', `${index}.bin`), forgedPackage) + await writeFile(transactionPath, `${JSON.stringify(transaction, null, 2)}\n`) + + await expect(promoteKnowledgeCandidate({ root, candidate })).rejects.toThrow( + /unsupported path: package.json/, + ) + await expect(readFile(packagePath, 'utf8')).resolves.toBe(originalPackage) + }) + }) + + it('promotes frozen measured bytes when the mutable workspace changes after approval', async () => { + await withKb(async (root) => { + const source = refundSource() + const staged = await improveKnowledgeBase({ + root, + goal: 'Build billing support refund-policy knowledge', + runId: 'changed-after-approval', + readinessSpecs: [refundSpec], + strict: true, + step: () => ({ + done: true, + sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], + proposalText: refundProposal(source.id), + }), + }) + const candidate = knowledgeImprovementCandidateRef(staged) + await writeFile( + join(mutableCandidateRoot(root, staged), 'knowledge', 'support', 'refund-policy.md'), + '# changed after approval\n', + ) + + await expect(promoteKnowledgeCandidate({ root, candidate })).resolves.toMatchObject({ + promoted: true, + }) + await expect( + readFile(join(root, 'knowledge', 'support', 'refund-policy.md'), 'utf8'), + ).resolves.toContain('refund within 30 days') + await expect( + readFile(join(root, 'knowledge', 'support', 'refund-policy.md'), 'utf8'), + ).resolves.not.toContain('changed after approval') + }) + }) + + it('resolves and promotes the measured snapshot after mutable workspace removal', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Keep the measured candidate independent from scratch files', + runId: 'removed-mutable-workspace', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'measured.md'), '# Measured\n') + return { applied: true, summary: 'created measured knowledge' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + await rm(mutableCandidateRoot(root, staged), { recursive: true, force: true }) + const runDir = knowledgeImprovementRunDir(root, candidate.runId) + let isolatedRoot = '' + + await withKnowledgeImprovementCandidate({ root, candidate }, async (resolved) => { + isolatedRoot = resolved.root + expect(relative(runDir, resolved.root)).toMatch(/^\.\./) + await expect( + readFile(join(resolved.root, 'knowledge', 'measured.md'), 'utf8'), + ).resolves.toBe('# Measured\n') + }) + await expect(stat(isolatedRoot)).rejects.toMatchObject({ code: 'ENOENT' }) + await expect(promoteKnowledgeCandidate({ root, candidate })).resolves.toMatchObject({ + promoted: true, + }) + }) + }) + + it('rejects a measured snapshot changed while an approved candidate is in use', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Keep approved candidate execution immutable', + runId: 'changed-during-candidate-use', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'measured.md'), '# Measured\n') + return { applied: true, summary: 'created measured knowledge' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + + await expect( + withKnowledgeImprovementCandidate({ root, candidate }, async (resolved) => { + await writeFile(join(resolved.root, 'knowledge', 'measured.md'), '# Changed\n') + }), + ).rejects.toThrow(/snapshot changed during use/) + }) + }) + + it('rejects a frozen measured copy changed after approval', async () => { + await withKb(async (root) => { + const source = refundSource() + const staged = await improveKnowledgeBase({ + root, + goal: 'Build billing support refund-policy knowledge', + runId: 'changed-frozen-candidate', + readinessSpecs: [refundSpec], + strict: true, + step: () => ({ + done: true, + sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], + proposalText: refundProposal(source.id), + }), + }) + const candidate = knowledgeImprovementCandidateRef(staged) + const snapshotRoot = join( + knowledgeImprovementRunDir(root, candidate.runId), + 'candidates', + candidate.candidateId, + 'snapshots', + candidate.candidateHash, + ) + await writeFile( + join(snapshotRoot, 'knowledge', 'support', 'refund-policy.md'), + '# tampered measured copy\n', + ) + + await expect(promoteKnowledgeCandidate({ root, candidate })).rejects.toThrow( + /snapshot changed after approval/, + ) + await expect( + readFile(join(root, 'knowledge', 'support', 'refund-policy.md'), 'utf8'), + ).rejects.toMatchObject({ code: 'ENOENT' }) + }) + }) + + it('promotes candidate deletions exactly', async () => { + await withKb(async (root) => { + const obsoletePage = join(root, 'knowledge', 'obsolete.md') + const obsoleteRaw = join(root, 'raw', 'sources', 'obsolete.txt') + await writeFile(obsoletePage, '# Obsolete\n') + await writeFile(obsoleteRaw, 'obsolete source\n') + const staged = await improveKnowledgeBase({ + root, + goal: 'Remove obsolete knowledge', + runId: 'approved-deletions', + updateKnowledge: async ({ candidateRoot }) => { + await rm(join(candidateRoot, 'knowledge', 'obsolete.md')) + await rm(join(candidateRoot, 'raw', 'sources', 'obsolete.txt')) + return { applied: true, summary: 'removed obsolete knowledge' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + + await promoteKnowledgeCandidate({ root, candidate }) + + await expect(readFile(obsoletePage, 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + await expect(readFile(obsoleteRaw, 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + await expect(hashKnowledgeBase(root)).resolves.toBe(candidate.candidateHash) + }) + }) + + it.skipIf(process.platform !== 'linux')('promotes a mode-only candidate exactly', async () => { + await withKb(async (root) => { + const page = join(root, 'knowledge', 'mode-only.md') + await writeFile(page, '# Same bytes\n') + await chmod(page, 0o600) + const staged = await improveKnowledgeBase({ + root, + goal: 'Make the measured page executable', + runId: 'mode-only-promotion', + updateKnowledge: async ({ candidateRoot }) => { + await chmod(join(candidateRoot, 'knowledge', 'mode-only.md'), 0o700) + return { applied: true, summary: 'changed the measured file mode' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + + expect(candidate.candidateHash).not.toBe(candidate.baseHash) + await expect(promoteKnowledgeCandidate({ root, candidate })).resolves.toMatchObject({ + promoted: true, + }) + expect(await readFile(page, 'utf8')).toBe('# Same bytes\n') + expect((await stat(page)).mode & 0o777).toBe(0o700) + }) + }) + + it('promotes only measured files when the mutable workspace gains an unmeasured entry', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Create one measured page', + runId: 'unsupported-entry', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'measured.md'), '# Measured\n') + return { applied: true, summary: 'created measured page' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + await symlink( + join(mutableCandidateRoot(root, staged), 'knowledge', 'measured.md'), + join(mutableCandidateRoot(root, staged), 'knowledge', 'unmeasured.md'), + ) + + await expect(promoteKnowledgeCandidate({ root, candidate })).resolves.toMatchObject({ + promoted: true, + }) + await expect(readFile(join(root, 'knowledge', 'measured.md'), 'utf8')).resolves.toBe( + '# Measured\n', + ) + await expect( + readFile(join(root, 'knowledge', 'unmeasured.md'), 'utf8'), + ).rejects.toMatchObject({ + code: 'ENOENT', + }) + }) + }) + + it('rejects an approval whose measured evidence identity was changed', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Create one measured page', + runId: 'changed-evidence', + updateKnowledge: async (input) => { + const page = join(input.candidateRoot, 'knowledge', 'measured.md') + await mkdir(dirname(page), { recursive: true }) + await writeFile(page, '# Measured\n') + return { applied: true, summary: 'created measured page' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + const changed = { ...candidate, evidenceHash: '0'.repeat(64) } + + await expect(promoteKnowledgeCandidate({ root, candidate: changed })).rejects.toThrow( + /does not match the measured candidate/, + ) + }) + }) + + it('binds approval to the full measured lifecycle evidence', async () => { + await withKb(async (root) => { + const source = refundSource() + const staged = await improveKnowledgeBase({ + root, + goal: 'Build billing support refund-policy knowledge', + runId: 'full-evidence', + readinessSpecs: [refundSpec], + strict: true, + step: () => ({ + done: true, + sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], + proposalText: refundProposal(source.id), + }), + }) + const candidate = knowledgeImprovementCandidateRef(staged) + const evidencePath = join( + knowledgeImprovementRunDir(root, candidate.runId), + 'candidates', + candidate.candidateId, + 'evidence.json', + ) + const evidence = JSON.parse(await readFile(evidencePath, 'utf8')) as { + lifecycle: { knowledgeUpdate: { summary: string } } + } + evidence.lifecycle.knowledgeUpdate.summary = 'different measured evidence' + await writeFile(evidencePath, `${JSON.stringify(evidence, null, 2)}\n`) + + await expect(promoteKnowledgeCandidate({ root, candidate })).rejects.toThrow( + /evidence changed after approval/, + ) + }) + }) + + it('fails loudly when persisted improvement state is malformed', async () => { + await withKb(async (root) => { + const runDir = knowledgeImprovementRunDir(root, 'malformed-state') + await mkdir(runDir, { recursive: true }) + await writeFile(join(runDir, 'state.json'), '{broken') + + await expect(loadKnowledgeImprovementState(root, 'malformed-state')).rejects.toThrow() + await writeFile(join(runDir, 'state.json'), '{}') + await expect(loadKnowledgeImprovementState(root, 'malformed-state')).rejects.toThrow() + }) + }) + + it('rejects evaluation copied into mutable run state', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Keep evaluation in immutable evidence only', + runId: 'mutable-evaluation-copy', + evaluate: passingMetric, + }) + const runDir = knowledgeImprovementRunDir(root, staged.runId) + const statePath = join(runDir, 'state.json') + const state = JSON.parse(await readFile(statePath, 'utf8')) as { + candidates: Array> + } + state.candidates[0]!.evaluation = passingMetric() + await writeFile(statePath, `${JSON.stringify(state, null, 2)}\n`) + + await expect(loadKnowledgeImprovementState(root, staged.runId)).rejects.toThrow() + await expect( + withKnowledgeImprovementCandidate( + { root, candidate: knowledgeImprovementCandidateRef(staged) }, + () => undefined, + ), + ).rejects.toThrow() + }) + }) + + it('rejects missing promotion evidence in a native current state', async () => { + await withKb(async (root) => { + const options = { + root, + goal: 'Reject corrupted current promotion state', + runId: 'native-promoted-state', + updateKnowledge: async ({ candidateRoot }: { candidateRoot: string }) => { + await writeFile(join(candidateRoot, 'knowledge', 'current.md'), '# Current\n') + return { applied: true, summary: 'created current page' } + }, + evaluate: passingMetric, + } + const promoted = await improveAndPromote(options) + const runDir = knowledgeImprovementRunDir(root, promoted.runId) + const statePath = join(runDir, 'state.json') + const current = JSON.parse(await readFile(statePath, 'utf8')) as { + candidates: Array> + } + delete current.candidates[0]?.evidenceHash + delete current.candidates[0]?.promotionPlanHash + await writeFile(statePath, `${JSON.stringify(current, null, 2)}\n`) + + await expect(improveKnowledgeBase(options)).rejects.toThrow( + /promoted candidates require content, evidence, and promotion-plan identities/, + ) + }) + }) + + it('encodes external run ids without allowing them to escape the improvement directory', async () => { + await withKb(async (root) => { + const runId = '../../victim' + const improvementsDir = join(root, '.agent-knowledge', 'improvements') + const runDir = knowledgeImprovementRunDir(root, runId) + expect(relative(improvementsDir, runDir)).not.toMatch(/^\.\.(?:\/|$)/) + await expect( + improveKnowledgeBase({ root, goal: 'External run identity', runId }), + ).resolves.toMatchObject({ runId }) + }) + }) + it('retries a running candidate after a crashed operator run', async () => { await withKb(async (root) => { const source = refundSource() @@ -225,7 +882,7 @@ describe('improveKnowledgeBase', () => { }), ).rejects.toThrow(/research worker crashed/) - const resumed = await improveKnowledgeBase({ + const resumed = await improveAndPromote({ root, goal: 'Build billing support refund-policy knowledge', runId: 'crash-resume', @@ -258,7 +915,6 @@ describe('improveKnowledgeBase', () => { root, goal: 'Let a runtime supervisor update the candidate KB', runId: 'candidate-update-root', - promote: false, async updateKnowledge(input) { seen.push({ root: input.root, @@ -282,15 +938,14 @@ describe('improveKnowledgeBase', () => { ) return { applied: true, summary: 'candidate workspace updated' } }, - evaluate: () => ({ score: 1, passed: true }), + evaluate: passingMetric, }) expect(result.state.status).toBe('candidate-ready') expect(seen).toHaveLength(1) + expect(seen[0]!.root).toBe(seen[0]!.candidateRoot) expect(seen[0]).toMatchObject({ - root: result.candidate?.candidateRoot, baselineRoot: root, - candidateRoot: result.candidate?.candidateRoot, runId: 'candidate-update-root', iteration: 1, }) @@ -299,7 +954,7 @@ describe('improveKnowledgeBase', () => { ).rejects.toThrow() await expect( readFile( - join(result.candidate!.candidateRoot, 'knowledge', 'runtime-supervisor.md'), + join(mutableCandidateRoot(root, result), 'knowledge', 'runtime-supervisor.md'), 'utf8', ), ).resolves.toContain('candidate workspace only') @@ -315,7 +970,6 @@ describe('improveKnowledgeBase', () => { runId: 'conflict', readinessSpecs: [refundSpec], strict: true, - promote: false, step: () => ({ done: true, sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], @@ -338,12 +992,9 @@ describe('improveKnowledgeBase', () => { ].join('\n'), ) - const blocked = await improveKnowledgeBase({ + const blocked = await promoteKnowledgeCandidate({ root, - goal: 'Build billing support refund-policy knowledge', - runId: 'conflict', - readinessSpecs: [refundSpec], - strict: true, + candidate: knowledgeImprovementCandidateRef(first), }) expect(blocked.blocked).toBe(true) @@ -355,17 +1006,26 @@ describe('improveKnowledgeBase', () => { it('fails fast when another active operator owns the run lease', async () => { await withKb(async (root) => { - const runDir = knowledgeImprovementRunDir(root, 'locked') - await mkdir(runDir, { recursive: true }) - await writeFile( - join(runDir, 'run.lock'), - `${JSON.stringify({ - ownerId: 'agent-a', - acquiredAt: '2026-01-01T00:00:00.000Z', - expiresAt: '2999-01-01T00:00:00.000Z', - pid: 123, - })}\n`, - ) + let releaseStep!: () => void + let markStarted!: () => void + const started = new Promise((resolve) => { + markStarted = resolve + }) + const released = new Promise((resolve) => { + releaseStep = resolve + }) + const first = improveKnowledgeBase({ + root, + goal: 'Locked run', + runId: 'locked', + updateKnowledge: async () => { + markStarted() + await released + return { applied: true, summary: 'finished' } + }, + evaluate: passingMetric, + }) + await started await expect( improveKnowledgeBase({ @@ -374,7 +1034,61 @@ describe('improveKnowledgeBase', () => { runId: 'locked', readinessSpecs: [refundSpec], }), - ).rejects.toThrow(/locked by agent-a/) + ).rejects.toThrow(/Lock file is already being held/) + releaseStep() + await first + }) + }) + + it('serializes package writers against approved promotion', async () => { + await withKb(async (root) => { + const staged = await improveKnowledgeBase({ + root, + goal: 'Create measured knowledge', + runId: 'writer-race', + updateKnowledge: async ({ candidateRoot }) => { + await writeFile(join(candidateRoot, 'knowledge', 'measured.md'), '# Measured\n') + return { applied: true, summary: 'created measured knowledge' } + }, + evaluate: passingMetric, + }) + const candidate = knowledgeImprovementCandidateRef(staged) + let releaseWriter!: () => void + let markStarted!: () => void + const started = new Promise((resolve) => { + markStarted = resolve + }) + const released = new Promise((resolve) => { + releaseWriter = resolve + }) + const writer = withKnowledgeMutation(root, async () => { + markStarted() + await released + await applyKnowledgeWriteBlocks( + root, + ['---FILE: knowledge/concurrent.md---', '# Concurrent', '---END FILE---'].join('\n'), + ) + }) + await started + + const promotion = promoteKnowledgeCandidate({ root, candidate }) + const beforeRelease = await Promise.race([ + promotion.then( + () => 'settled', + () => 'settled', + ), + new Promise<'waiting'>((resolve) => setTimeout(() => resolve('waiting'), 50)), + ]) + expect(beforeRelease).toBe('waiting') + releaseWriter() + await writer + await expect(promotion).resolves.toMatchObject({ promoted: false, blocked: true }) + await expect(readFile(join(root, 'knowledge', 'concurrent.md'), 'utf8')).resolves.toContain( + 'Concurrent', + ) + await expect(readFile(join(root, 'knowledge', 'measured.md'), 'utf8')).rejects.toMatchObject({ + code: 'ENOENT', + }) }) }) @@ -387,7 +1101,6 @@ describe('improveKnowledgeBase', () => { runId: 'retrieval', readinessSpecs: [refundSpec], strict: true, - promote: false, step: () => ({ done: true, sourceTexts: [{ uri: source.uri, text: source.text, title: source.title }], @@ -428,7 +1141,7 @@ describe('improveKnowledgeBase', () => { }, }) - expect(result.candidate?.retrievalWinnerConfig).toMatchObject({ k: 2 }) + expect(result.lifecycle?.retrieval?.winnerConfig).toMatchObject({ k: 2 }) expect(result.lifecycle?.phases.map((phase) => `${phase.phase}:${phase.status}`)).toContain( 'retrieval-tuning:completed', ) @@ -465,8 +1178,8 @@ describe('improveKnowledgeBase', () => { expect(result.promoted).toBe(false) expect(result.state.status).toBe('rejected') - expect(result.candidate?.evaluation?.passed).toBe(false) - expect(result.candidate?.evaluation?.notes).toContain('answer quality failed') + expect(result.evaluation?.passed).toBe(false) + expect(result.evaluation?.notes).toContain('answer quality failed') }) }) diff --git a/tests/mutation-lock.test.ts b/tests/mutation-lock.test.ts new file mode 100644 index 0000000..277846b --- /dev/null +++ b/tests/mutation-lock.test.ts @@ -0,0 +1,296 @@ +import { chmod, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as delay } from 'node:timers/promises' +import { describe, expect, it } from 'vitest' +import { + applyKnowledgeFileTransaction, + prepareKnowledgeFileTransaction, + rollbackKnowledgeFileTransaction, +} from '../src/file-transaction' +import { inspectPendingKnowledgeMutation, recoverPendingKnowledgeMutation } from '../src/index' +import { buildKnowledgeIndex } from '../src/indexer' +import { withKnowledgeMutation } from '../src/mutation-lock' +import { loadSourceRegistry } from '../src/sources' +import { initKnowledgeBase } from '../src/store' + +async function withRoot(fn: (root: string) => Promise): Promise { + const root = await mkdtemp(join(tmpdir(), 'agent-knowledge-read-')) + try { + await fn(root) + } finally { + await chmodTree(root, 0o700).catch(() => undefined) + await rm(root, { recursive: true, force: true }) + } +} + +describe('knowledge read epochs', () => { + it('does not change an empty root during pure reads', async () => { + await withRoot(async (root) => { + expect(await readdir(root)).toEqual([]) + + await expect(loadSourceRegistry(root)).resolves.toMatchObject({ sources: [] }) + await expect(buildKnowledgeIndex(root)).resolves.toMatchObject({ pages: [], sources: [] }) + + expect(await readdir(root)).toEqual([]) + }) + }) + + it.skipIf(process.platform === 'win32')( + 'reads an existing read-only knowledge base', + async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await mkdir(join(root, '.agent-knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'page.md'), '# Readable\n') + await writeFile( + join(root, '.agent-knowledge', 'sources.json'), + '{\n "generatedAt": "1970-01-01T00:00:00.000Z",\n "sources": []\n}\n', + ) + await chmodTree(root, 0o555, 0o444) + + const index = await buildKnowledgeIndex(root) + + expect(index.pages.map((page) => page.title)).toEqual(['Readable']) + await expect(readdir(join(root, '.agent-knowledge'))).resolves.toEqual(['sources.json']) + }) + }, + ) + + it('does not return a mixed view during an active multi-file mutation', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), '# One before\n') + await writeFile(join(root, 'knowledge', 'two.md'), '# Two before\n') + + let startReader!: () => void + const readerStart = new Promise((resolve) => { + startReader = resolve + }) + let readerSettled = false + const reader = readerStart + .then(() => buildKnowledgeIndex(root)) + .finally(() => { + readerSettled = true + }) + + await withKnowledgeMutation(root, async () => { + await writeFile(join(root, 'knowledge', 'one.md'), '# One after\n') + startReader() + await delay(50) + expect(readerSettled).toBe(false) + await writeFile(join(root, 'knowledge', 'two.md'), '# Two after\n') + }) + + await expect(reader).resolves.toMatchObject({ + pages: [{ title: 'One after' }, { title: 'Two after' }], + }) + }) + }) + + it('fails loudly on an abandoned odd mutation epoch', async () => { + await withRoot(async (root) => { + await mkdir(join(root, '.agent-knowledge'), { recursive: true }) + await writeFile( + join(root, '.agent-knowledge', 'mutation-epoch.json'), + '{\n "epoch": 1,\n "updatedAt": "2026-07-13T00:00:00.000Z"\n}\n', + ) + + await expect(buildKnowledgeIndex(root)).rejects.toThrow(/odd with no active writer/) + }) + }) + + it('leaves interrupted transactions unreadable until recovered', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), '# One before\n') + await writeFile(join(root, 'knowledge', 'two.md'), '# Two before\n') + + await expect( + withKnowledgeMutation(root, async (lock) => { + const transaction = await prepareKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + purpose: 'interrupted-read', + mutations: [ + { path: 'knowledge/one.md', content: '# One after\n' }, + { path: 'knowledge/two.md', content: '# Two after\n' }, + ], + }) + await applyKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + transaction: transaction!, + beforeCommit(entry) { + if (entry.path === 'knowledge/two.md') throw new Error('simulated interruption') + }, + }) + }), + ).rejects.toThrow(/simulated interruption/) + + await expect(buildKnowledgeIndex(root)).rejects.toThrow(/odd with no active writer/) + }) + }) + + it.each([ + { + action: 'apply' as const, + expectedOne: '# One after\n', + expectedTwo: '# Two after\n', + }, + { + action: 'rollback' as const, + expectedOne: '# One before\n', + expectedTwo: '# Two before\n', + }, + ])('supports public $action recovery for an interrupted mutation', async (scenario) => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'one.md'), '# One before\n') + await writeFile(join(root, 'knowledge', 'two.md'), '# Two before\n') + + await expect( + withKnowledgeMutation(root, async (lock) => { + const transaction = await prepareKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + purpose: 'public-recovery', + mutations: [ + { path: 'knowledge/one.md', content: '# One after\n' }, + { path: 'knowledge/two.md', content: '# Two after\n' }, + ], + }) + await applyKnowledgeFileTransaction({ + root, + transactionRoot: lock.transactionRoot, + transaction: transaction!, + beforeCommit(entry) { + if (entry.path === 'knowledge/two.md') throw new Error('simulated interruption') + }, + }) + }), + ).rejects.toThrow(/simulated interruption/) + + const pending = await inspectPendingKnowledgeMutation(root) + expect(pending).toMatchObject({ + purpose: 'public-recovery', + direction: 'apply', + paths: ['knowledge/one.md', 'knowledge/two.md'], + }) + await expect( + recoverPendingKnowledgeMutation(root, { + transactionId: '00000000-0000-4000-8000-000000000000', + action: scenario.action, + }), + ).rejects.toThrow(/does not match/) + await expect(inspectPendingKnowledgeMutation(root)).resolves.toEqual(pending) + + await recoverPendingKnowledgeMutation(root, { + transactionId: pending!.transactionId, + action: scenario.action, + }) + + await expect(readFile(join(root, 'knowledge', 'one.md'), 'utf8')).resolves.toBe( + scenario.expectedOne, + ) + await expect(readFile(join(root, 'knowledge', 'two.md'), 'utf8')).resolves.toBe( + scenario.expectedTwo, + ) + await expect(inspectPendingKnowledgeMutation(root)).resolves.toBeNull() + await expect(buildKnowledgeIndex(root)).resolves.toMatchObject({ + pages: [ + { title: scenario.expectedOne.trim().slice(2) }, + { title: scenario.expectedTwo.trim().slice(2) }, + ], + }) + }) + }) + + it('rejects apply after rollback has durably started', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'page.md'), '# Before\n') + const transactionRoot = join(root, '.agent-knowledge', 'file-transactions') + const transaction = await prepareKnowledgeFileTransaction({ + root, + transactionRoot, + purpose: 'interrupted-rollback', + mutations: [{ path: 'knowledge/page.md', content: '# After\n' }], + }) + await applyKnowledgeFileTransaction({ root, transactionRoot, transaction: transaction! }) + + let rollbackCheck = 0 + await expect( + rollbackKnowledgeFileTransaction({ + root, + transactionRoot, + transaction: transaction!, + beforeCommit() { + rollbackCheck += 1 + if (rollbackCheck === 2) throw new Error('simulated rollback interruption') + }, + }), + ).rejects.toThrow(/simulated rollback interruption/) + await expect(inspectPendingKnowledgeMutation(root)).resolves.toMatchObject({ + transactionId: transaction!.transactionId, + direction: 'rollback', + }) + + await expect( + recoverPendingKnowledgeMutation(root, { + transactionId: transaction!.transactionId, + action: 'apply', + }), + ).rejects.toThrow(/already rolling back/) + await expect(readFile(join(root, 'knowledge', 'page.md'), 'utf8')).resolves.toBe('# After\n') + + await recoverPendingKnowledgeMutation(root, { + transactionId: transaction!.transactionId, + action: 'rollback', + }) + await expect(readFile(join(root, 'knowledge', 'page.md'), 'utf8')).resolves.toBe('# Before\n') + await expect(inspectPendingKnowledgeMutation(root)).resolves.toBeNull() + }) + }) + + it('does not block source import recovery while re-initializing an existing KB', async () => { + await withRoot(async (root) => { + await initKnowledgeBase(root) + await prepareKnowledgeFileTransaction({ + root, + transactionRoot: join(root, '.agent-knowledge', 'file-transactions'), + purpose: 'pending-source-import', + mutations: [{ path: 'raw/sources/pending.txt', content: 'pending\n' }], + }) + + await expect(initKnowledgeBase(root)).resolves.toMatchObject({ root }) + }) + }) + + it('allows nested pure reads inside a writer', async () => { + await withRoot(async (root) => { + await mkdir(join(root, 'knowledge'), { recursive: true }) + await writeFile(join(root, 'knowledge', 'page.md'), '# Nested\n') + + await expect( + withKnowledgeMutation(root, () => buildKnowledgeIndex(root)), + ).resolves.toMatchObject({ + pages: [{ title: 'Nested' }], + }) + }) + }) +}) + +async function chmodTree( + root: string, + directoryMode: number, + fileMode = directoryMode, +): Promise { + const entries = await readdir(root, { withFileTypes: true }) + for (const entry of entries) { + const path = join(root, entry.name) + if (entry.isDirectory()) await chmodTree(path, directoryMode, fileMode) + await chmod(path, entry.isDirectory() ? directoryMode : fileMode) + } + await chmod(root, directoryMode) +} diff --git a/tests/retrieval-eval.test.ts b/tests/retrieval-eval.test.ts index cb0adc2..7c2a3a1 100644 --- a/tests/retrieval-eval.test.ts +++ b/tests/retrieval-eval.test.ts @@ -1,4 +1,4 @@ -import { inMemoryCampaignStorage } from '@tangle-network/agent-eval/campaign' +import { inMemoryCampaignStorage, runCampaign } from '@tangle-network/agent-eval/campaign' import { describe, expect, it } from 'vitest' import { buildRetrievalEvalDispatch, @@ -25,15 +25,6 @@ function testContext() { } as Parameters>[2] } -function testContextWithCost(observed: Array<{ amountUsd: number; source: string }>) { - return { - ...testContext(), - cost: { - observe: (amountUsd: number, source: string) => observed.push({ amountUsd, source }), - }, - } as Parameters>[2] -} - function fixtureIndex(): KnowledgeIndex { return { root: 'memory://retrieval-eval', @@ -150,8 +141,7 @@ describe('retrieval eval', () => { expect(scoreRetrievalArtifact(spanHit, scenario).recall).toBe(1) }) - it('records retriever-reported cost in the agent-eval cost meter', async () => { - const observed: Array<{ amountUsd: number; source: string }> = [] + it('accounts billable retrieval through the agent-eval paid-call path', async () => { const scenario: RetrievalEvalScenario = { id: 'q-cost', kind: 'retrieval-eval', @@ -159,19 +149,38 @@ describe('retrieval eval', () => { expected: { kind: 'page', pageId: 'page-1' }, } const dispatch = buildRetrievalEvalDispatch({ - retrieve: async () => ({ - costUsd: 0.25, - hits: [{ pageId: 'page-1', path: 'knowledge/page-1.md', rank: 1 }], - }), + retrieve: async ({ context }) => { + const paid = await context.cost.runPaidCall({ + actor: 'test-retriever', + model: 'retriever-fixture', + maximumCharge: { externallyEnforcedMaximumUsd: 0.25 }, + execute: async () => ({ + costUsd: 0.25, + hits: [{ pageId: 'page-1', path: 'knowledge/page-1.md', rank: 1 }], + }), + receipt: () => ({ + model: 'retriever-fixture', + inputTokens: 0, + outputTokens: 0, + usageUnknown: true, + actualCostUsd: 0.25, + }), + }) + if (!paid.succeeded) throw paid.error + return paid.value + }, + }) + const campaign = await runCampaign({ + scenarios: [scenario], + dispatch: (input, context) => dispatch(retrievalConfigSurface({ k: 1 }), input, context), + runDir: '/runs/retrieval-cost', + storage: inMemoryCampaignStorage(), + expectUsage: 'assert', }) - const artifact = await dispatch( - retrievalConfigSurface({ k: 1 }), - scenario, - testContextWithCost(observed), - ) - expect(artifact.costUsd).toBe(0.25) - expect(observed).toEqual([{ amountUsd: 0.25, source: 'agent-knowledge:retrieval' }]) + expect(campaign.cells[0]?.artifact).toMatchObject({ costUsd: 0.25 }) + expect(campaign.aggregates.totalCostUsd).toBe(0.25) + expect(campaign.aggregates.cost.totalCalls).toBe(1) }) it('builds parameter candidates and delegates proposal to agent-eval', async () => { diff --git a/tests/sources-ingestion.test.ts b/tests/sources-ingestion.test.ts new file mode 100644 index 0000000..0df44c0 --- /dev/null +++ b/tests/sources-ingestion.test.ts @@ -0,0 +1,77 @@ +import { + chmod, + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { addSourcePath, loadSourceRegistry } from '../src/sources' +import { initKnowledgeBase } from '../src/store' + +async function withRoot(fn: (root: string) => Promise): Promise { + const root = await mkdtemp(join(tmpdir(), 'agent-knowledge-sources-')) + try { + await initKnowledgeBase(root) + await fn(root) + } finally { + await rm(root, { recursive: true, force: true }) + } +} + +describe('source path ingestion', () => { + it.skipIf(process.platform === 'win32')( + 'rejects unsupported directory entries without partial raw or registry writes', + async () => { + await withRoot(async (root) => { + const sourceDir = join(root, 'source-dir') + await mkdir(sourceDir) + await writeFile(join(sourceDir, 'good.txt'), 'good source\n') + await symlink(join(sourceDir, 'good.txt'), join(sourceDir, 'linked.txt')) + const originalRegistry = await readFile( + join(root, '.agent-knowledge', 'sources.json'), + 'utf8', + ) + + await expect(addSourcePath(root, sourceDir)).rejects.toThrow(/unsupported directory entry/) + + await expect( + readFile(join(root, '.agent-knowledge', 'sources.json'), 'utf8'), + ).resolves.toBe(originalRegistry) + await expect(readdir(join(root, 'raw', 'sources'))).resolves.toEqual([]) + }) + }, + ) + + it.skipIf(process.platform === 'win32')( + 'preserves source file modes for single-file and directory imports', + async () => { + await withRoot(async (root) => { + const singlePath = join(root, 'single.txt') + await writeFile(singlePath, 'single source\n') + await chmod(singlePath, 0o640) + + const dir = join(root, 'batch') + await mkdir(dir) + const batchPath = join(dir, 'script.txt') + await writeFile(batchPath, 'batch source\n') + await chmod(batchPath, 0o750) + + const [single] = await addSourcePath(root, singlePath) + const [batch] = await addSourcePath(root, dir) + + expect((await stat(join(root, single!.uri))).mode & 0o777).toBe(0o640) + expect((await stat(join(root, batch!.uri))).mode & 0o777).toBe(0o750) + expect((await loadSourceRegistry(root)).sources.map((source) => source.id).sort()).toEqual( + [single!.id, batch!.id].sort(), + ) + }) + }, + ) +})