feat: add Record Provenance across memory records

[edheltzel]

Closes #42

What

Adds Record Provenance — the declared origin and transformation level of every memory record — as automatic write-path metadata per ADR-0001 and the CONTEXT.md glossary.

Schema

• Migration 8→9 adds a nullable provenance column to messages, decisions, learnings, breadcrumbs, and loa_entries, CHECK-constrained to verbatim | user_authored | extracted | derived (NULL allowed — legacy unknown stays representable).
• PRAGMA user_version advances through the existing migration system; fresh-install DDL carries the same column.
• Shared Provenance type propagates through all record interfaces and SearchResult.

Write-path stamping (no public override)

Path	Stamp
MCP memory_add	user_authored — schema exposes no provenance parameter
CLI recall add	user_authored — no --provenance flag
Hook extraction writers (hooks/lib/sqlite-writers.ts)	extracted (with legacy-DB column guard)
PreCompact flush, session import, conversation import, dump messages	verbatim
Structured extraction / LoA capture / import-legacy	extracted
Future derived-record paths	derived reserved in the vocabulary

Backfill — recall provenance backfill

Dry-run by default, --execute to apply. Never guesses: messages → verbatim (only historical write path is raw capture), loa_entries → extracted (all historical writers store machine-generated extracts), decisions/learnings → extracted only where category = 'auto-extracted', breadcrumbs only where category = 'extracted-idea'. Everything else is reported and left NULL. Never overwrites; never assigns user_authored.

Display contract

• Default recall search: quiet for known provenance, visibly flags unknown (⚠ [provenance: unknown]).
• recall search --show-provenance: shows every value.
• MCP memory_search, memory_hybrid_search, and memory_recall payloads carry provenance for every record type (legacy NULL reported as unknown).

Tests (500 pass, lint clean)

• Migration 8→9 upgrade path, CHECK vocabulary, legacy rows stay NULL
• Every write path above, including legacy-DB guards in hooks
• Backfill dry-run vs execute, idempotence, table filter, never-overwrite
• Search display contract (default + --show-provenance)
• search() payload provenance for all five tables
• Source-contract pins: MCP memory_add schema and CLI expose no provenance override

Docs

cli-reference.md, mcp-tools.md, architecture.md, slash-commands.md, /Recall:search command doc, and the FOR_CLAUDE/FOR_PI/FOR_OPENCODE CLI blocks (kept in sync).

Notes

• Scope: the display contract is implemented on recall search (the command that gained --show-provenance); the separate recall semantic/recall hybrid CLI display paths were left untouched.
• Resumes and completes the in-progress worktree from a previously terminated worker; rebased onto main over refactor: conversation imports behind internal source-adapter registry #56 (adapter registry) — the verbatim stamp sits on the single post-refactor write site.

[edheltzel]

Review — PR #58: Record Provenance (issue #42, ADR-0001)

Reviewed at head 49c18f4 against origin/main (32 files, +958/−66). Method: full diff read, codegraph caller audit of every memory writer, raw-INSERT sweep across src/ + hooks/, ADR-0001/CONTEXT.md/issue-#42 constraint check, plus four specialist passes (code quality, test coverage, silent failures, comment/doc accuracy). CI is green on the head commit (macOS primary + Ubuntu smoke, run headSha 49c18f4 matches).

Binding design constraints — all verified ✅

1. No public provenance override. The MCP memory_add zod schema exposes no provenance parameter (handler stamps user_authored); the CLI has no --provenance flag; hook inputs (DecisionInput/LearningInput/BreadcrumbInput/LoaInput) carry no provenance field — sqlite-writers.ts stamps a SQL literal the caller can't reach. Both absences are pinned by tests (tests/lib/provenance-write-paths.test.ts).
2. Capture stays frictionless. No classification prompt anywhere; every stamp is automatic from the write path.
3. Vocabulary correct. PROVENANCE_VALUES = user_authored | verbatim | extracted | derived, unknown = NULL (never a stored value), CHECK-enforced, matching CONTEXT.md and the Add non-destructive dedup with provenance-aware survivor selection #45 survivor priority.
4. Write-path completeness (codegraph codegraph_callers audit + raw-INSERT sweep): every caller of addMessagesBatch (coreDump, importConversations, importAllSessions) stamps verbatim; every caller of addDecision/addLearning/addBreadcrumb (CLI add → user_authored, structured extraction → extracted, MCP memory_add → user_authored) is stamped; all four createLoaEntry callers stamp extracted; the hook funnel (extraction-parsers.ts → sqlite-writers.ts, PreCompact) stamps with legacy-DB column guards. decision supersede only UPDATEs status. No unstamped write path found.
5. Migration 8→9 follows the existing migration system exactly (try/catch-ignore ALTER, same idiom as migrations 2→3, 5→6, 7→8; user_version advances through the runner). Additive nullable column, legacy rows stay NULL, upgrade path + CHECK vocabulary + NULL semantics all behaviorally tested, including against a simulated user_version = 8 legacy DB.
6. Backfill is genuinely conservative. Evidence markers (category = 'auto-extracted' / 'extracted-idea') verified to match exactly what the extraction writers stamp (checked against current code and git history); the blanket rules hold (runLoa is Fabric-mandatory, all historical LoA/message writers verified machine-extract/raw-transcript). provenance IS NULL gates every UPDATE (never overwrites, idempotent — tested); user_authored is never assigned.

Scope judgment: recall search-only display — acceptable

Issue #42's display contract is written around the search command and its --show-provenance flag, and the structured-MCP criterion is satisfied (memory_search, memory_hybrid_search, memory_recall all carry provenance; recent* use SELECT * so the values are real). The PR body discloses the scope decision honestly. Two residuals belong in a follow-up issue rather than this PR:

• Bare recall <query> defaults to the hybrid display path (embed.ts), which shows no provenance flagging at all — arguably the most-used CLI search surface.
• recall semantic / recall hybrid likewise untouched.

Non-blocking findings (ordered by priority)

1. src/mcp-server.ts:193-227 — hybrid vec-branch omits learnings. The vector-result content-fetch handles loa_entries/decisions/messages but learnings are also embedded (embed.ts backfills source_table = 'learnings'). A learning matched only by vector search returns empty content (pre-existing) and now provenance: unknown even though the row almost certainly carries extracted — "unknown" is specced to mean the DB doesn't know, not this branch didn't look. Narrow trigger (vec-only match, Ollama required), but it's the one place the PR misreports known provenance. Fix here or in the display follow-up issue; consider a logged default: case so the next embedded table can't regress silently.
2. src/db/migrations.ts:189-200 — bare catch {} on the ALTER. Repo-conventional (2→3, 5→6, 7→8 do the same), but this instance is the riskiest yet: it spans 5 tables, and src/lib/memory.ts now hard-references the column on both write and read (search()) paths — a non-duplicate ALTER failure would advance user_version past a missing column with no recovery path. Recommend the columnExists/PRAGMA table_info guard (the idiom already exists in hooks/lib/sqlite-writers.ts) or rethrow-unless-duplicate column name. At minimum the comment ("Column already exists — safe to ignore") claims narrower behavior than the code enforces.
3. Test gaps on outer stamps. coreDump (verbatim messages + extracted LoA), runLoa, and importAllSessions stamping have no behavioral test — pinned only by comments. Mitigation: a regression yields NULL rows the backfill classifies correctly, so data is recoverable — hence non-blocking. The MCP memory_add stamp is pinned only by source-text scraping; the schema-absence half of that test is a legitimate ADR tripwire, but the handler-stamps half should become behavioral (extract the handler into an importable function — good follow-up). Also add a vacuity guard (expect(handlerStart).toBeGreaterThan(-1)) so a refactor can't make the scrape pass on an empty slice.
4. src/commands/provenance.ts:115-118 — live report prints the pre-counted number. --execute reports the COUNT(*) taken before the UPDATE rather than .run().changes; the DB is shared with live hooks, so the printed count can drift from what was written. One-line fix.
5. src/types/index.ts:5-6 — citation nit. "Survivor-order vocabulary … (ADR-0001, CONTEXT.md)": neither cited source defines an ordering — it comes from issue Add non-destructive dedup with provenance-aware survivor selection #45's "Survivor priority is user_authored > verbatim > extracted > derived > unknown". Cite Add non-destructive dedup with provenance-aware survivor selection #45 (and fold the precedence into CONTEXT.md when Add non-destructive dedup with provenance-aware survivor selection #45 lands) so the comment doesn't read as fabricated.
6. Minor: provenance.ts evidence markers are write-path convention, not schema constraint (recall add decision --category auto-extracted would be backfilled as extracted) — worth one caveat line; a guard comment on the backfill's literal SQL interpolation (RULES is module-local, never caller input — keep it that way); columnExists catch-to-false in sqlite-writers.ts can degrade a real extracted stamp to NULL on a transient PRAGMA failure (window is tiny, backfill recovers — comment-worthy only).

Out of scope for this PR (already merged via #55/#56, surfaced during review)

• tests/helpers/concurrency_worker.ts sets busy_timeout after journal_mode = WAL — the extraction-semaphore flake the feat(db): add shared SQLite-safe chunking for input-scaled bind lists #55 fix targeted still fires under full-suite load (reproduced once in this review; passes in isolation). Worth its own issue.
• Adapter registry if (!adapter) continue silently skips files for an unknown programmatic format (CLI path is guarded).

Strengths

The ADR constraint is enforced and pinned by tripwire tests; the backfill's "never guess / never overwrite / never user_authored" invariants are behaviorally tested including idempotence and the pre-stamped-row accounting; both hook surfaces test the legacy-DB degradation path; the NULL-stays-NULL invariant has its own named test; docs are synced across cli-reference/mcp-tools/architecture/slash-commands and all three platform guides per the DRY convention; comment factual claims (evidence markers, write-path history, CHECK/NULL semantics) verify against code and git history.

Recommended follow-ups (for Themis to file/triage)

1. Provenance display on recall semantic / recall hybrid / bare recall <query> + the hybrid vec-branch learnings fix (finding 1).
2. Extract the MCP memory_add handler for behavioral stamp testing (finding 3).
3. concurrency_worker.ts pragma ordering (out-of-scope flake).

---

Verdict: APPROVE — all ADR-0001 binding constraints hold end-to-end, every write path is stamped (codegraph-audited), the migration is safe and convention-true, the backfill never guesses, CI is green on the head commit, and the scope cut is disclosed and reasonable. Findings above are non-blocking.