diff --git a/.github/instructions/local-db-grounding.instructions.md b/.github/instructions/local-db-grounding.instructions.md new file mode 100644 index 0000000..1a43766 --- /dev/null +++ b/.github/instructions/local-db-grounding.instructions.md @@ -0,0 +1,61 @@ +--- +description: "In a checkout, the resolved ros-help.db is untrusted until verified; the latest CI release DB is the grounding source of truth." +applyTo: "src/paths.ts, src/setup.ts, src/mcp.ts, scripts/db-doctor.ts, Makefile, MANUAL.md, VALIDATION.md" +--- +# Local DB grounding + +Dev mode (`resolveDbPath` → project root) serves the repo-root `ros-help.db` — a +git-ignored file whose contents are arbitrary local state. **It is not trusted by +default.** Before grounding any claim about shipped data on it, verify it. + +## The rule + +- **The latest CI-built release DB is the source of truth for grounding claims.** + It is what `bunx @tikoci/rosetta` consumers actually get. `make db-sync` + (`scripts/db-sync.ts`) fetches it into the resolved path (atomic replace — safe + while an MCP server holds the old file open). + - It does **not** use `--refresh`: that pins the URL to `package.json`'s + version, which in a checkout is a CI-rewritten placeholder (`v0.11.0-rc.0`) + with no release, and falls back to `/releases/latest` — the newest *stable*, + which lags the prerelease schema. `db-sync` instead uses `gh` to find the + newest release (prereleases **included**) that actually ships + `ros-help.db.gz`, then reuses the hardened `downloadDb()` (schema/content + validation + stale-sidecar cleanup + atomic swap) via its `urlsOverride` + param. Requires the `gh` CLI. Keep it this way. +- **`make extract` produces an *unstamped* local working DB** — for extraction + and pipeline work, not for claims about shipped data. `db-doctor` reports it as + `unstamped`; that is expected, not a failure to "fix" by re-extracting. +- **Verify before trusting.** `make db-doctor` (or `bun run db:doctor`) prints the + resolved path + `db_meta` provenance + a grounding verdict and exits non-zero + when not `ok`. In-session, one `routeros_stats` call returns the same + `provenance.grounding` block — prefer it over shelling into sqlite. + +## The verdict (`classifyDbGrounding`, `src/paths.ts`) + +Pure function, shared by `routeros_stats` (`getDbStats`), MCP startup, and +`db-doctor`. Status precedence: + +| status | meaning | typical fix | +|---|---|---| +| `schema_mismatch` | `PRAGMA user_version` ≠ code `SCHEMA_VERSION` | `make db-sync` | +| `internal_inconsistent` | `db_meta.schema_version` ≠ pragma — a corpus bumped in place by `initDb()`; provenance no longer describes the bytes (the #94 "Frankenstein") | `make db-sync` | +| `unstamped` | neither `release_tag` nor `source_commit` — a local `make extract` build | fine for extraction; `make db-sync` to ground | +| `provenance_incomplete` | claims release identity but is missing one of the four CI stamps (`release_tag`/`source_commit`/`built_at`/`schema_version`), or a stamped version that won't parse — fail closed | `make db-sync` | +| `tag_behind` | `release_tag` base version behind the checkout | `make db-sync` | +| `ok` | all four stamps present, schema coherent, tag current | — | + +`ok` means the DB is **schema/release-compatible** with this build (coherent schema, complete provenance, release not behind) — not proof it was built from the exact checked-out commit. A release DB is legitimately built from an ancestor commit; `db-doctor` reports that source commit for inspection. + +The rc/beta counter is intentionally ignored — a dev checkout's `package.json` +routinely reads `-rc.0` while the correct published DB is a much higher rc. Only +the `MAJOR.MINOR.PATCH` base is a staleness axis. + +## Startup behavior + +In dev mode, MCP startup emits a **loud but non-fatal** banner when the verdict +is not `ok` (never fetches — a contributor's local build must not be clobbered). +`checkDbFreshness` still hard-fails a genuine schema mismatch and auto-redownloads +in package/compiled mode; this warning is purely additive surfacing. + +Keep the classifier, its three consumers, and this file in sync when the verdict +shape or `db_meta` key set changes. Cross-reference: `db-meta-stamping.instructions.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 15af56c..a4f7c0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ uses [Semantic Versioning](https://semver.org/). ### Added +- **The resolved DB now reports whether it can be trusted to ground claims about the code querying it** (issue #94, B-0022 export-audit umbrella). `routeros_stats` gains a `provenance` block — resolved `db_path`, invocation `mode`, the `db_meta` stamps (`release_tag`/`source_commit`/`built_at`/`schema_version`), and a **grounding verdict** (`ok` / `schema_mismatch` / `internal_inconsistent` / `tag_behind` / `unstamped`) — so one call self-checks the DB against the checkout instead of shelling into sqlite (the TUI `stats` screen shows it too, via shared `getDbStats`). A new `make db-doctor` / `bun run db:doctor` prints the same verdict and exits non-zero when not `ok` (CI/pre-commit-ready); `make db-sync` / `bun run db:sync` fetches the latest CI release DB — the documented grounding source of truth — into the resolved path (atomic, safe while a server holds the file open). In a dev checkout, MCP startup now emits a loud but **non-fatal** banner when the served DB has drifted (e.g. a corpus whose `PRAGMA user_version` was bumped in place so its stamped provenance no longer describes its bytes); it never fetches, so a contributor's local build is never clobbered. The load-bearing new signal is `internal_inconsistent` (`db_meta.schema_version` ≠ pragma), which every prior check missed. Durable guidance lives in `.github/instructions/local-db-grounding.instructions.md`. + - **New `rosetta export ` command writes a DB-only dataset directory for local audit** (issue #101, the B-0022 export track). Produces `manifest.toml` plus seven flat TSVs — `changelog.tsv`, `callouts.tsv`, `properties.tsv` (joined to pages with a resolved `section_anchor`), `videos.tsv` (metadata + per-video transcript segment/word/byte counts), `commands.tsv`, and the paired `pages.tsv` (one row per page) / `sections.tsv` (one row per section, pivot-able) — from the runtime SQLite database alone — dataset generation reads only the resolved DB, with no caches or re-extraction (startup may still download/refresh the DB, as every command does). The serialization contract is a reversible Postgres-COPY-style escape (`\t`/`\n`/`\r`/`\\` backstopped on every value, whole-field `\N` for SQL NULL as distinct from the empty string) with stable ordering so a rebuild on the same DB is byte-identical, and `manifest.toml` carries `db_meta` provenance, per-file row counts, and honest disclosures of what the DB cannot provide (transcript provenance #21, per-version architecture #91). A column the DB cannot produce is omitted and disclosed, never recovered from a source artifact. - **`rosetta export` now emits `tables.tsv`, an inventory of every captured doc table** (issue #111, under the #104 B-0022 export track). One row per `page_tables` record (855 on the rc.99 corpus) with page/section provenance, the table's shape (`column_count`/`data_row_count`/`is_ragged`), source size (`raw_bytes`, UTF-8 length of the stored Markdown), and a `table_url` that deep-links to the section containing the table (`url#anchor`; the bare page URL when no section resolves, since Docusaurus has no per-table anchor). This is the "what tables exist and where" list — it answers most table-audit questions without exporting the cell data itself (the per-fragment cell files remain E4). Reuses the settled TSV/manifest contract and is pruned/owned the same way as every other dataset. diff --git a/CLAUDE.md b/CLAUDE.md index d05338c..23f45ba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,6 +61,7 @@ Agent-specific entrypoints are thin shims: Claude starts here, Codex starts with - Need **manual-site migration context**? MikroTik moved future docs to Docusaurus at ; read `DESIGN.md` and `briefings/B-0012-docusaurus-manual-migration.md` before touching extraction, MCP, or TUI search. - Need **external agent-benchmark context**? `~/GitHub/bench-routeros-tools` tests rosetta as one grounding approach for live RouterOS command generation; read `DESIGN.md` → "External benchmark feedback loop" before changing retrieval, skills, or command-validation surfaces. - Need **current corpus counts**? Use `routeros_stats` rather than hard-coded numbers here. +- Need to **trust the local DB before grounding a claim**? In a checkout, repo-root `ros-help.db` is untrusted until verified. Run `make db-doctor` (or read `routeros_stats` → `provenance.grounding`); `make db-sync` fetches the latest CI release DB (the grounding source of truth). See `.github/instructions/local-db-grounding.instructions.md`. - Need **work status or open ideas**? Use `gh issue list` (active work), `briefings/`, and `BACKLOG.md`; `tasks/` is a frozen archive. Nothing else should be duplicated here. If this file starts growing operational detail, schema blocks, or long rule lists again, move that content back into its canonical home instead. diff --git a/MANUAL.md b/MANUAL.md index df9b8d0..7ab0af9 100644 --- a/MANUAL.md +++ b/MANUAL.md @@ -115,6 +115,16 @@ The database combines multiple MikroTik data sources into a single SQLite file w Documentation covers RouterOS **v7 only**. v6 had different syntax and major subsystems — answers for v6 are unreliable. Prose is now sourced live from , which also has a CLI Reference with `/console/inspect`-derived command menus and argument types (not yet ingested — see B-0012's proposed follow-up tasks). +## Local DB grounding (dev checkouts) + +In a checkout, dev mode serves the repo-root `ros-help.db` — a git-ignored file whose contents are arbitrary local state (a leftover `make extract` build, a partially-migrated corpus, or a schema bumped in place over stale data). **It is not trusted by default.** Before grounding a claim about shipped data on it, verify it. + +- **`make db-doctor`** (or `bun run db:doctor`) prints the resolved DB path, its `db_meta` provenance, and a grounding verdict; exits non-zero when the verdict is not `ok`. In-session, one `routeros_stats` call returns the same `provenance.grounding` block. +- **`make db-sync`** (or `bun run db:sync`) fetches the newest CI-built release DB **matching this checkout's schema** into the resolved path — the **source of truth** for grounding claims, since it is what `bunx @tikoci/rosetta` consumers get. It uses `gh` to find the newest release that ships the DB asset, prereleases included (not `--refresh`, which would grab the newest *stable* and its older schema). The replace is atomic, so it is safe to run while an MCP server holds the old file open. Requires the `gh` CLI. +- A local `make extract` build is intentionally **unstamped** (no `release_tag`/`source_commit`) — fine for extraction and pipeline work, but `db-doctor` reports it as `unstamped` to signal it is not a grounding source. + +Verdict statuses: `schema_mismatch` (pragma ≠ code schema), `internal_inconsistent` (`db_meta.schema_version` ≠ pragma — a corpus bumped in place, the classic "which DB is this?" trap), `unstamped` (a local build with no release stamps), `provenance_incomplete` (claims release identity but is missing one of the four CI stamps — fail closed), `tag_behind`, and `ok`. `ok` means schema/release-**compatible** with this build, not that the DB was built from the exact checked-out commit (a release DB is built from an ancestor). In dev mode, MCP startup emits a loud but non-fatal banner when the verdict is not `ok` (it never fetches — a contributor's local build is never clobbered). See `.github/instructions/local-db-grounding.instructions.md`. + ## Re-extracting a Local Database `make extract` and `make extract-full` fetch `/docs` prose live from manual.mikrotik.com and rebuild the versioned command tree, devices, changelogs, Dude cache, and skills: diff --git a/Makefile b/Makefile index fce892e..4a3b2c0 100644 --- a/Makefile +++ b/Makefile @@ -11,6 +11,7 @@ VERSION ?= link gc-versions assess assess-hardware assess-www device-map device-map-check \ extract-hardware-catalog search browse serve \ typecheck lint test preflight verify \ + db-doctor db-sync \ install setup clean eval eval-update eval-self eval-self-update # ── Development ── @@ -21,6 +22,22 @@ install: serve: bun run src/mcp.ts +# Report whether the resolved ros-help.db matches this checkout (#94). Prints the +# resolved path + db_meta provenance + a grounding verdict; non-zero when not "ok". +# The latest CI-built release DB is the source of truth for grounding claims — a +# local `make extract` build is for extraction work, not for claims about shipped data. +db-doctor: + bun run scripts/db-doctor.ts + +# Fetch the newest CI-built release DB that matches this checkout's schema into +# the resolved DB path (dev: repo root), so the MCP server / db-doctor ground on +# what npm consumers actually get. Discovers the newest release shipping the DB +# asset via `gh` (prereleases included — `--refresh`/`/releases/latest` would +# grab the newest *stable*, an older schema). Atomic replace — safe to run while +# an MCP server holds the old file open. Requires the `gh` CLI. +db-sync: + bun run scripts/db-sync.ts + search: bun run src/search.ts $(query) diff --git a/VALIDATION.md b/VALIDATION.md index da5ecf8..124f071 100644 --- a/VALIDATION.md +++ b/VALIDATION.md @@ -27,6 +27,7 @@ Tasks in `tasks/` reference these IDs in their `validation:` frontmatter. When a | V-db-min-content | Built DB has ≥200 pages, ≥1000 commands, ≥100 devices, ≥1000 properties, ≥800 generic page tables, ≥8000 table data rows, ≥4000 properties linked to exact table rows, ≥200 hardware_catalog, ≥600 device_aliases before publish | `qa.yml` db-content gate (single definition of the floors; run by `release.yml`'s `qa` job with `db_source=artifact`, so it floors the exact DB about to publish) | blocking | — | | V-db-wipe-guard | Tests cannot accidentally open and overwrite the on-disk DB | `src/query.test.ts` `:memory:` guard + post-extraction guard in CI | blocking | — | | V-db-meta | Released DB carries `db_meta` provenance (release_tag, built_at, source_commit, schema_version) | `scripts/stamp-db-meta.ts` in `release.yml`'s `build` job; presence re-checked by `qa.yml`'s db-meta gate | blocking | — | +| V-db-grounding | The grounding verdict (`classifyDbGrounding`) correctly classifies a schema-coherent-but-stale corpus as `internal_inconsistent`, an unstamped local build as `unstamped`, and a coherent CI artifact as `ok`; `make db-doctor` exits non-zero on any non-`ok` verdict | `src/paths.test.ts` (`classifyDbGrounding` cases) + `src/query.test.ts` (`getDbStats` provenance) in `bun test`; `scripts/db-doctor.ts` exit code | blocking (unit) | #94 | | V-bunx-macos | `bunx @tikoci/rosetta` installs + serves on macOS | `bunx-smoke` matrix in `release.yml` | blocking | — | | V-bunx-linux | Same on Linux | `bunx-smoke` matrix in `release.yml` | blocking | — | | V-bunx-windows | Same on Windows — catches v0.8.x EBUSY / readonly-WAL / temp-file class of bugs | `windows-latest` row in `bunx-smoke` matrix in `release.yml` | blocking | — | diff --git a/package.json b/package.json index 3f5c747..fcd5da8 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,9 @@ "lint": "biome check src/", "lint:md": "markdownlint-cli2 \"**/*.md\"", "lint:spell": "cspell lint . --no-progress", - "browse": "bun run src/mcp.ts browse" + "browse": "bun run src/mcp.ts browse", + "db:doctor": "bun run scripts/db-doctor.ts", + "db:sync": "bun run scripts/db-sync.ts" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.27.1", diff --git a/scripts/db-doctor.ts b/scripts/db-doctor.ts new file mode 100644 index 0000000..2b9b90c --- /dev/null +++ b/scripts/db-doctor.ts @@ -0,0 +1,59 @@ +#!/usr/bin/env bun + +/** + * db-doctor.ts — report whether the resolved ros-help.db can be trusted to + * ground claims about this checkout (#94). + * + * Resolves the DB exactly as the MCP server would (DB_PATH / --db / auto-detect), + * reads its db_meta provenance, and prints a grounding verdict. Exits non-zero + * when the verdict is not "ok", so it doubles as a CI/pre-commit gate. + * + * Usage: + * bun run scripts/db-doctor.ts # resolved DB (dev: repo-root ros-help.db) + * DB_PATH=/path/to/ros-help.db bun run scripts/db-doctor.ts + * + * Fix a stale/mismatched DB with `make db-sync` (fetches the latest CI release DB). + */ + +import { classifyDbGrounding, detectMode, resolveDbPath, resolveVersion, SCHEMA_VERSION } from "../src/paths.ts"; +import { probeDb } from "../src/setup.ts"; + +const dbPath = resolveDbPath(import.meta.dirname); +const mode = detectMode(import.meta.dirname); +const codeVersion = resolveVersion(import.meta.dirname); + +console.log(`rosetta db-doctor`); +console.log(` code : v${codeVersion} (schema v${SCHEMA_VERSION}, mode ${mode})`); +console.log(` resolved DB : ${dbPath}`); + +const p = probeDb(dbPath); +if (!p) { + console.log(` status : NO_DB — no readable SQLite database at the resolved path.`); + console.log(` fix : run 'make db-sync' to fetch the latest CI release DB.`); + process.exit(1); +} + +const verdict = classifyDbGrounding({ + pragmaSchema: p.schemaVersion, + metaSchema: p.metaSchemaVersion, + releaseTag: p.releaseTag, + builtAt: p.builtAt, + sourceCommit: p.sourceCommit, + codeSchema: SCHEMA_VERSION, + codeVersion, + mode, +}); + +console.log(` release_tag : ${p.releaseTag ?? "(unstamped)"}`); +console.log(` source_commit: ${p.sourceCommit ?? "(unstamped)"}`); +console.log(` built_at : ${p.builtAt ?? "(unstamped)"}`); +console.log(` schema : pragma=${p.schemaVersion} meta=${p.metaSchemaVersion ?? "—"} code=${SCHEMA_VERSION}`); +console.log(` pages : ${p.pages} commands: ${p.commands}`); +console.log(` status : ${verdict.status.toUpperCase()}`); +console.log(` detail : ${verdict.detail}`); + +if (!verdict.ok) { + console.log(` fix : run 'make db-sync' to fetch the latest CI release DB for grounding.`); + process.exit(1); +} +process.exit(0); diff --git a/scripts/db-sync.ts b/scripts/db-sync.ts new file mode 100644 index 0000000..79785a5 --- /dev/null +++ b/scripts/db-sync.ts @@ -0,0 +1,75 @@ +#!/usr/bin/env bun + +/** + * db-sync.ts — install the newest CI-built release DB that matches this checkout, + * into the resolved DB path (#94). + * + * Why not `--refresh`? refreshDb() pins its download URL to package.json's + * version, which in a checkout is a CI-rewritten placeholder (e.g. v0.11.0-rc.0) + * with no matching release, then falls back to /releases/latest — the newest + * *stable*, which lags the prerelease line and ships an older schema. This script + * instead asks GitHub for the newest release (prereleases included) that actually + * ships ros-help.db.gz, and installs that via the hardened downloadDb() path + * (lock + schema/content validation + stale-sidecar cleanup + atomic swap), so it + * is safe to run while an MCP server holds the old file open. + * + * Requires `gh` (already used across this repo). Public repo — no auth needed to + * read releases, but `gh` handles auth transparently if present. + * + * Usage: + * bun run scripts/db-sync.ts # → resolved DB path (dev: repo root) + * DB_PATH=/path bun run scripts/db-sync.ts + */ + +import { $ } from "bun"; +import { resolveDbPath, SCHEMA_VERSION } from "../src/paths.ts"; +import { downloadDb } from "../src/setup.ts"; + +const REPO = "tikoci/rosetta"; +const ASSET = "ros-help.db.gz"; +const dbPath = resolveDbPath(import.meta.dirname); + +// All non-draft releases (prerelease OR stable) that carry the DB asset, newest +// first (the releases API returns newest-first by created_at). We hand the WHOLE +// ordered list to downloadDb: it validates each candidate's schema against this +// build and walks down to the newest release that actually MATCHES — so an older +// checkout still resolves a compatible DB even after a newer, higher-schema +// release exists above it. Selecting only the newest tag would fail there. +const jq = `[.[] | select(.draft | not) | select([.assets[].name] | index("${ASSET}")) | .tag_name]`; + +let tags: string[] = []; +try { + const out = (await $`gh api ${`repos/${REPO}/releases?per_page=50`} --jq ${jq}`.text()).trim(); + tags = JSON.parse(out) as string[]; +} catch (e) { + console.error("✗ Could not query GitHub releases via gh."); + console.error(` ${e instanceof Error ? e.message : e}`); + console.error(" Install/authenticate the gh CLI, or fetch a release DB manually (MANUAL.md)."); + process.exit(1); +} + +if (tags.length === 0) { + console.error(`✗ No release ships ${ASSET} yet — nothing to sync.`); + process.exit(1); +} + +const urls = tags.map((tag) => `https://github.com/${REPO}/releases/download/${tag}/${ASSET}`); +console.log(`Releases with ${ASSET} (newest first): ${tags.join(", ")}`); +console.log(`Target schema: v${SCHEMA_VERSION} Target DB path: ${dbPath}`); + +try { + const probe = await downloadDb(dbPath, console.log, urls); + console.log(`✓ Synced: schema v${probe.schemaVersion}, ${probe.pages} pages, release ${probe.releaseTag ?? "unknown"}.`); + process.exit(0); +} catch (e) { + const msg = e instanceof Error ? e.message : String(e); + console.error(`✗ Sync failed: ${msg}`); + if (msg.includes("schema=")) { + console.error( + ` None of the ${tags.length} candidate release(s) ship a DB matching this checkout's schema (v${SCHEMA_VERSION}).\n` + + ` This checkout is likely ahead of any published release DB. Use 'make extract' for a local\n` + + ` (unstamped) working DB until a matching release is published.`, + ); + } + process.exit(1); +} diff --git a/src/browse.ts b/src/browse.ts index 0e4098a..59889f7 100644 --- a/src/browse.ts +++ b/src/browse.ts @@ -1096,7 +1096,12 @@ function renderStats(): string { if (stats.schema_version != null) { out.push(` ${dim("Schema:")} v${stats.schema_version}`); } - out.push(` ${dim("Export:")} ${stats.doc_export}`); + out.push(` ${dim("Provenance:")} ${stats.doc_export}`); + const grounding = stats.provenance?.grounding; + if (grounding) { + const label = grounding.ok ? "OK" : grounding.status.toUpperCase(); + out.push(` ${dim("Grounding:")} ${label}${grounding.ok ? "" : ` — ${grounding.detail}`}`); + } out.push(""); const kv = (label: string, value: string | number) => { diff --git a/src/db.ts b/src/db.ts index 71004b3..3c4886a 100644 --- a/src/db.ts +++ b/src/db.ts @@ -38,7 +38,7 @@ */ import sqlite from "bun:sqlite"; -import { resolveDbPath, SCHEMA_VERSION } from "./paths.ts"; +import { classifyDbGrounding, detectMode, resolveDbPath, resolveVersion, SCHEMA_VERSION } from "./paths.ts"; export { SCHEMA_VERSION }; @@ -1030,10 +1030,43 @@ export function getDbStats() { return null; } })(); + const provenance = (() => { + const releaseTag = getDbMeta("release_tag"); + const sourceCommit = getDbMeta("source_commit"); + const builtAt = getDbMeta("built_at"); + const metaSchemaRaw = getDbMeta("schema_version"); + const metaSchema = metaSchemaRaw === null ? null : Number(metaSchemaRaw); + const mode = detectMode(import.meta.dirname); + const codeVersion = resolveVersion(import.meta.dirname); + const grounding = classifyDbGrounding({ + pragmaSchema: schemaVersion ?? -1, + metaSchema, + releaseTag, + builtAt, + sourceCommit, + codeSchema: SCHEMA_VERSION, + codeVersion, + mode, + }); + return { + db_path: DB_PATH, + mode, + is_ci_artifact: releaseTag !== null && sourceCommit !== null, + release_tag: releaseTag, + source_commit: sourceCommit, + built_at: builtAt, + schema_version_meta: metaSchema, + schema_version_pragma: schemaVersion, + code_schema_version: SCHEMA_VERSION, + code_version: codeVersion, + grounding, + }; + })(); return { db_path: DB_PATH, db_size_bytes: dbSizeBytes, schema_version: schemaVersion, + provenance, pages: count("SELECT COUNT(*) AS c FROM pages"), sections: count("SELECT COUNT(*) AS c FROM sections"), properties: count("SELECT COUNT(*) AS c FROM properties"), @@ -1079,7 +1112,16 @@ export function getDbStats() { versions.sort(cmp); return { ros_version_min: versions[0], ros_version_max: versions[versions.length - 1] }; })(), - doc_export: "2026-03-25 (Confluence HTML)", + // Derived from db_meta provenance, not hard-coded — a hard-coded export date + // silently lied whenever the resolved DB was a different corpus (#94). The + // string is self-describing (it is a DB build stamp, not a doc-export date), + // and a stamped artifact missing only `built_at` reports "build time unknown" + // rather than being mislabeled a local build. + doc_export: provenance.built_at + ? `DB built ${provenance.built_at}${provenance.release_tag ? ` (release ${provenance.release_tag})` : ""}` + : provenance.is_ci_artifact + ? `DB build time unknown${provenance.release_tag ? ` (release ${provenance.release_tag})` : ""}` + : "unstamped local build", }; } diff --git a/src/mcp.ts b/src/mcp.ts index c2edf20..e77d669 100644 --- a/src/mcp.ts +++ b/src/mcp.ts @@ -71,7 +71,7 @@ function link(url: string, display?: string): string { * crashing startup over it. */ async function ensureDbReady(log: (msg: string) => void): Promise { - const { resolveDbPath, SCHEMA_VERSION, resolveVersion, detectMode } = await import("./paths.ts"); + const { resolveDbPath, SCHEMA_VERSION, resolveVersion, detectMode, classifyDbGrounding, isDevInvocation } = await import("./paths.ts"); const { checkDbFreshness, cleanupAbandonedTempArtifacts, downloadDb, hasMinimumDbContent, probeDb } = await import( "./setup.ts" ); @@ -165,6 +165,30 @@ async function ensureDbReady(log: (msg: string) => void): Promise { // Quietly emit a one-line provenance banner so MCP-client logs show what's loaded. const tagInfo = p.releaseTag ? `, release ${p.releaseTag}` : ""; log(`rosetta v${runningVersion} ready (DB schema v${p.schemaVersion}, ${p.pages} pages${tagInfo}).`); + + // Dev-mode grounding warning (#94): checkDbFreshness intentionally ignores + // release-tag drift in a checkout so it never clobbers a contributor's local + // build — but that means a schema-matching, wrong-corpus DB (e.g. a pragma + // bumped in place over a stale corpus) starts up silently. Surface it loudly. + // Never triggers a fetch; the served DB is unchanged. + if (isDevInvocation(mode)) { + const verdict = classifyDbGrounding({ + pragmaSchema: p.schemaVersion, + metaSchema: p.metaSchemaVersion, + releaseTag: p.releaseTag, + builtAt: p.builtAt, + sourceCommit: p.sourceCommit, + codeSchema: SCHEMA_VERSION, + codeVersion: runningVersion, + mode, + }); + if (!verdict.ok) { + log(`⚠ DB grounding: ${verdict.status.toUpperCase()} — ${verdict.detail}`); + log(` Resolved DB: ${dbPath}`); + log(` This DB may not match the code you are reading. Run 'make db-sync' to fetch the latest CI release DB,`); + log(` or 'bun run db:doctor' for full provenance. See MANUAL.md → "Local DB grounding (dev checkouts)".`); + } + } } if (args.includes("--version") || args.includes("-v")) { @@ -917,15 +941,21 @@ server.registerTool( { description: `Get database statistics for the RouterOS documentation index. -Returns page count, property count, callout count, changelog count, command count, link coverage, -version range, documentation export date, and available agent skills. +Returns live corpus counts (pages, properties, callouts, changelogs, commands, link coverage), +version range, available agent skills, and a **provenance** block: the resolved DB path, its +db_meta stamps (release_tag / source_commit / built_at / schema_version), and a **grounding +verdict** (status ok / schema_mismatch / internal_inconsistent / provenance_incomplete / tag_behind +/ unstamped). Use the verdict to confirm the DB you are querying is schema/release-compatible with +the code you are reading before trusting counts — "ok" means compatible (schema coherent, all four +stamps present, release not behind), not that the DB was built from this exact commit (#94). All +counts here are live — do not rely on any numbers baked into prose elsewhere. Skills: Community-created agent guides from tikoci/routeros-skills are available as MCP resources at rosetta://skills/{name}. Use the resource listing to browse available skills. Knowledge boundaries: -- Documentation: March 2026 Confluence HTML export (317 pages), aligned with long-term ~7.22 -- Command tree: RouterOS 7.9–7.23beta2 from inspect.json (with extra-packages from CHR) +- Documentation corpus: see provenance.built_at / provenance.release_tag for what this DB actually is +- Command tree: RouterOS 7.9+ from inspect.json (with extra-packages from CHR); see ros_version range - No RouterOS v6 data available — v6 syntax and subsystems differ significantly from v7 - For versions older than 7.9, no command tree data exists - Versions older than current long-term are unpatched by MikroTik diff --git a/src/paths.test.ts b/src/paths.test.ts new file mode 100644 index 0000000..2033533 --- /dev/null +++ b/src/paths.test.ts @@ -0,0 +1,166 @@ +/** + * paths.test.ts — grounding verdict classifier (#94). + * + * classifyDbGrounding is pure (no DB), so it is tested directly with the exact + * provenance shapes the three surfaces (routeros_stats, MCP startup, db-doctor) + * feed it. The load-bearing case is `internal_inconsistent`: the real-world + * "Frankenstein" DB whose PRAGMA user_version was bumped in place over a stale + * corpus, which every other check misses. + */ + +import { describe, expect, test } from "bun:test"; +import { classifyDbGrounding } from "./paths.ts"; + +const base = { + codeSchema: 10, + codeVersion: "0.11.0-rc.0", + mode: "dev" as const, +}; + +describe("classifyDbGrounding", () => { + test("coherent CI artifact on the same base version → ok (rc counter ignored)", () => { + // Dev package.json reads rc.0 while the published DB is rc.102 — same base, + // must NOT be flagged behind. + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "v0.11.0-rc.102", + sourceCommit: "abc123", + builtAt: "2026-07-17T00:00:00.000Z", + }); + expect(v.status).toBe("ok"); + expect(v.ok).toBe(true); + }); + + test("Frankenstein: db_meta.schema_version disagrees with pragma → internal_inconsistent", () => { + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, // bumped in place → matches code, so NOT schema_mismatch + metaSchema: 5, // stamped provenance still describes the old corpus + releaseTag: "v0.10.0", + sourceCommit: "65fc229", + builtAt: "2026-07-08T00:00:00.000Z", + }); + expect(v.status).toBe("internal_inconsistent"); + expect(v.ok).toBe(false); + }); + + test("pragma below code schema → schema_mismatch (takes precedence over everything)", () => { + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 8, + metaSchema: 8, + releaseTag: "v0.11.0-rc.50", + sourceCommit: "abc123", + builtAt: "2026-07-01T00:00:00.000Z", + }); + expect(v.status).toBe("schema_mismatch"); + }); + + test("no release_tag / source_commit → unstamped (a local extraction build)", () => { + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, + metaSchema: 10, + releaseTag: null, + sourceCommit: null, + builtAt: null, + }); + expect(v.status).toBe("unstamped"); + expect(v.ok).toBe(false); + }); + + test("release base behind the running code → tag_behind", () => { + const v = classifyDbGrounding({ + ...base, + codeVersion: "0.12.0-rc.0", + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "v0.11.0-rc.102", + sourceCommit: "abc123", + builtAt: "2026-07-17T00:00:00.000Z", + }); + expect(v.status).toBe("tag_behind"); + }); + + test("release base ahead of the running code → ok (not flagged)", () => { + const v = classifyDbGrounding({ + ...base, + codeVersion: "0.11.0-rc.0", + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "v0.12.0-rc.1", + sourceCommit: "abc123", + builtAt: "2026-07-20T00:00:00.000Z", + }); + expect(v.status).toBe("ok"); + }); + + test("partial provenance (missing built_at) fails closed as provenance_incomplete", () => { + // A real CI release stamps all four; release_tag + source_commit but no + // built_at is a malformed artifact and must not reach ok. + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "v0.11.0-rc.102", + sourceCommit: "abc123", + builtAt: null, + }); + expect(v.status).toBe("provenance_incomplete"); + expect(v.ok).toBe(false); + }); + + test("release_tag present but schema_version stamp missing → provenance_incomplete", () => { + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, + metaSchema: null, // no db_meta.schema_version stamp + releaseTag: "v0.11.0-rc.102", + sourceCommit: "abc123", + builtAt: "2026-07-17T00:00:00.000Z", + }); + expect(v.status).toBe("provenance_incomplete"); + expect(v.ok).toBe(false); + }); + + test("unparseable release_tag is never classified ok — freshness is unverified", () => { + const v = classifyDbGrounding({ + ...base, + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "nightly", // no MAJOR.MINOR.PATCH → compareBaseVersion returns null + sourceCommit: "abc123", + builtAt: "2026-07-17T00:00:00.000Z", + }); + expect(v.status).toBe("provenance_incomplete"); + expect(v.ok).toBe(false); + }); + + test("unknown code version is never classified ok", () => { + const v = classifyDbGrounding({ + ...base, + codeVersion: "unknown", + pragmaSchema: 10, + metaSchema: 10, + releaseTag: "v0.11.0-rc.102", + sourceCommit: "abc123", + builtAt: "2026-07-17T00:00:00.000Z", + }); + expect(v.ok).toBe(false); + }); + + test("internal_inconsistent outranks tag_behind", () => { + const v = classifyDbGrounding({ + ...base, + codeVersion: "0.12.0-rc.0", + pragmaSchema: 10, + metaSchema: 5, + releaseTag: "v0.10.0", + sourceCommit: "65fc229", + builtAt: "2026-07-08T00:00:00.000Z", + }); + expect(v.status).toBe("internal_inconsistent"); + }); +}); diff --git a/src/paths.ts b/src/paths.ts index bc20d9a..5be9336 100644 --- a/src/paths.ts +++ b/src/paths.ts @@ -80,6 +80,12 @@ export function detectMode(srcDir: string): InvocationMode { return "package"; } +/** True when the invocation is a git checkout (dev mode). Keeps the "dev" mode + * policy comparison in one place so callers don't hard-code the literal. */ +export function isDevInvocation(mode: InvocationMode): boolean { + return mode === "dev"; +} + /** * Schema version for ros-help.db. * Increment when making destructive schema changes (DROP/RENAME table or column). @@ -133,3 +139,153 @@ export function resolveVersion(srcDir: string): string { return "unknown"; } } + +/** + * Parse the MAJOR.MINOR.PATCH base of a version string, ignoring a `v` prefix + * and any prerelease suffix (`-rc.N`, `-beta.N`, `beta2`, …). Returns null when + * no numeric base is recoverable ("unknown"). + * + * We deliberately drop the prerelease counter: a dev checkout's package.json + * routinely reads `0.11.0-rc.0` while the *published* DB it should ground on is + * `v0.11.0-rc.102`. Comparing rc counters would flag the correct DB as "ahead" + * on every session. Only the base triple is a meaningful staleness axis. + */ +function parseBaseVersion(v: string): [number, number, number] | null { + const m = v.trim().replace(/^v/, "").match(/^(\d+)\.(\d+)\.(\d+)/); + if (!m) return null; + return [Number(m[1]), Number(m[2]), Number(m[3])]; +} + +/** -1 / 0 / 1 comparing the base triples of two version strings. null if either is unparseable. */ +function compareBaseVersion(a: string, b: string): number | null { + const pa = parseBaseVersion(a); + const pb = parseBaseVersion(b); + if (!pa || !pb) return null; + for (let i = 0; i < 3; i++) { + if (pa[i] !== pb[i]) return pa[i] < pb[i] ? -1 : 1; + } + return 0; +} + +export type DbGroundingStatus = + | "ok" + | "schema_mismatch" + | "internal_inconsistent" + | "provenance_incomplete" + | "tag_behind" + | "unstamped"; + +export type DbGroundingVerdict = { + status: DbGroundingStatus; + detail: string; + /** True only when status === "ok". Convenience for exit codes / boolean gates. */ + ok: boolean; +}; + +/** + * Classify whether a resolved DB can be trusted to ground claims about the code + * that is querying it. Pure — no DB access — so it is shared unchanged by + * getDbStats (open connection), MCP startup, and the db-doctor CLI (probe). + * + * Precedence (first match wins): + * 1. schema_mismatch — PRAGMA user_version ≠ code SCHEMA_VERSION. The + * on-disk shape is unqueryable by this build; the hard case already forced + * a redownload in checkDbFreshness. Terminal signal. + * 2. internal_inconsistent — db_meta.schema_version ≠ PRAGMA user_version. The + * dead giveaway of a DB whose pragma was bumped in place by initDb() over a + * stale corpus (the #94 "Frankenstein"): stamped provenance no longer + * describes the bytes. + * 3. unstamped — neither release_tag nor source_commit. A local + * `make extract` working build: fine for extraction, not for grounding + * claims about shipped data. + * 4. provenance_incomplete — claims release identity (has release_tag OR + * source_commit) but is missing one of the four stamps a CI release always + * writes (release_tag, source_commit, built_at, schema_version). Fail + * closed: partial provenance can't be trusted as a grounding source. + * 5. tag_behind — release_tag's MAJOR.MINOR.PATCH is behind the + * running code's. Content predates this checkout. + * 6. ok — all four stamps present, schema coherent, tag + * current (schema/release-compatible with this build — not proof the DB was + * built from this exact commit; a release DB is built from an ancestor). + */ +export function classifyDbGrounding(input: { + pragmaSchema: number; + metaSchema: number | null; + releaseTag: string | null; + builtAt: string | null; + sourceCommit: string | null; + codeSchema: number; + codeVersion: string; + mode: InvocationMode; +}): DbGroundingVerdict { + const verdict = (status: DbGroundingStatus, detail: string): DbGroundingVerdict => ({ + status, + detail, + ok: status === "ok", + }); + + if (input.pragmaSchema !== input.codeSchema) { + return verdict( + "schema_mismatch", + `DB schema v${input.pragmaSchema} ≠ code schema v${input.codeSchema}; this build cannot query the on-disk shape.`, + ); + } + + if (input.metaSchema !== null && input.metaSchema !== input.pragmaSchema) { + return verdict( + "internal_inconsistent", + `db_meta.schema_version=${input.metaSchema} disagrees with PRAGMA user_version=${input.pragmaSchema} — the DB was bumped in place over a stale corpus; its provenance no longer describes its contents.`, + ); + } + + const hasReleaseIdentity = input.releaseTag !== null || input.sourceCommit !== null; + if (!hasReleaseIdentity) { + return verdict( + "unstamped", + "No release_tag/source_commit — a local extraction build, not a CI release. Fine for extraction work; not a grounding source for claims about shipped data.", + ); + } + + // Claims release identity — a real CI release stamps all four. Missing any is a + // malformed/partial artifact: fail closed rather than trust it (built_at is + // required here, so it is not merely informational). + if ( + input.releaseTag === null || + input.sourceCommit === null || + input.builtAt === null || + input.metaSchema === null + ) { + const missing = [ + input.releaseTag === null ? "release_tag" : null, + input.sourceCommit === null ? "source_commit" : null, + input.builtAt === null ? "built_at" : null, + input.metaSchema === null ? "schema_version" : null, + ].filter(Boolean).join(", "); + return verdict( + "provenance_incomplete", + `Partial db_meta provenance (missing ${missing}); a CI release stamps all four, so this DB's origin cannot be trusted for grounding.`, + ); + } + + const cmp = compareBaseVersion(input.releaseTag, `v${input.codeVersion}`); + if (cmp === null) { + // All four stamps are present, but the release_tag (or code version) has no + // parseable MAJOR.MINOR.PATCH, so freshness cannot be verified. Never fall + // through to "ok" — an unverifiable version is not a grounded one. + return verdict( + "provenance_incomplete", + `Cannot compare DB release ${input.releaseTag} with running code v${input.codeVersion}; version is unparseable, so freshness is unverified.`, + ); + } + if (cmp < 0) { + return verdict( + "tag_behind", + `DB release ${input.releaseTag} is behind the running code (v${input.codeVersion}); its content predates this checkout.`, + ); + } + + return verdict( + "ok", + `DB release ${input.releaseTag} (schema v${input.pragmaSchema}) is coherent with this build.`, + ); +} diff --git a/src/query.test.ts b/src/query.test.ts index 80554a5..5a75286 100644 --- a/src/query.test.ts +++ b/src/query.test.ts @@ -2157,6 +2157,23 @@ describe("getDbStats", () => { expect(stats).toHaveProperty("skills"); expect(typeof stats.skills).toBe("number"); }); + + test("exposes a provenance block with a grounding verdict (#94)", () => { + // A sibling test writes db_meta on the shared in-memory DB; clear it so this + // test controls its own provenance state (verdict logic itself is unit-tested + // in paths.test.ts — here we only assert getDbStats wires the block through). + db.run("DELETE FROM db_meta"); + const stats = getDbStats(); + expect(stats).toHaveProperty("provenance"); + const p = stats.provenance; + expect(p.schema_version_pragma).toBe(SCHEMA_VERSION); + expect(p.code_schema_version).toBe(SCHEMA_VERSION); + expect(p.grounding).toHaveProperty("status"); + expect(typeof p.grounding.ok).toBe("boolean"); + // No release_tag/source_commit stamped → a genuinely unstamped local build. + expect(p.grounding.status).toBe("unstamped"); + expect(p.grounding.ok).toBe(false); + }); }); // --------------------------------------------------------------------------- diff --git a/src/release.test.ts b/src/release.test.ts index dd9e25c..ece3264 100644 --- a/src/release.test.ts +++ b/src/release.test.ts @@ -103,6 +103,41 @@ describe("bin/rosetta.js", () => { // Build constants declarations // --------------------------------------------------------------------------- +// --------------------------------------------------------------------------- +// db-sync / db-doctor — local grounding tooling (#94) +// --------------------------------------------------------------------------- + +describe("db grounding tooling", () => { + test("package.json wires db:doctor and db:sync to the scripts (db:sync is NOT --refresh)", () => { + const pkg = JSON.parse(readText("package.json")); + expect(pkg.scripts["db:doctor"]).toBe("bun run scripts/db-doctor.ts"); + expect(pkg.scripts["db:sync"]).toBe("bun run scripts/db-sync.ts"); + // --refresh pins to the placeholder package version and falls back to the + // newest *stable* (older schema) — the exact bug db-sync exists to avoid. + expect(pkg.scripts["db:sync"]).not.toContain("--refresh"); + }); + + test("db-sync discovers releases (prereleases included) via gh and passes the ordered candidate list to downloadDb", () => { + const src = readText("scripts/db-sync.ts"); + expect(src).toContain("gh api"); + expect(src).toContain("ros-help.db.gz"); + // Must hand downloadDb the WHOLE ordered candidate list (not just the newest + // tag) so it can walk down past a higher-schema release to the newest one + // that matches this checkout — finding #2 in the Codex review of #113. + expect(src).toContain("tags.map"); + expect(src).toContain("downloadDb(dbPath, console.log, urls)"); + }); + + test("Makefile db-sync/db-doctor targets are declared .PHONY and call the scripts", () => { + const mk = readText("Makefile"); + const phony = getPhonyBlock(mk); + expect(phony).toContain("db-doctor"); + expect(phony).toContain("db-sync"); + expect(mk).toContain("bun run scripts/db-doctor.ts"); + expect(mk).toContain("bun run scripts/db-sync.ts"); + }); +}); + describe("build-time constants", () => { test("mcp.ts imports resolveVersion from paths.ts", () => { const src = readText("src/mcp.ts"); diff --git a/src/setup.test.ts b/src/setup.test.ts index c2e3191..b3b9754 100644 --- a/src/setup.test.ts +++ b/src/setup.test.ts @@ -254,11 +254,20 @@ describe("probeDb", () => { describe("checkDbFreshness", () => { const opts = { schemaVersion: 8, runningVersion: "0.11.0-beta.93", mode: "package" as const }; + // checkDbFreshness only reads schemaVersion + releaseTag; the extra db_meta + // provenance fields (added for the #94 grounding verdict) default to null here. + const probe = (schemaVersion: number, releaseTag: string | null) => ({ + schemaVersion, + pages: 500, + commands: 5000, + releaseTag, + metaSchemaVersion: releaseTag === null ? null : schemaVersion, + sourceCommit: releaseTag === null ? null : "abc123", + builtAt: releaseTag === null ? null : "2026-07-01T00:00:00.000Z", + }); + test("schema mismatch redownloads and is fatal on failure, regardless of release tag", () => { - const result = checkDbFreshness( - { schemaVersion: 7, pages: 500, commands: 5000, releaseTag: "v0.11.0-beta.93" }, - opts, - ); + const result = checkDbFreshness(probe(7, "v0.11.0-beta.93"), opts); expect(result.redownload).toBe(true); expect(result.hardFailOnDownloadError).toBe(true); expect(result.reason).toContain("schema mismatch"); @@ -266,51 +275,36 @@ describe("checkDbFreshness", () => { test("same-schema release-tag mismatch redownloads but is non-fatal on failure", () => { // Package-mode user cached beta.92's DB; bunx just resolved beta.93. - const result = checkDbFreshness( - { schemaVersion: 8, pages: 500, commands: 5000, releaseTag: "v0.11.0-beta.92" }, - opts, - ); + const result = checkDbFreshness(probe(8, "v0.11.0-beta.92"), opts); expect(result.redownload).toBe(true); expect(result.hardFailOnDownloadError).toBe(false); expect(result.reason).toContain("stale"); }); test("matching schema and release tag needs no redownload", () => { - const result = checkDbFreshness( - { schemaVersion: 8, pages: 500, commands: 5000, releaseTag: "v0.11.0-beta.93" }, - opts, - ); + const result = checkDbFreshness(probe(8, "v0.11.0-beta.93"), opts); expect(result.redownload).toBe(false); }); test("dev mode never triggers on release-tag drift — a dev DB isn't tied to a published release", () => { - const result = checkDbFreshness( - { schemaVersion: 8, pages: 500, commands: 5000, releaseTag: "v0.10.0" }, - { ...opts, mode: "dev" }, - ); + const result = checkDbFreshness(probe(8, "v0.10.0"), { ...opts, mode: "dev" }); expect(result.redownload).toBe(false); }); test("dev mode still redownloads on a genuine schema mismatch", () => { - const result = checkDbFreshness( - { schemaVersion: 7, pages: 500, commands: 5000, releaseTag: "v0.10.0" }, - { ...opts, mode: "dev" }, - ); + const result = checkDbFreshness(probe(7, "v0.10.0"), { ...opts, mode: "dev" }); expect(result.redownload).toBe(true); expect(result.hardFailOnDownloadError).toBe(true); }); test("compiled mode is treated like package mode for release-tag drift", () => { - const result = checkDbFreshness( - { schemaVersion: 8, pages: 500, commands: 5000, releaseTag: "v0.11.0-beta.92" }, - { ...opts, mode: "compiled" }, - ); + const result = checkDbFreshness(probe(8, "v0.11.0-beta.92"), { ...opts, mode: "compiled" }); expect(result.redownload).toBe(true); expect(result.hardFailOnDownloadError).toBe(false); }); test("null releaseTag (pre-v5 DB) never triggers a release-tag mismatch by itself", () => { - const result = checkDbFreshness({ schemaVersion: 8, pages: 500, commands: 5000, releaseTag: null }, opts); + const result = checkDbFreshness(probe(8, null), opts); expect(result.redownload).toBe(false); }); }); diff --git a/src/setup.ts b/src/setup.ts index 3da2c89..2f8e49f 100644 --- a/src/setup.ts +++ b/src/setup.ts @@ -47,6 +47,11 @@ type DbProbe = { pages: number; commands: number; releaseTag: string | null; + /** db_meta.schema_version — the schema the DB *claims*, which can disagree + * with the PRAGMA user_version above when a corpus was bumped in place. */ + metaSchemaVersion: number | null; + sourceCommit: string | null; + builtAt: string | null; }; export type DbFreshnessCheck = { @@ -158,12 +163,7 @@ function looksLikeSqliteFile(dbPath: string): boolean { * fail to open readonly on macOS until a read-write connection initialises the * WAL shared-memory file. probeDb always operates on a temp or new file so * read-write access is safe. */ -export function probeDb(dbPath: string): { - schemaVersion: number; - pages: number; - commands: number; - releaseTag: string | null; -} | null { +export function probeDb(dbPath: string): DbProbe | null { if (!looksLikeSqliteFile(dbPath)) return null; let check: SQLiteDatabase | null = null; @@ -174,17 +174,28 @@ export function probeDb(dbPath: string): { const pages = sqliteGet<{ c: number }>(check, "SELECT COUNT(*) AS c FROM pages"); const cmds = sqliteGet<{ c: number }>(check, "SELECT COUNT(*) AS c FROM commands"); let releaseTag: string | null = null; + let metaSchemaVersion: number | null = null; + let sourceCommit: string | null = null; + let builtAt: string | null = null; try { - const meta = sqliteGet<{ value: string } | null>(check, "SELECT value FROM db_meta WHERE key = 'release_tag'"); - releaseTag = meta?.value ?? null; + const readMeta = (key: string) => + sqliteGet<{ value: string } | null>(check as SQLiteDatabase, "SELECT value FROM db_meta WHERE key = ?", key)?.value ?? null; + releaseTag = readMeta("release_tag"); + sourceCommit = readMeta("source_commit"); + builtAt = readMeta("built_at"); + const ms = readMeta("schema_version"); + metaSchemaVersion = ms === null ? null : Number(ms); } catch { - // db_meta missing — pre-v5 schema, leave releaseTag null + // db_meta missing — pre-v5 schema, leave provenance fields null } return { schemaVersion: ver.user_version, pages: pages.c, commands: cmds.c, releaseTag, + metaSchemaVersion, + sourceCommit, + builtAt, }; } catch { return null; @@ -199,10 +210,10 @@ export function probeDb(dbPath: string): { } } -function sqliteGet(db: SQLiteDatabase, sql: string): T { +function sqliteGet(db: SQLiteDatabase, sql: string, ...params: unknown[]): T { const stmt = db.prepare(sql); try { - return stmt.get() as T; + return stmt.get(...params) as T; } finally { stmt.finalize(); } @@ -411,23 +422,35 @@ export function dbDownloadUrls(version: string): string[] { export async function downloadDb( dbPath: string, log: (msg: string) => void = console.log, + urlsOverride?: string[], ): Promise { + // An explicit URL override (db-sync) is a request to install a *specific* + // release, so the lock-contention shortcut must NOT silently reuse whatever + // schema-compatible DB happens to be on disk — that would report success while + // retaining an older/unrelated release. In that case we still wait for the + // other process to release the lock (via waitForUsableDb, which returns once + // the lock file disappears), but then re-acquire the lock and download the + // requested release below rather than returning the reused DB. + const forceDownload = !!(urlsOverride && urlsOverride.length > 0); + let lock = tryAcquireDownloadLock(dbPath); if (!lock) { const reused = await waitForUsableDb(dbPath, log); - if (reused) { - const probe = probeDb(dbPath); - if (probe) { - log(` Reused existing database: ${formatProbeSummary(probe)}`); - return probe; + if (!forceDownload) { + if (reused) { + const probe = probeDb(dbPath); + if (probe) { + log(` Reused existing database: ${formatProbeSummary(probe)}`); + return probe; + } } - } - // Re-probe once — the lock may have been released with a healthy DB - const fallbackProbe = probeDb(dbPath); - if (isUsableDbProbe(fallbackProbe)) { - log(` Reused existing database: ${formatProbeSummary(fallbackProbe)}`); - return fallbackProbe; + // Re-probe once — the lock may have been released with a healthy DB + const fallbackProbe = probeDb(dbPath); + if (isUsableDbProbe(fallbackProbe)) { + log(` Reused existing database: ${formatProbeSummary(fallbackProbe)}`); + return fallbackProbe; + } } lock = tryAcquireDownloadLock(dbPath); @@ -439,7 +462,11 @@ export async function downloadDb( } } - const urls = dbDownloadUrls(RELEASE_VERSION); + // In dev the package.json version is a CI-rewritten placeholder with no + // matching release, so callers (db-sync) can pass explicit candidate URLs — + // e.g. the newest prerelease that actually ships the DB — instead of the + // version-pinned + /latest/ pair, which resolves to the newest *stable*. + const urls = urlsOverride && urlsOverride.length > 0 ? urlsOverride : dbDownloadUrls(RELEASE_VERSION); let lastError: Error | null = null; try {