#53: retrieval-quality gate hardening — golden set 24→35 + direct-surface eval matrix

.github/workflows/copilot-gate.yml

@@ -0,0 +1,36 @@
+name: Copilot Gate
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review, review_requested, review_request_removed]
+  pull_request_review:
+    types: [submitted, dismissed]
+
+# Minimal token scope: the gate only reads PR requested-reviewer state via `gh api`.
+permissions:
+  contents: read
+  pull-requests: read
+
+# One gate run per PR: cancel an in-flight run when a newer review event lands so a
+# stale result can't outlive the reviewer-state change that superseded it.
+concurrency:
+  group: copilot-gate-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  check-pending:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check Copilot Review Status
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          echo "Checking if Copilot is a pending requested reviewer on PR #$PR_NUMBER..."
+          PENDING=$(gh api "repos/$REPO/pulls/$PR_NUMBER" --jq '.requested_reviewers[].login')
+          if echo "$PENDING" | grep -qxF 'copilot-pull-request-reviewer'; then
+             echo "::error::Copilot review is currently pending! Blocking merge."
+             exit 1
+          fi
+          echo "Copilot is not a pending reviewer. Gate is green."

CHANGELOG.md

@@ -24,6 +24,7 @@ uses [Semantic Versioning](https://semver.org/).
 
 ### Added
 
+- **Retrieval-quality net hardened for the 0.11 corpus swap — golden set 24 → 35, plus a direct-surface eval matrix (issue #53).** The Phase 0 golden set (`fixtures/eval/queries.json`) grows by 11 hand-verified queries: +2 natural-language (`pppoe`, `dns`), +4 property, +3 changelog (fixed-CVE / feature-introduced / version-scoped), +2 video — closing the zero-coverage gap on property/changelog/video retrieval that the Confluence→Docusaurus swap left untested. `src/eval/retrieval.ts` gains a per-query `surface` field: `search` (default) still drives the durable `routeros_search` recall/MRR/classifier gate (the entry point most exposed to the swap), while `property`/`changelog`/`video` dispatch to the dedicated tools (`lookupProperty`/`searchChangelogs`/`searchVideos`) and feed a **separate, informational `surface_matrix`** (per-surface hit@5) kept out of the gated aggregates so those stay sensitive — a starting coverage board to grow over time. Golden `_thresholds` recomputed for the larger set (0.88/0.82/0.75/0.90; the old 0.85/0.70/0.65 had drifted loose over 27 search queries — the 2pp regression check vs `baseline.json` remains the tight forward guard). Both `baseline.json` and the **stale April-2026 Confluence-era `self-supervised-baseline.json`** regenerated against the live Docusaurus corpus. Grounded audit (0.10.0 vs current compare-and-contrast via the new one-off `src/eval/corpus-compare.ts`, blocking self-eval rehearsal, and an ETL Bug Ledger) recorded in `briefings/B-0020-0.11-retrieval-quality-audit.md`: **no net retrieval regression from the corpus swap** (14/16 topic-hit on both corpora; existing golden set 91.7% Recall@5 on Docusaurus), and #26's two "regressions" confirmed to miss on 0.10.0 too (pre-existing query-core weakness, not swap-induced). Queries that legitimately fail today stay pointed at the correct answer and fail loudly — no fixture was loosened to green the board.
 - **Hardware overlay Phase 2A — `hardware_catalog`/`device_aliases` now drive device lookup and enrichment (MCP + TUI).** `routeros_device_lookup` and `routeros_search` resolve devices from free-form input far more accurately: an **alias stage** (exact-normalized, authoritative — never fuzzy) maps product codes, old names, and www/hardware slugs the matrix name misses to the canonical device (`cap_ac`→cAP ac, `RB750Gr3`→hEX), reported as `mode: "alias"` with the `matched_alias`. A single matrix-linked result now carries a compact **`hardware` overlay block** — `rosetta_device_id` (stable, persistable key), `category`, `discontinued`, `also_known_as` (name-like aliases only; slug/table artifacts excluded), genuine `non_default_ips`, and reconstructed `product_page_url` (mikrotik.com/product) + `hardware_page_url` — whose `note` steers agents to the device tool rather than fetching a page. Non-matrix entities (accessories, series, legacy/EOL) surface in a new `catalog` array as labeled thin rows with a `kind` field (`accessory`/`series`/`discontinued`/`device`), so an agent never mistakes a GPeR for a router. The classifier gains catalog-prominent families (Chateau, Audience, OmniTik, PowerBox, KNOT, Cube, DISC) plus a whole-input alias probe in `searchAll` for bare codes the regexes miss; `routeros_search`'s `related.devices` now carries `category`/`discontinued`; the browse TUI device card shows the overlay and a thin-row variant for catalog entities, and its stats screen lists the two overlay tables. No `/hardware` prose enters the FTS corpus (fields only). Reads the clean data from the #47/#48 cleanup. See `briefings/B-0019-hardware-overlay-phase2-mcp-tui-surfacing.md` and issue #49.
 - **New Docusaurus `/docs` prose extractor (`extract-docusaurus.ts`) replaces `extract-html.ts` as the default prose source.** Discovers pages via `sitemap.xml`, fetches raw Markdown from manual.mikrotik.com, and populates `pages`/`sections`/`properties`/`callouts` — 360 in-scope `/docs` pages as of 2026-07-07 (CLI Reference and `/hardware` remain out of scope, tracked as follow-up work). `make extract`/`make extract-full` now run it by default; the legacy Confluence pipeline survives as `make extract-legacy-confluence` for rebuilding historical release DBs.
 - **New `pages.rosetta_id` column** (schema v6) gives Docusaurus-sourced pages a stable, URL-derived identifier alongside the existing integer `id` — see `DESIGN.md` and `briefings/B-0012-docusaurus-manual-migration.md` "H7 — Identity / rosetta-id design".

VALIDATION.md

@@ -22,8 +22,8 @@ Tasks in `tasks/` reference these IDs in their `validation:` frontmatter. When a
 | V-tool-registry          | 14-tool MCP surface is frozen — adds, removals, renames are intentional                                              | `EXPECTED_TOOLS` block in `src/mcp-contract.test.ts` (Block A)         | blocking (test.yml)                          | —          |
 | V-tool-shapes            | Tool response shapes stable for 5 canonical queries                                                                  | `src/mcp-contract.test.ts` Block C, `qa.yml` contract gate (run by `release.yml`'s `qa` job against the artifact DB) | blocking                | —          |
 | V-tool-budget            | Token-budget guardrails on 10 canonical queries                                                                      | `src/mcp-contract.test.ts` Block B, `qa.yml` contract gate (run by `release.yml`'s `qa` job) | blocking               | —          |
-| V-retrieval-floor        | Recall@10 ≥ baseline − 2pp on 20 hand-curated golden queries; classifier accuracy stable                             | `src/eval/retrieval.ts`, `qa.yml` eval-golden gate (run by `release.yml`'s `qa` job)          | blocking               | —          |
-| V-retrieval-self         | Self-supervised eval (~170 auto-generated queries) holds within 5pp of baseline                                      | `qa.yml` eval-self gate (run by `release.yml`'s `qa` job; `continue-on-error` unless `eval_self_blocking`) | non-blocking (pending promotion) | —     |
+| V-retrieval-floor        | Recall/MRR/classifier meet absolute floors AND stay within 2pp of baseline on 35 hand-curated golden queries (27 search-surface, gated — the corpus-swap-sensitive `routeros_search` entry point); a separate **informational** `surface_matrix` tracks per-surface hit@5 for the dedicated tools (property/changelog/video), kept OUT of the gated aggregates (#53) | `src/eval/retrieval.ts`, `qa.yml` eval-golden gate (run by `release.yml`'s `qa` job)          | blocking (search aggregates) / non-blocking (surface_matrix) | —          |
+| V-retrieval-self         | Self-supervised eval (~200 auto-generated queries) holds within 5pp of baseline (baseline regenerated against the Docusaurus corpus in #53 — the April-2026 Confluence-era baseline was stale, so every post-swap run showed a false regression) | `qa.yml` eval-self gate (run by `release.yml`'s `qa` job; `continue-on-error` unless `eval_self_blocking`) | non-blocking (pending promotion) | —     |
 | V-db-min-content         | Built DB has ≥200 pages, ≥1000 commands, ≥100 devices, ≥1000 properties, ≥200 hardware_catalog, ≥600 device_aliases before publish | `qa.yml` db-content gate (single definition of the floors; run by `release.yml`'s `qa` job with `db_source=artifact`, so it floors the exact DB about to publish) | blocking | —          |
 | V-db-wipe-guard          | Tests cannot accidentally open and overwrite the on-disk DB                                                          | `src/query.test.ts` `:memory:` guard + post-extraction guard in CI     | blocking                                     | —          |
 | V-db-meta                | Released DB carries `db_meta` provenance (release_tag, built_at, source_commit, schema_version)                      | `scripts/stamp-db-meta.ts` in `release.yml`'s `build` job; presence re-checked by `qa.yml`'s db-meta gate | blocking          | —          |
@@ -70,6 +70,7 @@ When promoting non-blocking → blocking, remove `continue-on-error: true` from
 `V-retrieval-self` is non-blocking because its first landing suppressed a real −10pp regression (`continue-on-error` on the self-supervised eval; see `briefings/B-0014-ci-testing-qa-cleanup.md`). Since Phase B (#42) the eval lives only in `qa.yml`, which the release calls, so there is a single knob to flip. Promotion criteria, proposed here so the row has an owner and a bar rather than drifting forever:
 
 1. **Rehearse blocking on demand first.** `qa.yml`'s `eval_self_blocking: true` input runs the self-supervised eval as a hard gate against a fresh `local-build` DB without touching the release path — use it to observe real build-to-build drift.
+   - **#53 audit run (2026-07-12):** the first blocking rehearsal FAILED, but only on regression-vs-baseline — every *absolute* strategy floor (title/section/property/cmd-path) passed on the current Docusaurus corpus. Root cause was a **stale April-2026 Confluence-era baseline** (never regenerated after the T-0035/T-0036 swap), not a real regression; top misses were generic ambiguous section headings ("Example #1", "Configuration example"), a known low-signal artifact of the section strategy — no golden-set row warranted. Baseline regenerated against Docusaurus; blocking rehearsal now green. This is rehearsal #1 of the ≥3 needed below. See `briefings/B-0020-0.11-retrieval-quality-audit.md`.
 2. **Bar:** at least 3 consecutive `eval_self_blocking` rehearsals (or release runs) green with no legitimate-drift red, i.e. every red in that window was a genuine retrieval regression that got fixed, not tolerated.
 3. **Owner:** the maintainer flips it, in one PR that removes `continue-on-error` from `qa.yml`'s eval-self default (a single definition now — the release inherits it since `release.yml`'s `qa` job doesn't override `eval_self_blocking`).
 4. **Prerequisite, not yet built:** trend persistence. Today each run's numbers live only in that run's `$GITHUB_STEP_SUMMARY`; a durable baseline/trend store (cf. centrs' `qa-history` branch pattern) should land before blocking, so a promotion decision reads history instead of one run.

briefings/B-0020-0.11-retrieval-quality-audit.md

@@ -0,0 +1,90 @@
+# B-0020 — 0.11 retrieval-quality audit (golden-set expansion, 0.10.0 compare, self-eval, Bug Ledger)
+
+**Status:** research + decision record for issue #53. Grounded against real DBs (see below).
+**Date:** 2026-07-12. **Sibling:** #26 (ranking), #28 (umbrella "did we regress from HTML"), B-0012 (migration), B-0010 (eval phases).
+
+## What was measured, and against what
+
+Three DBs, all real (no assumptions that main == 0.10.0):
+
+| DB | corpus | how obtained |
+|----|--------|--------------|
+| current | Docusaurus `/docs` (363 pages, 4402 props, 1311 changelogs, 1953 video segments) | fresh local `make extract` off live manual.mikrotik.com, 2026-07-12 |
+| v0.10.0 | Confluence HTML (321 pages, no `rosetta_id`) | published `v0.10.0` release DB, clean |
+| v0.10.0-compat | v0.10.0 + empty schema-v8 tables | `db.ts` idempotent init on a copy (pages untouched) — only so current `searchAll` can run against it (see BL-6) |
+
+## Finding 1 — No net regression from the Confluence→Docusaurus swap
+
+`src/eval/corpus-compare.ts` (one-off, not a CI gate) runs the golden search queries against
+both corpora and matches on a **corpus-agnostic topic token** (the golden set keys on
+`rosetta_id`, which v0.10.0 lacks entirely, so page-recall can't be compared directly).
+
+**Result: 14/16 (88%) topic-hit@5 on BOTH corpora. Zero topic-level regressions.** The existing
+24-query golden set also scores Recall@5 **91.7%** / MRR 86.8% / classifier 100% on the current
+corpus — at or above the pre-swap baseline. The corpus swap did not degrade what `routeros_search`
+finds for bread-and-butter questions.
+
+## Finding 2 — #26's two "regressions" fail on 0.10.0 too (premise correction)
+
+`nl-firewall-filter` ("block traffic from a specific IP address") and `nl-bgp` ("establish a BGP
+peer with my upstream") miss on the **v0.10.0 Confluence corpus** as well:
+
+- 0.10.0 firewall query top-5: ROSE-storage, General Properties, First Time Configuration,
+  Scripting, HotSpot — no Filter page.
+- 0.10.0 BGP query top-5: Routing Protocol Overview, VRF, DHCP — no dedicated BGP page.
+
+So #26 ("ranking regressions **from the Docusaurus corpus swap**") is mis-titled: this is a
+**pre-existing query-core weakness** (AND-only FTS candidate pool + BM25 length bias on
+verbose NL questions), independent of corpus. Still worth fixing in #26; just not swap-induced.
+Posted as a comment on #26.
+
+## Finding 3 — Self-supervised eval: floors pass, baseline was stale
+
+The blocking self-eval rehearsal (`eval_self_blocking=true`, maintainer CI run 29210060933, and
+reproduced locally) FAILED — but only on **regression vs baseline**, while every absolute
+strategy floor passed on the current corpus (title hit@5 96.7%, section hit@10 78.8%, property
+hit@10 80%, cmd-path hit@5 70%, mrr 59.4%). Root cause: `self-supervised-baseline.json` was
+generated **2026-04-22 against the Confluence corpus** and never regenerated after the swap, so
+every post-swap run shows a phantom ~5–7pp "regression." Top misses are generic ambiguous
+headings ("Example #1", "Configuration example", "Read-only properties") — inherent low signal
+of the section strategy, not retrieval bugs; none earn a golden-set row. **Action taken:**
+regenerated the baseline against Docusaurus (200 queries) → blocking rehearsal now green. Counts
+as rehearsal #1 of the ≥3 in the `V-retrieval-self` promotion path (VALIDATION.md).
+
+## Bug Ledger — ETL/MCP issues surfaced (NOT fixed here; per maintainer, surface don't hide)
+
+Discipline: golden fixtures point at the CORRECT answer and fail loudly; none were loosened to
+green the board. Items below are real defects found while authoring the net.
+
+- **BL-1 — command-path→page→property link broken on core menus (has a durable failing anchor).**
+  `lookupProperty(name, path)` returns only low-confidence rows (no HIGH resolution) for
+  `/ip/firewall/filter action` (19 rows, 0 high), `/ip/dhcp-server lease-time`, `/ip/ipsec/peer
+  exchange-mode`, `/routing/bgp/connection distance`. Worse, `/ip/dhcp-server address-pool`
+  resolves HIGH to the **wrong page** (hotspot-captive-portal, not DHCP). The property exists and
+  the command path exists; they just aren't linked. Pinned by golden query
+  `prop-firewall-filter-action` (known-failing). Owning follow-up issue filed.
+- **BL-2 — noisy command→page links.** Some `commands.page_id` links are semantically wrong,
+  e.g. `/ip/service/print` → docs/containers, `/certificate/scep-server/*` → capsman page.
+  Same root area as BL-1. Tracked in the BL-1 follow-up.
+- **BL-3 — dotted WiFi properties not extracted.** `/interface/wifi security.authentication-types`,
+  `configuration.mode`, `channel.width` return **0 property rows**; `/interface/wifi ssid` returns
+  9 rows but 0 high-confidence. The new manual's nested/dotted wifi property structure isn't
+  captured by the Docusaurus property parser. Tracked in the BL-1 follow-up.
+- **BL-4 — classifier false-positive on an NL question.** "port forward to an internal server"
+  is classified as command path `/port/forward/to/an/internal/server`. NL questions with
+  path-shaped words get mis-detected. Candidate for the classifier-improvement backlog.
+- **BL-5 — see Finding 2** (#26 is not swap-induced). Commented on #26.
+- **BL-6 — `searchAll` crashes on a pre-v8 DB.** `SQLiteError: no such table: device_aliases`
+  (query.ts:391, the alias probe added in #51). No graceful degradation when a schema-v8 table
+  is absent. Only bites cross-version DB use (e.g. this audit); noted for robustness.
+
+## What shipped in #53 (the framework)
+
+- Golden set 24 → 35: +2 nl (pppoe, dns), +4 property (2 clean HIGH anchors, 1 free-form search,
+  1 known-failing BL-1 anchor), +3 changelog (CVE / feature / version-scoped), +2 video.
+- `src/eval/retrieval.ts`: `surface` dispatch (property/changelog/video → the dedicated tools) +
+  informational `surface_matrix` (per-surface hit@5), kept out of the gated search aggregates;
+  soft per-query "was passing, now misses" surface-regression WARNING.
+- `_thresholds` recomputed for 27 search queries (0.88/0.82/0.75/0.90) — the pre-#53 0.85/0.70/0.65
+  had drifted loose over the larger set; the 2pp regression check remains the tight forward guard.
+- Both baselines regenerated against the current Docusaurus corpus.