Match video title in search, not just transcript (#89)

CHANGELOG.md

@@ -65,6 +65,7 @@ uses [Semantic Versioning](https://semver.org/).
 
 ### Fixed
 
+- **Video search now matches the video title, not just the transcript — correctly-titled changelog videos no longer get buried** (issue #89). `searchVideos()` queried only `video_segments_fts` (chapter title + transcript), so a query word that lives only in the human-curated title — "changelog" is never spoken in the narration — was unmatchable, and a video like *"RouterOS 7.23 changelog — VRF offloading, …"* ranked #15 on `changelog 7.23 routeros`, below the display cutoff. It now also consults the already-trigger-maintained `videos_fts` (title + description; **no schema change**), ranking strong-before-weak: title-AND, then transcript-AND, then the OR fallbacks in the same order, deduped by video. On the live corpus that target video moves from #15 to **#1**, and `changelog 7.NN routeros` resolves to the matching release video for each version. Deliberately scoped to the FTS match surface: the shared `extractTerms`/stop-word handling that shreds dotted versions (`7.23` → `7`,`23`) and drops `routeros` is left untouched here — recognizing RouterOS version strings as first-class query tokens across `routeros_search` is tracked as separate, broader work.
 - **The shared version comparator now orders same-base prereleases instead of collapsing them** (issue #107). `compareVersions()` in `src/version-compare.ts` stripped `beta\d*`/`rc\d*` wholesale, so `7.24beta1`, `7.24beta2`, `7.24beta3` all compared **equal** (likewise `7.20rc1` vs `7.20rc2`) — deterministic only because callers (`changelog.tsv`, `command_versions`/`routeros_command_diff` ordering) bolt on a secondary sort key. The comparator now folds the prerelease *number* into the sort so the order is strict and total for one base: `beta1 < beta2 < beta3 < rc1 < 7.24` (a bare `beta`/`rc` with no number sorts below its numbered siblings). Every caller improves at once; the module stays dependency-free so the DB-only `export.ts` boundary is unaffected. The honesty disclosure this bug forced into `rosetta export`'s `changelog.tsv` manifest is removed, and a focused `src/version-compare.test.ts` pins the precedence.
 - **Agent-facing corpus counts no longer carry stale Confluence-era numbers** (issue #88). `src/mcp.ts`'s schema-guide resource and `routeros_search`'s tool-description "Documentation: 317 pages from March 2026 Confluence export" line, plus `MANUAL.md`'s Tables reference, still advertised frozen Confluence figures (`pages` 317, `properties` 4,860, `callouts` 1,034) and drifted command/version counts (`command_versions` 1.67M vs an actual ~2.2M) even though the live Docusaurus extractor has been the default prose source since the `T-0035` cutover. Rather than re-baking exact numbers that drift every extraction, these surfaces now state **order-of-magnitude** counts, correct the Docusaurus provenance wording, and point at `routeros_stats` (already the live single source of truth) for exact figures — so the counts can't go silently wrong again. The `#87` caveat note in `MANUAL.md` is dropped.
 - **165 properties were silently destroyed on every build; `properties` and `callouts` now carry a real `section_id` (issue #90, schema v9).** `properties.section` stored **heading text**, and `UNIQUE(page_id, name, section)` plus `INSERT OR IGNORE` dropped any property whose page repeated a heading — the shipped `v0.11.0-rc.97` DB holds 4,416 of 4,581 parsed properties, and nothing logged the gap (the extractor's own "Properties:" line counted *attempted* inserts, so it reported rows the DB did not contain). `ppp-aaa` alone lost most of its user/profile properties to its four `Properties` headings. A corpus-wide measurement showed the obvious fix is insufficient: keying on `section_id` still destroys 87 rows, because the manual documents one property name several times *within a single section* (`dot1x` defines `interface` twice under `Server` — once for the server table, once for the client one), so section is not an identity at all. **The constraint is therefore removed**, not re-keyed; extractors now assert parsed == stored and fail the build on any gap (`V-extractor-no-silent-drops`). All 4,575 parsed properties are stored, and the 141 distinct properties the constraint had been eating — `name` = "Name used for authentication" and friends — are searchable for the first time. `properties.section_id` and the brand-new `callouts.section_id` (callouts previously had no section attribution whatsoever) resolve for 99.7% and 89% of rows respectively; the remainder are genuinely above any heading and stay NULL rather than being forced into a section. Property→section attribution rises from 72% resolvable to 99.7%. A property under an h4–h6 folds to its enclosing h1–h3 section, so `sections` remains the retrieval unit and `routeros_get_page` is unchanged; `section` keeps the raw h4 text, so the finer heading stays recoverable.

src/query.test.ts

@@ -516,6 +516,35 @@ beforeAll(() => {
     (id, video_id, chapter_title, start_s, end_s, transcript, sort_order)
     VALUES
     (3, 2, NULL, 0, NULL, 'BGP peering and route reflection allow scalable routing in large networks.', 0)`);
+  // Changelog-style video (issue #89): the word "changelog" and the version live only in
+  // the TITLE; the narration never says either, so a transcript-only search buries it.
+  db.run(`INSERT INTO videos
+    (id, video_id, title, description, channel, upload_date, duration_s, url, has_chapters)
+    VALUES
+    (3, 'chg723', 'RouterOS 7.23 changelog - VRF offloading, DoH with HTTP/2, option82 changes and more', 'Release highlights walkthrough', 'MikroTik', '20240301', 500,
+     'https://www.youtube.com/watch?v=chg723', 0)`);
+  db.run(`INSERT INTO video_segments
+    (id, video_id, chapter_title, start_s, end_s, transcript, sort_order)
+    VALUES
+    (4, 3, NULL, 0, NULL, 'In this release VRF offloading gets hardware support and DoH now runs over HTTP two.', 0)`);
+  // Dedup regression pair (issue #89): both videos match "gigabit throughput tuning" only in
+  // the transcript (never the title). video 4 has THREE matching segments; without per-video
+  // dedup its segments consume the SQL limit and hide video 5.
+  db.run(`INSERT INTO videos
+    (id, video_id, title, description, channel, upload_date, duration_s, url, has_chapters)
+    VALUES
+    (4, 'perf-a', 'Lab Session A', 'Benchmarking notes', 'MikroTik', '20240401', 700,
+     'https://www.youtube.com/watch?v=perf-a', 1)`);
+  db.run(`INSERT INTO videos
+    (id, video_id, title, description, channel, upload_date, duration_s, url, has_chapters)
+    VALUES
+    (5, 'perf-b', 'Lab Session B', 'Benchmarking notes', 'MikroTik', '20240402', 300,
+     'https://www.youtube.com/watch?v=perf-b', 0)`);
+  db.run(`INSERT INTO video_segments (id, video_id, chapter_title, start_s, end_s, transcript, sort_order) VALUES
+    (5, 4, 'Part 1', 0, 100, 'We measure gigabit throughput tuning on the test bench.', 0),
+    (6, 4, 'Part 2', 100, 200, 'More gigabit throughput tuning with different packet sizes.', 1),
+    (7, 4, 'Part 3', 200, 300, 'Final gigabit throughput tuning results and analysis.', 2),
+    (8, 5, NULL, 0, NULL, 'A quick look at gigabit throughput tuning for small setups.', 0)`);
 
   // Dude wiki page fixtures for searchDude tests
   db.run(`INSERT INTO dude_pages
@@ -2399,6 +2428,23 @@ describe("searchVideos", () => {
     expect(results[0].chapter_title).toBeNull();
   });
 
+  test("surfaces a changelog-titled video whose transcript never says 'changelog' (issue #89)", () => {
+    // "changelog" and "7.23" appear only in the title; segment-only search buried this at #15.
+    const results = searchVideos("changelog 7.23 routeros");
+    expect(results[0].video_id).toBe("chg723");
+    // Proves the hit came from the title, not the narration.
+    expect(results[0].excerpt.toLowerCase()).toContain("changelog");
+  });
+
+  test("a video's multiple matching segments don't crowd out distinct videos (issue #89 dedup)", () => {
+    // video 4 (perf-a) has 3 matching segments, video 5 (perf-b) has 1 — both transcript-only.
+    const results = searchVideos("gigabit throughput tuning", 2);
+    const ids = results.map((r) => r.video_id);
+    expect(new Set(ids).size).toBe(ids.length); // each video appears at most once
+    expect(ids).toContain("perf-b"); // not hidden behind perf-a's three segments
+    expect(results.length).toBe(2);
+  });
+
   test("returns empty array for empty query", () => {
     expect(searchVideos("")).toEqual([]);
   });

src/query.ts

@@ -2620,36 +2620,111 @@ export type VideoSearchResult = {
   excerpt: string;
 };
 
-/** Search YouTube video transcripts via FTS, joining segment → video metadata. */
+/**
+ * Search YouTube videos, matching both the transcript AND the video title/description
+ * (issue #89). Title-bearing words like "changelog" never appear in the narration, so a
+ * segment-only search buries correctly-titled videos; `videos_fts` (title + description,
+ * already trigger-maintained — no schema change) is consulted alongside `video_segments_fts`.
+ *
+ * Ranking is strong-before-weak: a title-AND hit, then a transcript-AND hit, then the
+ * OR fallbacks in the same order. Title matches lead because the title is the curated
+ * summary of the video, so a query word found only there is a strong relevance signal.
+ * Results dedupe by video (first occurrence wins) and cap at `limit`.
+ */
 export function searchVideos(query: string, limit = 5): VideoSearchResult[] {
   const terms = extractTerms(query);
   if (terms.length === 0) return [];
 
-  let ftsQuery = buildFtsQuery(terms, "AND");
-  if (!ftsQuery) return [];
-  let results = runVideosFtsQuery(ftsQuery, limit);
+  const andQuery = buildFtsQuery(terms, "AND");
+  if (!andQuery) return [];
+  const orQuery = terms.length > 1 ? buildFtsQuery(terms, "OR") : "";
+
+  const merged: VideoSearchResult[] = [];
+  const seen = new Set<string>();
+  const add = (rows: VideoSearchResult[]) => {
+    for (const r of rows) {
+      if (merged.length >= limit) return;
+      if (seen.has(r.video_id)) continue;
+      seen.add(r.video_id);
+      merged.push(r);
+    }
+  };
 
-  // Fallback to OR if AND returns nothing and we have multiple terms
-  if (results.length === 0 && terms.length > 1) {
-    ftsQuery = buildFtsQuery(terms, "OR");
-    results = runVideosFtsQuery(ftsQuery, limit);
-  }
+  // Strong matches: every term present, title before transcript.
+  add(runVideosTitleQuery(andQuery, limit));
+  if (merged.length < limit) add(runVideosFtsQuery(andQuery, limit));
+  // Weak fallback (multi-term only): any term present, same title-before-transcript order.
+  if (merged.length < limit && orQuery) add(runVideosTitleQuery(orQuery, limit));
+  if (merged.length < limit && orQuery) add(runVideosFtsQuery(orQuery, limit));
 
-  return results;
+  return merged;
 }
 
 function runVideosFtsQuery(ftsQuery: string, limit: number): VideoSearchResult[] {
+  if (!ftsQuery) return [];
+  try {
+    // A video with several matching segments would otherwise consume multiple of the
+    // top-`limit` rows and, after the caller dedupes by video, hide eligible videos ranked
+    // just below the cutoff (issue #89 — "firewall filter" yielded 7 videos from 10 rows).
+    // fts5's snippet() can't run alongside a window function, so this is two bounded steps:
+    // (1) pick the best-ranked segment PER VIDEO, then (2) fetch the excerpt for just those.
+    const rows = db
+      .prepare(
+        `SELECT seg_id, video_id, title, url, upload_date, chapter_title, start_s
+         FROM (
+           SELECT video_segments_fts.rowid AS seg_id, v.video_id AS video_id, v.title AS title,
+                  v.url AS url, v.upload_date AS upload_date, vs.chapter_title AS chapter_title,
+                  vs.start_s AS start_s, video_segments_fts.rank AS r,
+                  ROW_NUMBER() OVER (PARTITION BY v.id ORDER BY video_segments_fts.rank) AS rn
+           FROM video_segments_fts
+           JOIN video_segments vs ON vs.id = video_segments_fts.rowid
+           JOIN videos v ON v.id = vs.video_id
+           WHERE video_segments_fts MATCH ?
+         )
+         WHERE rn = 1
+         ORDER BY r
+         LIMIT ?`,
+      )
+      .all(ftsQuery, limit) as Array<VideoSearchResult & { seg_id: number }>;
+    if (rows.length === 0) return [];
+
+    const segIds = rows.map((r) => r.seg_id);
+    const excerpts = db
+      .prepare(
+        `SELECT rowid AS seg_id,
+                snippet(video_segments_fts, 1, '${EXCERPT_MARK_START}', '${EXCERPT_MARK_END}', '...', 25) AS excerpt
+         FROM video_segments_fts
+         WHERE video_segments_fts MATCH ? AND rowid IN (${segIds.map(() => "?").join(",")})`,
+      )
+      .all(ftsQuery, ...segIds) as Array<{ seg_id: number; excerpt: string }>;
+    const byId = new Map(excerpts.map((e) => [e.seg_id, e.excerpt]));
+
+    return rows.map(({ seg_id, ...rest }) => ({ ...rest, excerpt: byId.get(seg_id) ?? "" }));
+  } catch {
+    return [];
+  }
+}
+
+/**
+ * Match the video title/description via `videos_fts` (issue #89). A title-only word like
+ * "changelog" is unsearchable through the transcript index, so this surfaces it. The row
+ * shape stays identical to a transcript hit by attaching the video's first segment for
+ * `chapter_title`/`start_s` (LEFT JOIN — a video with no transcript still returns, with
+ * NULL chapter and start 0); the excerpt is a snippet of the matched title/description.
+ */
+function runVideosTitleQuery(ftsQuery: string, limit: number): VideoSearchResult[] {
   if (!ftsQuery) return [];
   try {
     return db
       .prepare(
         `SELECT v.video_id, v.title, v.url, v.upload_date,
-                vs.chapter_title, vs.start_s,
-                snippet(video_segments_fts, 1, '${EXCERPT_MARK_START}', '${EXCERPT_MARK_END}', '...', 25) as excerpt
-         FROM video_segments_fts fts
-         JOIN video_segments vs ON vs.id = fts.rowid
-         JOIN videos v ON v.id = vs.video_id
-         WHERE video_segments_fts MATCH ?
+                vs.chapter_title, COALESCE(vs.start_s, 0) AS start_s,
+                snippet(videos_fts, -1, '${EXCERPT_MARK_START}', '${EXCERPT_MARK_END}', '...', 25) AS excerpt
+         FROM videos_fts fts
+         JOIN videos v ON v.id = fts.rowid
+         LEFT JOIN video_segments vs
+           ON vs.id = (SELECT id FROM video_segments WHERE video_id = v.id ORDER BY sort_order LIMIT 1)
+         WHERE videos_fts MATCH ?
          ORDER BY rank
          LIMIT ?`,
       )