Match video title in search, not just transcript (#89)

[mobileskyfi]

The 0.11.0 low-hanging slice of #89, per the scope split on the issue.

Problem

searchVideos() matched only video_segments_fts (chapter title + transcript). Words that live only in the video title — "changelog" is never spoken in the narration — were unmatchable, so "RouterOS 7.23 changelog — VRF offloading, DoH with HTTP/2, …" ranked #15 on changelog 7.23 routeros, below every display cutoff, even though the video was correctly extracted.

Fix — the title into the FTS match surface (no schema change)

videos_fts (FTS5 over title + description) already exists and is trigger-maintained; searchVideos() just never consulted it. It now queries both indexes and merges strong-before-weak:

1. title-AND (videos_fts), then transcript-AND (video_segments_fts)
2. OR fallbacks in the same title-before-transcript order (multi-term only)

Deduped by video, capped at limit. A title-only hit borrows the video's first segment for chapter_title/start_s (LEFT JOIN, so a transcript-less video still returns) and its excerpt is a snippet of the matched title/description — so the result shape is identical to a transcript hit.

Title-AND leads because the title is the curated summary; a query word found only there is a strong relevance signal. Existing anchors are unaffected: they're multi-term AND queries the short titles don't fully satisfy, so title-AND returns nothing and the prior transcript behavior stands.

Explicitly out of scope (deferred, broader effort)

The shared extractTerms/STOP_WORDS changes — preserving dotted versions (7.23 → one token instead of 7,23) and reconsidering routeros as a stop word — are not touched here. They live on the page-search path too, so they move the golden/eval baseline, and "RouterOS version strings as first-class query tokens across routeros_search" deserves a deliberate design pass rather than a video-search bolt-on. The fix works today regardless: unicode61 already splits the title's 7.23 into 7/23, so title-AND(changelog, 23) matches.

Verification

• Live-DB repro (ros-help.db, rc.102): target 8X0Aewng-BM moves npm prerelease dist-tag channel (alpha/beta/rc/next) + coverage reporting #15 → Add SafeSkill security badge (73/100 — Passes with Notes) #1 on changelog 7.23 routeros; changelog 7.22 routeros → the 7.22 changelog video Add SafeSkill security badge (73/100 — Passes with Notes) #1; vlan bridge still returns VLAN videos first (no obvious regression).
• New anchor test (src/query.test.ts): a changelog-titled fixture whose transcript never says "changelog" now surfaces Add SafeSkill security badge (73/100 — Passes with Notes) #1, with the excerpt proving the hit came from the title.
• Phase 0 retrieval eval: Recall@5 92.9% (Δ +0.3pp), video surface hit@5 100%, changelog 100% — no regression (the 3 flagged queries are the pre-existing Retrieval ranking regressions from Docusaurus corpus swap (nl-firewall-filter, nl-bgp golden queries) #26/Property lookup: command-path→page→property links broken on core menus (BL-1/2/3 from #53) #58 weaknesses).
• bun test 899 pass / 0 fail; typecheck + biome + markdownlint clean.

Note: the CHANGELOG.md ### Fixed bullet will likely conflict trivially with #114 (both add at the top of the same section) — resolve by keeping both bullets when the second merges.

Issue linking

Closes #89 — the video-title match slice is delivered here.

The deferred remainder (dotted-version tokenization + routeros stop-word handling across the shared query path) is split into follow-up #116 per the mandatory partial-landing rule, carrying over #89's un-done acceptance bullets.

Summary by CodeRabbit

• Bug Fixes
  • Video search now matches titles and descriptions as well as transcript content.
  • Improved relevance ranking helps videos with matching titles appear higher in results.
  • Search results now avoid duplicate entries when multiple transcript segments match the same video.
  • Result pagination is more consistent, ensuring distinct matching videos are included.
• Tests
  • Added coverage for title-only matches, relevance ranking, deduplication, and result limits.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6734b045-2771-403d-a2d4-41ff98d31c17

📥 Commits

Reviewing files that changed from the base of the PR and between 4402464 and bad7525.

📒 Files selected for processing (3)

• CHANGELOG.md
• src/query.test.ts
• src/query.ts

📜 Recent review details

🧰 Additional context used

📓 Path-based instructions (7)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use Bun and TypeScript, including Bun-native runtime APIs such as bun:sqlite, Bun.serve, and bunx.

Rosetta uses Bun and TypeScript; prefer bun, bun test, and make verify where applicable, rather than Node/npm-oriented substitutes.

Files:

• src/query.test.ts
• src/query.ts

src/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

Rosetta is read-only documentation/schema context and must not connect to or modify a user's router.

Files:

• src/query.test.ts
• src/query.ts

**/*.{ts,tsx,md}

📄 CodeRabbit inference engine (AGENTS.md)

Keep the attribution boundary visible when community RouterOS skills surface, because they are supplemental rather than official MikroTik documentation.

Files:

• src/query.test.ts
• CHANGELOG.md
• src/query.ts

*

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Keep this Copilot instructions file short; put substantive rules in narrow instruction files under .github/instructions/*.instructions.md.

Files:

• CHANGELOG.md

**/*.md

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.md: Keep each project documentation file limited to its canonical role; prefer the documented canonical home instead of creating a new top-level Markdown file.
Do not duplicate content from this routing index; move operational detail, schema blocks, or long rule lists to their canonical documentation or instruction files.
Apply the repository's Markdown linting and fenced-code conventions, while excluding LLM instruction files where the repository explicitly requires that exclusion.

Files:

• CHANGELOG.md

CHANGELOG.md

📄 CodeRabbit inference engine (CLAUDE.md)

Record user-visible shipped changes in CHANGELOG.md under [Unreleased] or release sections.

Files:

• CHANGELOG.md

src/{query,mcp,browse}.ts

📄 CodeRabbit inference engine (AGENTS.md)

Route MCP, query, classifier, TUI, and canonicalizer behavior through shared core code, usually src/query.ts; keep src/mcp.ts and src/browse.ts as thin adapters.

Files:

• src/query.ts

🔇 Additional comments (3)

src/query.ts (1)

2623-2661: LGTM!

Also applies to: 2663-2706, 2708-2735

src/query.test.ts (1)

519-547: LGTM!

Also applies to: 2431-2447

CHANGELOG.md (1)

68-68: LGTM!

---

📝 Walkthrough

Walkthrough

searchVideos now searches video titles/descriptions alongside transcripts, prioritizes AND then OR matches, deduplicates results per video, and selects one best transcript segment per video. Fixtures and tests cover title-only matching and pagination.

Changes

Video search

Layer / File(s)	Summary
Combined video search orchestration src/query.ts	searchVideos combines title/description and transcript FTS queries, orders AND/OR results, deduplicates by video_id, and applies the result limit.
Ranked title and transcript query helpers src/query.ts	Transcript results retain the best-ranked segment per video before snippet generation; title results include title snippets and first-segment metadata.
Search fixtures, integration tests, and changelog src/query.test.ts, CHANGELOG.md	Fixtures and tests cover title-only changelog matches and per-video deduplication; the changelog records title matching.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• tikoci/rosetta#31: Modifies shared FTS snippet markers used by video search excerpts.

Suggested labels: area:docusaurus

Sequence Diagram(s)

sequenceDiagram
  participant SearchVideos
  participant TitleFTS
  participant TranscriptFTS
  SearchVideos->>TitleFTS: Query title and description
  TitleFTS-->>SearchVideos: Ranked title results
  SearchVideos->>TranscriptFTS: Query transcript segments
  TranscriptFTS-->>SearchVideos: Best segment per video with excerpts
  SearchVideos->>SearchVideos: Deduplicate video_id and apply limit

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main change: adding video title search in src/query.ts plus regression tests in src/query.test.ts.
Linked Issues check	✅ Passed	The PR implements #89’s title-indexing fix and adds coverage; the dotted-version/routeros follow-up is explicitly deferred to #116.
Out of Scope Changes check	✅ Passed	Changes stay within search ranking, deduplication, tests, and release notes; no unrelated code paths were added.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/89-video-title-search

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Improves Rosetta’s video search retrieval so queries can match against video titles/descriptions (via the existing videos_fts) in addition to transcript segments, addressing the ranking gap described in issue #89.

Changes:

• Update searchVideos() to query both videos_fts and video_segments_fts, merging results in a strong-before-weak order with per-video deduping.
• Add an integration test proving a title-only keyword (e.g., “changelog”) surfaces the expected video even when the transcript never contains that term.
• Document the user-visible behavior change under [Unreleased] in CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/query.ts	Adds title/description FTS querying to searchVideos() and merges results with dedupe/ordering.
src/query.test.ts	Adds a fixture + test asserting title-only hits are surfaced for issue #89.
CHANGELOG.md	Records the user-visible fix for video search title matching.

[mobileskyfi]

Codex Review

Not ready to merge yet. The implementation works, but I found one code correctness issue and one mandatory project-process issue.

1. searchVideos() can under-fill results after deduplication. src/query.ts:2653 limits transcript segment rows before add() deduplicates them by video. Multiple
   matching segments from one video consume the SQL limit, hiding eligible unique videos.

   Live DB evidence:

   • searchVideos("firewall filter", 10) returns 8, while limit 20 finds 15 unique videos.
   • searchVideos("vlan bridge", 10) returns 9, while limit 20 finds 13.

   Deduplicate/select the best segment per video before applying the limit, or fetch progressively until the unique-video limit is filled. Add a regression test
   with multiple matching segments from one video.

2. The issue split is incomplete. The PR calls itself a partial slice of Video search buries changelog-title videos: dotted versions shredded, title text unindexed #89 and defers dotted-version and stop-word work, but there is no follow-up issue and
   the PR body lacks Closes Video search buries changelog-title videos: dotted versions shredded, title text unindexed #89. The repository’s mandatory linking rule requires creating the follow-up, referencing it from the PR, and closing Video search buries changelog-title videos: dotted versions shredded, title text unindexed #89 through
   the PR body. The commit-message Closes Video search buries changelog-title videos: dotted versions shredded, title text unindexed #89 is insufficient.

Everything else looks healthy:

• Current head: 194ee0d
• All GitHub checks green; mergeStateStatus: CLEAN
• Both Copilot threads addressed and resolved
• make verify: 905 passed, 0 failed
• Retrieval eval: Recall@5 92.9%, +0.3pp; video hit@5 100%
• Live changelog queries correctly rank the 7.23 and 7.22 videos first
• CodeRabbit did not perform a substantive review because its review limit was reached

[mobileskyfi]

Both Codex findings addressed in bad7525:

1. Under-fill after dedup (correctness). Confirmed and fixed. The root cause was intra-source: runVideosFtsQuery limited transcript rows before the caller deduped by video, so a video with several matching segments consumed the top-limit rows and hid distinct videos below the cutoff. runVideosFtsQuery now selects the best-ranked segment per video before LIMIT via a ROW_NUMBER() window query. Because fts5 snippet() can't run alongside a window function, it's two bounded steps — window-pick the segment rowids, then a second MATCH fetches excerpts for just those. Live DB now fills correctly: firewall filter → 10/10 unique (was 8), vlan bridge → 10/10 (was 9), changelog target stays #1. Added a regression test: one video with 3 matching segments must not crowd out a second distinct video at limit 2.

2. Issue-linking (process). Closes #89 is now in the PR body (not just the commit), and the deferred remainder — dotted-version tokenization + routeros stop-word handling across the shared query path — is split into follow-up #116, carrying over #89's un-done acceptance bullets, and referenced from the body.

Verification after both fixes: bun test 906 pass / 0 fail; Phase 0 eval Recall@5 92.9% (+0.3pp), video & changelog surfaces 100%; typecheck + biome + markdownlint clean.

[mobileskyfi]

@coderabbitai we hit a review limit. do a review of the PR as it stands now.

[coderabbitai]

@mobileskyfi Reviewing the PR at its current head, including the FTS ranking/deduplication path and regression coverage.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.