Give pages a lead (H0) section so section coverage is total (B-0023)#105
Conversation
parseSections split the page body only on h1–h3, so pre-first-heading prose (the AI-summary blockquote, the intro paragraph, and the entire body of pages whose only heading is the title) belonged to no `sections` row — 11.7% of the corpus by word and 40 whole pages on rc.99, none of it rolling up to the page or carrying a resolvable section_id. Mint a synthetic lead fragment per page (level 0, reserved anchor_id "_lead", heading = page title) holding that content. `_lead` is provably collision-free: slugify() strips every character outside [a-z0-9\s-], so no heading can produce an underscore. attributeSection() now resolves pre-heading rows to the lead instead of NULL. A title-only lead mints nothing (no empty fragments; #93 rule). Rebuilt from the local cache: section words cover 98.2% of page words (was 88.3%; residual is heading-text lines), 365 lead fragments, and tables/ properties/callouts with no section context all drop to 0 (from 16/12/104). No schema DDL change (sections already stores level/anchor_id) and no MCP tool-surface change — this ships the data a future #27 retrieval-unit change would need. Export pages.tsv/pages_summary.tsv now roll up; the section_id disclosure and a fixture coverage-invariant test reflect the fuller coverage. Briefing: B-0023-page-section-normalization. Part of #95; folds in #93's empty-section rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughThe Docusaurus extractor now creates a synthetic ChangesLead Section Normalization
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant DocusaurusMarkdown
participant parseSections
participant parsePage
participant Storage
participant rosettaExport
DocusaurusMarkdown->>parseSections: page markdown
parseSections->>parsePage: sections including optional _lead
parsePage->>Storage: pages, sections, tables, callouts
Storage->>rosettaExport: section and page rollups
rosettaExport-->>Storage: updated export disclosures
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Normalizes Docusaurus page sectioning so sections becomes a near-total overlay of each page by introducing a synthetic lead (“H0”) fragment for all prose before the first real h1–h3 heading. This improves rollups/exports and makes pre-heading tables/properties/callouts reliably attributable to a resolvable section_id, aligned with the B-0023 briefing and #95’s schema/ETL audit track.
Changes:
- Add a reserved lead section anchor (
_lead) and updateparseSections()/attributeSection()so pre-first-heading content resolves to that lead fragment when present. - Expand unit + storage/integration tests to cover lead minting, ordering, attribution behavior, and coverage invariants.
- Update export disclosures and add a user-visible
CHANGELOG.mdentry documenting the behavior change.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/extract-docusaurus.ts | Introduces LEAD_ANCHOR, emits lead (H0) fragments, updates section attribution rules, and reports lead/coverage metrics in the extractor summary. |
| src/extract-docusaurus.test.ts | Updates/extends parsing + attribution tests to validate the new lead-fragment behavior and adjusted expectations. |
| src/extract-docusaurus-storage.test.ts | Adds a DB-level coverage invariant and lead/level-0 assertions to catch regressions after extraction. |
| src/export.ts | Updates disclosure text to reflect that pre-heading content now typically resolves to _lead. |
| CHANGELOG.md | Adds an [Unreleased] entry describing the new lead fragment and its effects on attribution/exports. |
| briefings/B-0023-page-section-normalization.md | Records the finalized, maintainer-reviewed decision and resolved questions for the lead-fragment approach. |
|
@coderabbitai review |
✅ Action performedReview finished.
|
Codex Review/Fixes• PR #105 (#105) is reviewed, fixed, and ready to merge. Pushed commit 8f34216, which:
Validation:
|
Makes a page's
sectionsa total overlay of the page instead of a partial one, by minting a synthetic lead ("H0") fragment for the prose before the first h1–h3 heading. Homework, options, and the maintainer-reviewed decisions are inbriefings/B-0023-page-section-normalization.md.Why
parseSections()split the body only on h1–h3, so the lead-in prose — the AI-summary blockquote, the intro paragraph, and the entire body of pages whose only heading is the title — belonged to nosectionsrow. On rc.99 that was 11.7% of the corpus by word (76,815 words) and 40 whole pages with no section at all, none of it rolling up to the page total or carrying a resolvablesection_id. Worst casehotspot-customisation: one late h3 section covered 1,170 of 5,256 words (78% orphaned).This is the pivot-table-friendliness gap the B-0022 export surfaced (
pages.tsvsections didn't sum to the page), and the precondition for any future #27 section-level retrieval ("select sections X, Y, Z" instead oflength=).What changed
parseSections()emits a lead fragment for each page with non-empty lead prose:level = 0, reservedanchor_id = "_lead", heading = page title,sort_order = 0(real sections shift to1..n). A title-only lead mints nothing — no empty fragments (the Empty sections (129 / 4.4%): heading with no body wastes a get_page section= tool call #93 empty rule)._leadis provably collision-free, not merely unlikely:slugify()deletes every character outside[a-z0-9\s-], so no heading — and nofoo/foo-1suffix — can produce ananchor_idcontaining_.attributeSection()resolves pre-first-heading rows to the lead anchor instead ofNULL(only genuinely lead-less pages stayNULL).section_iddisclosure updated.Verification (rebuilt from the local page cache)
hotspot-customisation1,170 → 5,250 covered,software-specifications47 → 1,294,address-lists(a no-section page) 0 → 187.rosetta exportregenerated:pages.tsvcarries the_leadrows and rolls up.Scope / non-goals
sectionsalready storeslevel/anchor_id, so no migration and noSCHEMA_VERSIONbump; a re-extract simply produces the fuller rows.Tests
Full suite green (877 pass / 0 fail). Anchor tests that encoded the old "preamble → null / zero sections" behavior were intentionally flipped with rationale; added lead-fragment / title-only-lead / lead-attribution unit tests and a fixture-built-DB coverage-invariant assertion (
section_words / page_words > 0.9, lead present,level 0reserved for the lead).Briefing:
briefings/B-0023-page-section-normalization.md. Part of #95; folds in #93's empty-section rule.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Exports