refactor(markdown): replace tree-sitter kind mimicry with pulldown-native NodeKind#174
Conversation
…tive NodeKind
`mehen-markdown` migrated off tree-sitter to pulldown-cmark, but kept the
generated tree-sitter kind table (`grammar.rs`) as its internal node
vocabulary and wrapped the owned tree in a tree-sitter-shaped API. This
carried real cost for no benefit:
- a 685-line `u16` `Markdown` enum with grammar ordinals, ~half of whose
variants pulldown-cmark can never construct;
- a `kind_id() -> u16` + `From<u16>` round-trip at 48 call sites, when the
node already stores the enum;
- a mutable `Cursor` (`goto_first_child`/`goto_next_sibling`) over 45
hand-rolled loops, when all navigation is strictly top-down;
- a named-field table used only for heading `level`/`content`;
- a ~330-line dead `From<Markdown> for &'static str` impl (escaped the
dead-code lint via the trait-impl exemption).
Replace it with a hand-authored `NodeKind` enum (`src/kind.rs`) modeling
only the ~62 kinds the builder emits. Numbered tree-sitter families
(`atx_heading2..6`, `section1..6`, `list_item2..5`, task variants) fold
into data-carrying variants (`Heading { level, style }`, `Section { level }`,
`ListItem { task }`); `kind_id()` becomes `kind() -> NodeKind`; the cursor
becomes a `children()` iterator; the field table is dropped.
Behavior-preserving: heading level and atx/setext style survive as variant
data, Halstead's per-level heading-marker operators and MCC's list-vs-task
weights are kept intact. Drops the now-unused `num` / `num-derive` /
`num-traits` dependencies.
Verified: `cargo clippy --all-targets --all-features --locked` clean
(incl. `-W dead_code -W unreachable_pub`); `cargo insta test --workspace
--check --unreferenced reject` passes 905/905 with no snapshot changes.
Copy/Paste Detection🟢 No duplications found in 20 changed Rust file(s) (threshold: 100 tokens). |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the mehen-markdown crate to fully migrate away from tree-sitter and its generated grammar.rs file. It introduces a hand-authored NodeKind enum in src/kind.rs that mirrors the pulldown-cmark event surface, consolidating various numbered variants (such as headings, list items, and sections) into data-carrying variants. Additionally, the mutable Cursor traversal API has been replaced with a clean, top-down Children iterator, and unused dependencies (num, num-derive, num-traits) have been removed. All metric passes and helpers have been updated to use the new NodeKind and iterator-based traversal. I have no feedback to provide as there are no review comments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Warning Review limit reached
Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThe Markdown analyzer replaces its generated Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Binary Size ReportMeasured artifact: release Linux
Size historyxychart-beta
title "Binary size (bytes)"
x-axis ["build(deps): bump ra_ap_syntax", "build(deps): bump gix from 0.8", "build(deps): bump log from 0.4", "build(deps): bump the ruff gro", "build(deps): bump the mago gro", "build: sync workspace crate ve", "build(deps): bump the oxc grou", "build(deps): bump tree-sitter ", "build(deps): bump camino from ", "feat: add Java analyzer and up", "build(deps): bump sqruff-lib-d", "build(deps): bump the mago gro", "build(deps): bump ra_ap_syntax", "build(deps): bump the oxc grou", "build(deps): bump mago-syntax-", "chore(main): release 1.3.0 (#1", "docs: surface SQL + ANTLR acro", "This PR"]
y-axis "Bytes"
bar [29151560, 29154336, 29184128, 29154616, 29184664, 29184664, 29194704, 29194696, 29194704, 30767472, 30833880, 30833888, 30834016, 30811952, 30811944, 30811984, 31135680, 31113224]
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3820ebe3f9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex P1: CLAUDE.md was updated to note that mehen-markdown is pulldown-cmark-backed and has no generated grammar.rs, but AGENTS.md still carried the unqualified "never edit grammar.rs" rule — leaving the authoritative agent instructions in conflict with this PR's deletion of crates/mehen-markdown/src/grammar.rs. Mirror the CLAUDE.md carve-out here. Addresses: #174 (comment)
…ion_mark
CI runs clippy 1.97 (stricter `question_mark` lint under `-D warnings`)
and flagged `extract_domain`'s `match dest.find("://") { Some(pos) => …,
None => return None }`. Rewrite as `let pos = dest.find("://")?;`.
Behavior-identical; local clippy (1.96) did not surface this.
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/mehen-markdown/src/syntax_tree.rs | 350 (main: 382) 🟢 | 164 (main: 179) 🟢 | 97 (main: 100) 🟢 | 485 (main: 509) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/mrpc.rs | 134 (main: 146) 🟢 | 103 (main: 138) 🟢 | 37 ⚪ | 267 (main: 294) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/prose/lang_detect.rs | 129 (main: 135) 🟢 | 112 (main: 132) 🟢 | 28 ⚪ | 190 (main: 210) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/mcc.rs | 106 (main: 135) 🟢 | 101 (main: 141) 🟢 | 26 (main: 29) 🟢 | 226 (main: 267) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/filler.rs | 86 (main: 90) 🟢 | 63 (main: 73) 🟢 | 20 ⚪ | 170 (main: 178) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/grounding.rs | 110 (main: 116) 🟢 | 88 (main: 103) 🟢 | 18 ⚪ | 179 (main: 191) 🟢 | 0 ⚪ |
| crates/mehen-markdown/src/halstead.rs | 77 (main: 86) 🟢 | 39 (main: 52) 🟢 | 17 (main: 16) 🔴 | 142 (main: 151) 🟢 | 1.87 (main: 0) 🟢 |
| crates/mehen-markdown/src/sections.rs | 42 (main: 69) 🟢 | 30 (main: 61) 🟢 | 13 (main: 16) 🟢 | 79 (main: 116) 🟢 | 13.98 (main: 6.31) 🟢 |
| crates/mehen-markdown/src/loc.rs | 41 (main: 44) 🟢 | 16 (main: 25) 🟢 | 9 ⚪ | 43 (main: 48) 🟢 | 17.68 (main: 15.05) 🟢 |
| crates/mehen-markdown/src/tree_helpers.rs | 38 (main: 48) 🟢 | 32 (main: 67) 🟢 | 8 ⚪ | 41 (main: 65) 🟢 | 18.70 (main: 14.10) 🟢 |
| crates/mehen-markdown/src/tables.rs | 44 (main: 54) 🟢 | 34 (main: 54) 🟢 | 7 ⚪ | 89 (main: 121) 🟢 | 15.81 (main: 12.24) 🟢 |
| crates/mehen-markdown/src/visuals.rs | 43 (main: 48) 🟢 | 29 (main: 36) 🟢 | 7 ⚪ | 83 (main: 90) 🟢 | 12.92 (main: 11.53) 🟢 |
| crates/mehen-markdown/src/nearby.rs | 15 (main: 17) 🟢 | 11 (main: 16) 🟢 | 5 ⚪ | 27 (main: 31) 🟢 | 29.40 (main: 26.48) 🟢 |
| crates/mehen-markdown/src/prose/mod.rs | 32 (main: 36) 🟢 | 19 (main: 27) 🟢 | 5 ⚪ | 59 (main: 67) 🟢 | 14.23 (main: 12.73) 🟢 |
| crates/mehen-markdown/src/analyzer.rs | 25 (main: 27) 🟢 | 18 (main: 23) 🟢 | 4 ⚪ | 87 (main: 91) 🟢 | 9.58 (main: 8.90) 🟢 |
| crates/mehen-markdown/src/ecu.rs | 20 (main: 26) 🟢 | 18 (main: 36) 🟢 | 4 ⚪ | 43 (main: 56) 🟢 | 25.35 (main: 22.55) 🟢 |
| crates/mehen-markdown/src/kind.rs | 14 🆕 | 2 🆕 | 4 🆕 | 4 🆕 | 24.38 🆕 |
| crates/mehen-markdown/src/math_burden.rs | 17 (main: 19) 🟢 | 17 (main: 22) 🟢 | 3 ⚪ | 39 (main: 43) 🟢 | 28.12 (main: 27.18) 🟢 |
| crates/mehen-markdown/src/words.rs | 9 (main: 11) 🟢 | 5 (main: 10) 🟢 | 3 ⚪ | 12 (main: 16) 🟢 | 39.53 (main: 37.37) 🟢 |
| crates/mehen-markdown/src/grammar.rs | 0 (was: 330) 🟢 | 0 (was: 1) 🟢 | 0 (was: 4) 🟢 | 0 (was: 5) 🟢 | 0 (was: 0) ⚪ |
Generated by mehen v1.3.0 — the code quality watcher.
📝 Documentation Metrics (this PR vs main)
| File | DMI | Words | FKGL | Link Debt | Filler Risk |
|---|---|---|---|---|---|
| AGENTS.md | 97 (main: 97) ⚪ | 112 (main: 92) ⚪ | 5.8 (main: 5.8) ⚪ | 0.00 ⚪ | 0.32 (main: 0.31) ⚪ |
| CLAUDE.md | 93 (main: 93) ⚪ | 400 (main: 353) ⚪ | 7.8 (main: 8.1) ⚪ | 0.00 ⚪ | 0.35 ⚪ |
Full metric breakdown (structural · wording · lexical · readability)
Structural / review
| File | RCI | MCC | MRPC | Evidence | Grounding |
|---|---|---|---|---|---|
AGENTS.md |
5 ⚪ | 5 ⚪ | 1 ⚪ | 0.15 ⚪ | 0.20 ⚪ |
CLAUDE.md |
21 ⚪ | 20 ⚪ | 1 ⚪ | 0.18 ⚪ | 0.25 ⚪ |
English wording quality
| File | WQS | Passive % | Hedges /100w | Long sent. | Nominalizations |
|---|---|---|---|---|---|
AGENTS.md |
0.99 (main: 1.00) ⚪ | 18% (main: 19%) ⚪ | 0.0 ⚪ | 0 ⚪ | 0% ⚪ |
CLAUDE.md |
1.00 (main: 1.00) ⚪ | 8% (main: 8%) ⚪ | 0.0 ⚪ | 0 ⚪ | 2% (main: 3%) ⚪ |
English lexical & readability ensemble
| File | MATTR₅₀ | Hapax | Fog | SMOG | ARI | Coleman-Liau |
|---|---|---|---|---|---|---|
AGENTS.md |
0.85 (main: 0.86) ⚪ | 0.71 (main: 0.71) ⚪ | 6.7 (main: 6.8) ⚪ | — | 5.0 (main: 5.0) ⚪ | 8.5 (main: 8.6) ⚪ |
CLAUDE.md |
0.86 ⚪ | 0.72 (main: 0.74) 🟢 | 8.8 (main: 9.0) ⚪ | 9.9 (main: 9.9) ⚪ | 7.2 (main: 7.4) ⚪ | 11.1 (main: 11.5) ⚪ |
Legend: 🟢 improvement · 🔴 regression ·
⚠️ attention · 🆕 new file · ⚪ no material change
Generated by mehen — the code quality watcher.
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
mehen-markdownmigrated off tree-sitter topulldown-cmark, but it kept the generated tree-sitter kind table (src/grammar.rs) as its internal node vocabulary and wrapped the owned tree in a tree-sitter-shaped API. That mimicry cost us without buying anything:grammar.rs— a 685-lineu16Markdownenum with grammar ordinals, ~half of whose variants (MdxJsx*,HtmlBlock1/3..7,_line_repeat1, …) pulldown-cmark can never construct.kind_id() -> u16+From<u16>round-trip at 48 call sites, even though each node already stores the enum.Cursor(goto_first_child/goto_next_sibling) driving 45 hand-rolled child-iteration loops, when all navigation in the crate is strictly top-down.level,content), one of which fed dead code.From<Markdown> for &'static strimpl that escaped the dead-code lint via the trait-impl exemption.This surfaced from the question "why does
mehen-markdown/src/grammar.rsexist?" — it's a fossil from the pre-migration tree-sitter implementation, repurposed as a costume the crate no longer needs.What
Replace the generated enum with a hand-authored
NodeKind(src/kind.rs) modeling only the ~62 kinds the builder actually emits, and make the owned tree pulldown-native:u16enum, ~half deadkind_id() -> u16then.into()×48kind() -> NodeKinddirectlyCursor+goto_*(45 loops)children() -> impl IteratorAtxHeading2), read via marker child +child_by_field_name("level")Heading { level, style }num,num-derive,num-traitsgrammar.rsused them)Numbered tree-sitter families fold into data-carrying variants —
AtxHeading2..6/SetextHeading2→Heading { level, style },Section1..6→Section { level },ListItem2..5/TaskListItem*→ListItem { task }— because every consumer matched them as whole groups, never individually.Behavior preservation
This is a behavior-preserving refactor — no metric formula, weight, threshold, or control-flow change. The two spots folding could have broken are explicitly preserved:
#≠##) viaHeadingMarker { level, style }+ aheading_marker_op()mapping.ListItem { task }.The degenerate
heading_kindsfallback (a setext underline detected at level >2 coerces to ATX H1) is reproduced inresolve_heading().Verification
cargo clippy -p mehen-markdown --all-targets --all-features --locked— 0 warnings (incl. the crate's stricter-W dead_code -W unreachable_pubsweep).cargo insta test --workspace --all-features --check --unreferenced reject— 905/905 tests pass, no snapshot changes, no unreferenced snapshots. SinceNodeKindis never serialized, byte-identical snapshots are the proof that metric outputs are unchanged.cargo check --workspace(dependents compile),cargo machete(no unused deps),cargo xtask tree-sitter check-generated(still green — markdown was never a codegen target).Net
−1,346 lines (
+877 / −2,223), the bulk from deleting the 685-linegrammar.rsand collapsing tree-sitter-shaped boilerplate across 17 metric passes.CLAUDE.md is updated to document the pulldown-cmark backing and carve
mehen-markdownout of the "never edit generatedgrammar.rs" hard rule (it no longer has one;kind.rsis hand-authored and edited directly).