Move away from custom regex-based parsing of PEG expressions

[pmolodo]

(Low priority... mostly wanted to record some findings in case we want to act on them later...)

Currently, our implementation of railroad diagrams relies on peg_to_peg.py, which converts from "classic ford-style flavor PEG grammar" to "pegen-flavor PEG grammar".

It then uses the pegen parser to generate an AST, and iterates that to generate the railroad diagrams.

However, the implementation of peg_to_peg.py relies on some custom regexes to convert between the two grammar flavors, which seems somewhat fragile + high maintenance.

As an alternative, we could either:

• just write the rules with pegen flavor to begin with (see comment)
• instead of relying on the pegen parser for generating railroad diagrams, use a different parser that understands "classic ford-style flavor PEG grammar" out-of-the-box - options include:
  • arpeggio with the "classic" PEG frontend
    • con: requires ALL rules to be terminated by ";"
  • the pe module
    • con: much less widely used, single dev
  • probably leaning toward arpeggio since it's much easier to fix the ";" termination now than it is to have to switch the parser later...

[pmolodo]

On investigation of the pegen grammar, I think it's not a viable in-source replacement:

Main differences for us are:

Concept	Ford	Pegen
rule generation	<-	:
alternation	/	|
character classes	ie, [A-Z]	no support - manual expansion via alternation

First two are minor, but the lack of character classes is big loss for something human-readable - we'd have to go from:

AsciiIdentifierStart <- [A-Z] / [a-z] / Underscoreor

or

AsciiIdentifierStart <- [A-Za-z_]

to:

AsciiIdentifierStart:
      | 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'G' | 'H' | 'I' | 'J'
      | 'K' | 'L' | 'M' | 'N' | 'O' | 'P' | 'Q' | 'R' | 'S' | 'T'
      | 'U' | 'V' | 'W' | 'X' | 'Y' | 'Z'
      | 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i' | 'j'
      | 'k' | 'l' | 'm' | 'n' | 'o' | 'p' | 'q' | 'r' | 's' | 't'
      | 'u' | 'v' | 'w' | 'x' | 'y' | 'z'
      | Underscore

So on second thought, switching to pegen probably isn't gonna cut it

[pmolodo]

Note: Our peg-to-pegen translator doesn't generate the above long-alternation pegen rule above, but actually pre-processes the first input above to insert fancy unicode characters, and ends up with:

AsciiIdentifierStart: ⟦A−Z⟧ | ⟦a−z⟧ | Underscore

(Note that the middle dash is a special unicode dash as well)

However, in pure pegen grammar, the fancy unicode characters ⟦, −, and ⟧ have no special meaning... but, as of python 3.13, pegen DOES allow ⟦ as the start of an identifier... so ⟦A−Z⟧ is just parsed as a non-terminal name.

The railroad diagram generator then has special logic to find nonterminals like ⟦A−Z⟧ and render them nicely...

[pmolodo]

For more detail - here's some claude chat logs while investigating:

========================================================================================================================
(Claude session from core-spec-wg)

❯ Railroad diagram generation is failing. It was last known good at 52e596ea9b9c4a8a925ef860bb1bc04f3713353d and failing at a72cad7f9d4b60106976be53364da9a0097c2e42. Bisect the git history to determine where the failure was introduced. Recommend making a script that signals good/fail via 0/1 return value, then using git bisect run. Note that pixi run main build will return 0 (success) exit code even if railroad diagrams error, so you will either need to check stdout/stderr or presence of railroad diagrams on disk to determine success or failure.

● Bisect result

First bad commit: cdde310f932b7be10d9e2803266101d3d088a8b1
"Fix PEG grammar for schema family and version" by Matt Kuruc (2026-04-16)

The commit adds a new PEG rule to specification/schemas/README.md:

Family <- (AsciiStart) (!('_' Version !AsciiContinue) AsciiContinue)*

The !(...) negative lookahead/predicate is what the railroad-diagram filter chokes on. When that one rule fails, the whole pandoc invocation for the Schemas doc errors, and build_docs.py:31 falls back to the unconverted
source with the warning Could not generate railroad diagrams for Schemas doc.

Detection method used

pixi run main build exits 0 even when the fallback fires, so the bisect script (/src/aousd/core-spec-wg/.claude_temp_scripts/.../bisect_railroad.sh) builds with --no-pdf --no-docx (faster) and flags BAD on either:

1. The literal warning string in build output, or
2. A mismatch between

    block count and <img role="img" src="data:image/svg count in the rendered HTML.

At good commits both counts equal 190; at the bad commit the warning fires and the schemas peg blocks have no diagrams.

Notes

• I tried to log this in beads but bd returned failed to open database: embeddeddolt: ... no database selected - the dolt server appears unhealthy. Per your rule I did not touch the server; just reporting it.
• No code changes to commit; the bisect script lives under .claude_temp_scripts/.
• HEAD is currently a72cad7 (the bad tip) after git bisect reset.

❯ Why does the filter choke on the !(...) negative lookahead/predicate? Is it an error in the PEG grammar or our railroad generation script?

● Root cause: bug in the railroad filter, not the grammar

The breaking PEG is well-formed; pegen parses both Family <- (AsciiStart) (!('_' Version !AsciiContinue) AsciiContinue)* and the new AsciiStart/AsciiContinue rules fine when fed individually. The negative-lookahead
!(...) is a red herring.

The actual bug is in doc_build's filter_railroad.py:74:

old_peg = "".join(code.split("\n"))

It strips all newlines from a peg ``` fenced block before calling convert_standard_peg_to_pegen, which then does standard_peg.splitlines() -- so what was meant to be a per-rule loop only ever sees one line.

For a fence with one rule that's fine. For the new fence the spec added in cdde310:

AsciiStart <- [A-Za-z_]
AsciiContinue <- [A-Za-z0-9_]

the joined string is AsciiStart <- [A-Za-z_]AsciiContinue <- [A-Za-z0-9_]. The rule regex ([\w<>]+)\s*<-\s*(.+) greedily captures rule_body = "[A-Za-z_]AsciiContinue <- [A-Za-z0-9_]", which still contains <-. The
unconditional replace_brackets then maps every < -> ⟨, producing ⟨-, which pegen rejects:

AsciiStart: ⟦A−Za−z_⟧AsciiContinue ⟨- ⟦A−Za−z0−9_⟧
^^^^^^^^^^^^^^
SyntaxError

Pandoc returns 1, the schemas filter aborts, and build_docs.py swallows the CalledProcessError with the warning -- which is why the schemas section ends up with 7 unrendered peg ``` blocks.

Two ways to fix

1. In the spec (one-line workaround, verified): split the two-rule fence into two single-rule fences. With that change pandoc returns 0 and all railroad diagrams render.
2. In doc_build: drop the "".join(code.split("\n")) flatten on line 74 -- pandoc already gives you the original newlines in code. (If that line was originally there to support a single rule wrapped over multiple lines,
   the right fix is to recognize rule boundaries by leading name <-, not to obliterate all newlines.)

Option 1 unblocks the spec build today; option 2 prevents anyone else hitting this whenever they put two rules in one fence (which any reader would expect to work, given how the rest of the toolchain treats PEG).

❯ Is there a way to rely less on our own ad-hoc parsing, and utilitize a 3rd party module that does basic parsing (or at least tokenization) of peg grammars, to make this more robust?

● Yes, and the win would mostly come from dropping peg_to_peg.py entirely, not from changing how you parse. The fragility isn't pegen -- pegen does fine. It's the layer that string-rewrites the user's PEG into pegen's
dialect via Unicode lookalikes (< -> ⟨, ' -> ＇, etc.) plus the "".join(code.split("\n")) flatten on top. That layer is where your bug lives, and it will keep producing surprises whenever someone writes legal PEG that
happens to contain a character the rewriter cares about.

The clean shape is: pick a grammar library whose native input language is standard PEG (rule with <-, choice with /, classes [...], lookaheads & !, quantifiers * + ?), parse the user's fenced block directly, then walk
the resulting AST to emit railroad nodes. Same code path you have today after pegen parses, just with a parser that doesn't need a translation layer.

Reasonable candidates to evaluate (all on PyPI, all support standard-PEG syntax close to what the spec uses):

• arpeggio -- has a cleanpeg front-end that takes Rule <- ... directly, exposes the parsed grammar as an inspectable tree. Most natural fit for the spec's dialect.
• pe -- specifically aimed at standard PEG; smaller surface area than arpeggio.
• TatSu -- broader/EBNF-leaning; usable but more than you need.

Tradeoffs you'd be choosing between:

• This is a doc_build change, not a spec change -- so it's a separate repo, separate PR, and you have to verify all 190-ish existing diagrams render identically (pixel diff or just visual spot check on the published
  HTML).
• The cheap path is the one-character spec workaround (split the offending fence into two) plus also fixing line 74 to not flatten newlines. That's ~5 lines in doc_build and gets you robust against the specific class of
  bug we just hit, without committing to a library swap.

I'd suggest the cheap fix now and file an issue against doc_build for the bigger rewrite, unless you've been hitting other rewriter-related bugs that would justify doing it now. Want me to draft either?

❯ What are the differences between peg and pegen grammar? Why do we need pegen?

● TL;DR

Standard PEG (Bryan Ford, 2004) and pegen's input language are different DSLs that happen to overlap. Pegen is a parser-generator the CPython team built to write Python's own grammar in -- it's vendored into doc_build
and used here as a grammar parser (i.e. a parser that takes grammar text and gives you back an AST you can walk). The conversion layer in peg_to_peg.py exists because the spec is written in standard PEG but pegen doesn't
accept standard PEG.

Concrete syntax differences

┌───────────────────────────────────────────────────────────────────────────────┬─────────────────┬─────────────────────────────────────────────────────────────────────────┬───────────┐
│ Feature │ Standard PEG │ Pegen │ Spec uses │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Rule definition │ Rule <- expr │ rule: expr │ <- │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Ordered choice │ a / b │ a | b │ / │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Char class │ [A-Za-z] │ [expr] means optional │ [A-Za-z] │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ String literal │ 'x' or "x" │ 'x' = hard keyword; "x" = soft keyword (consumed via Python's tokenize) │ 'x' │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Lookahead & / ! │ same │ same │ same │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Sequence, group, * + ? │ same │ same │ same │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Cut, actions {...}, named capture a=rule, type annotations rule[T], forced && │ none │ yes │ n/a │
├───────────────────────────────────────────────────────────────────────────────┼─────────────────┼─────────────────────────────────────────────────────────────────────────┼───────────┤
│ Tokenizer │ grammar-defined │ Python's tokenize module │ n/a │
└───────────────────────────────────────────────────────────────────────────────┴─────────────────┴─────────────────────────────────────────────────────────────────────────┴───────────┘

The two starred rows are why a translation layer is needed at all:

1. [...] is overloaded. In standard PEG it's a character class; in pegen it's an optional group (a synonym for expr?). So [A-Za-z_] in the spec would parse as "optionally match the rule reference A, then -, then ..." --
   nonsense. The translator works around this by mapping [...] -> ⟦...⟧ (math angle brackets) and then gen_svg.py recognises that lookalike to emit a char-class railroad node.
2. Pegen runs the source through Python's tokenizer. That means '_' is a Python string literal, identifiers must be Python-legal, and stray punctuation like < or > becomes Python operators. The translator replaces ' with
   ＇ (full-width apostrophe), < with ⟨, > with ⟩, etc., to neutralise them so the Python tokenizer doesn't choke or misinterpret them.

That second point is also why the "".join(code.split("\n")) flatten existed -- the Python tokenizer cares about NEWLINE tokens between statements, and the author was probably trying to avoid that. It's also where last
week's bug came from.

Sources:

• pegen grammar reference -- documents : for rules, | for alternation, [expr] for optional, &&, ~, named captures, actions, type annotations
• pegen "PEG Parsers" overview -- explains 'x' vs "x" (hard vs soft keywords), uppercase tokens, invalid_* rules, and that pegen relies on an external (Python) tokenizer
• we-like-parsers/pegen on GitHub -- origin: spun out of CPython's PEP 617 work
• Wikipedia: Parsing expression grammar -- standard PEG syntax (<-, /, [...] as char class)

Why pegen specifically

Strictly speaking, you don't need pegen -- you need some library that turns grammar text into an inspectable AST so gen_svg.convert_node has something to walk. Pegen got picked because:

• It's a real, battle-tested PEG parser (it parses the CPython grammar).
• It exposes its own grammar_parser as a Python-callable API, so you can parse a grammar from a string with a few lines of glue.
• It comes with its own pegen.grammar AST node types (Alt, Rhs, Group, Repeat0, Lookahead, etc.) that map cleanly onto railroad-diagram primitives.

The cost is that its input dialect was tuned for Python's needs (Python tokenizer, Python actions, type annotations) rather than for accepting grammars written in the academic standard-PEG syntax the spec uses. That
mismatch is what peg_to_peg.py is paying off, awkwardly. A library whose native input is standard PEG (arpeggio's cleanpeg, pe, etc.) wouldn't need a translation layer, which is why the previous answer pointed at
swapping out pegen rather than patching the translator.

❯ What would an approach that didn't rely on pegen look like?

● The cleanest shape is a small meta-grammar that describes spec-PEG, run through any general-purpose parser (parsimonious, lark, arpeggio, hand-rolled -- doesn't matter much). The user-facing PEG dialect stays exactly
what the spec uses today; you just trade ad-hoc string rewrites for a declarative grammar that you own.

Three layers

┌─────────────────────────────────────────────────────────────────────┐
│ 1. Meta-grammar (PEG-of-PEG, ~20 lines) │
│ Describes what spec-PEG looks like │
├─────────────────────────────────────────────────────────────────────┤
│ 2. Visitor (~30 lines) │
│ Turns parse tree into your own domain AST │
│ (Rule, Sequence, Choice, ZeroOrMore, Lookahead, CharClass, …) │
├─────────────────────────────────────────────────────────────────────┤
│ 3. Renderer (mostly already exists in gen_svg.py) │
│ Walks the AST, emits railroad.Diagram(...) calls │
└─────────────────────────────────────────────────────────────────────┘

Today, layer 1 is peg_to_peg.py (regex hacks + Unicode lookalikes), and the parser is pegen, whose AST you then walk in gen_svg.convert_node. The tangle is that pegen's AST node types (Alt, Rhs, NamedItem, ...) don't
match PEG concepts cleanly, so the walker has to filter out pegen-isms. With this approach, both go away.

Concrete sketch (parsimonious as the meta-parser)

from parsimonious.grammar import Grammar
from parsimonious.nodes import NodeVisitor

PEG_META = r"""
grammar = _ (rule )+
rule = name _ "<-" _ expression
expression = sequence ( "/" _ sequence)*
sequence = prefixed (_ prefixed)*
prefixed = lookahead? suffixed
lookahead = "&" / "!"
suffixed = primary suffix?
suffix = "?" / "" / "+"
primary = group / char_class / literal / name
group = "(" _ expression _ ")"
char_class = ~r"[(?:\.|[^\]\\])+]"
literal = ~r"'(?:\.|[^'\\])'" / ~r'"(?:\.|[^"\\])"'
name = ~r"[A-Za-z_][A-Za-z_0-9]"
_ = ~r"(?:\s|#[^\n])"
"""

META = Grammar(PEG_META)

class ToAst(NodeVisitor):
def visit_rule(self, n, c): return Rule(c[0], c[4])
def visit_expression(self, n, c): return _flatten_choice(c)
def visit_sequence(self, n, c): return _flatten_seq(c)
def visit_prefixed(self, n, c):
la, atom = c
return Lookahead(la, atom) if la else atom
def visit_suffixed(self, n, c):
atom, suf = c
return {"?": Opt, "*": Star, "+": Plus}suf if suf else atom
def visit_group(self, n, c): return c[2]
def visit_char_class(self, n, c): return CharClass(n.text[1:-1])
def visit_literal(self, n, c): return Literal(n.text[1:-1])
def visit_name(self, n, c): return RuleRef(n.text)
def generic_visit(self, n, c): return c or n.text

def parse_peg(text: str) -> list[Rule]:
return ToAst().visit(META.parse(text))

Then the railroad emit step is essentially what gen_svg.convert_node already does, but dispatching on your node types (Rule, Choice, Sequence, Star, Lookahead, CharClass, Literal, RuleRef) instead of pegen's. That walker
is short -- one method per node type, each ~3 lines mapping to a railroad.* constructor.

What goes away

• All of peg_to_peg.py (~70 lines of regex + Unicode rewrites)
• The vendored pegen/ package (~3 MB, 10 files)
• The Python-tokenize-based parsing in filter_railroad.py:80-90
• The whole class of bugs caused by Python tokenizer interactions, including the line-flatten that triggered last week's failure

What you keep

• filter_railroad.py's pandoc-filter shell (read CodeBlock, emit CodeBlock + Image)
• railroad.py (the rendering library)
• Most of gen_svg.py (the simplify() + railroad-shape selection logic), with the dispatch retargeted to the new AST

Tradeoffs / things to actually verify

• Add a small dep (parsimonious is ~1k LOC pure Python, MIT, mature; or use pe which natively accepts <-///[...] if you'd rather skip the meta-grammar altogether).
• One-time audit: walk the existing four PEG-bearing docs and make sure the meta-grammar parses every form they use (line continuations? comments? Unicode codepoint escapes like U+0041 -- the current code has special
  handling for those).
• Visual diff: render every existing diagram with both pipelines and compare; aim for byte-identical SVGs, fall back to spot-check if the layout heuristics differ.

Want me to file this as an issue against doc_build (with the sketch above as the design note), or prototype it in a branch first?

Sources:

• Parsimonious README -- meta-grammar syntax (=, /, ~"regex", &/!), NodeVisitor API
• pe (Python PEG) -- alternative library that natively accepts <-, /, [a-z]
• Arpeggio CleanPEG -- third option, uses = and regex char classes
• we-like-parsers/pegen -- the library being replaced

❯ I don't want to create our own 3rd grammar flavor - I want authors to rely on writing standard PEG grammar. What solutions exist that take standard PEG grammar as input, and allow railroad diagram generation, without passing through pegen? Third party libraries that handle standard PEG grammar parsing are fine.

● I just verified this end-to-end against the actual spec content. Recommendation:

pe (Python pe library) -- best fit

goodmami/pe, pure-Python, MIT, on PyPI as pe. It natively accepts standard PEG -- <-, /, [a-z], 'literal', "literal", *, +, ?, &, !, (...), # comments -- with no preprocessing for any of those. And critically, it exposes
the parsed grammar as an inspectable AST, which is what you need to drive the railroad emitter.

I fed it the exact text from specification/schemas/README.md (the rules that broke the pipeline) and it parses cleanly:

from pe.parse import loads
, defs = loads("""
... AsciiStart <- [A-Za-z]
... AsciiContinue <- [A-Za-z0-9]
... SchemaName <- Family ('' Version)?
... Family <- (AsciiStart) (!('' Version !AsciiContinue) AsciiContinue)*
... Version <- [0-9]+
... """)
defs["Family"]
(Operator.SEQ, ([
(Operator.SYM, ('AsciiStart',)),
(Operator.STR, ((Operator.SEQ, ([
(Operator.NOT, ((Operator.SEQ, ([
(Operator.LIT, ('_',)),
(Operator.SYM, ('Version',)),
(Operator.NOT, ((Operator.SYM, ('AsciiContinue',)),)),
],)),)),
(Operator.SYM, ('AsciiContinue',)),
],)),)),
],))

Each node is (Operator., args). The tags map almost 1:1 onto railroad primitives:

┌──────────────────────┬─────────────────────────────────────────────┐
│ pe tag │ Railroad │
├──────────────────────┼─────────────────────────────────────────────┤
│ LIT (literal) │ Terminal │
├──────────────────────┼─────────────────────────────────────────────┤
│ CLS (char class) │ Terminal (or rendered specially as a class) │
├──────────────────────┼─────────────────────────────────────────────┤
│ SYM (rule reference) │ NonTerminal │
├──────────────────────┼─────────────────────────────────────────────┤
│ SEQ │ Sequence │
├──────────────────────┼─────────────────────────────────────────────┤
│ CHC │ Choice │
├──────────────────────┼─────────────────────────────────────────────┤
│ OPT │ Optional │
├──────────────────────┼─────────────────────────────────────────────┤
│ STR (zero-or-more) │ ZeroOrMore │
├──────────────────────┼─────────────────────────────────────────────┤
│ PLS (one-or-more) │ OneOrMore │
├──────────────────────┼─────────────────────────────────────────────┤
│ AND, NOT │ lookahead annotations │
├──────────────────────┼─────────────────────────────────────────────┤
│ DOT │ Terminal("any") │
└──────────────────────┴─────────────────────────────────────────────┘

So the entire emitter becomes one dispatch on the tag with one railroad.* constructor per case -- the same shape gen_svg.convert_node has today, just shorter and with no dialect mismatch to paper over.

What this replaces

• peg_to_peg.py (gone -- spec PEG is the input directly)
• the vendored pegen/ package (gone)
• the tokenize.generate_tokens(...) step in filter_railroad.py (gone -- this was where the line-flatten bug lived)
• most of gen_svg.convert_node's pegen-AST-walking branches (rewritten as a tag dispatch on pe's Operator)

Kept: the pandoc-filter shell (filter_railroad.py), the railroad.py rendering library, the layout/sizing logic in gen_svg.py.

Caveats I hit during verification

1. The spec is not pure standard PEG -- it uses [U+0020] and [U+0020-U+007E] for Unicode codepoints, which Bryan Ford PEG does not define. pe rejects these. You'd need a tiny preprocessing pass (U+([0-9A-Fa-f]+) ->
   \uXXXX) -- one regex, ~3 lines, declarative, much narrower than what peg_to_peg.py does today. Or upstream a [U+XXXX] extension to pe if you want zero preprocessing.
2. pe's char-class escapes are strict -- it accepts \t \n \v \f \r " ' [ ] \ \xNN \uNNNN \UNNNNNNNN, but not -. The spec puts - at the start ([-0-9a-fA-F]) which works fine; just something to watch for if anyone
   adds - later.

Other libraries I looked at and ruled out

• arpeggio (cleanpeg) -- uses = not <-, and uses r'regex' not [a-z] for char classes. Same dialect-translation problem you have today, just shifted to a different library. (docs)
• parsimonious -- uses = and ~"regex". Same issue. (repo)
• canopy -- compiles grammars to Python source; designed to generate parsers, not to expose an AST you can walk. (repo)
• GrammKit -- exactly the right tool conceptually (PEG.js grammar -> railroad SVG via tabatkins/railroad-diagrams) but it's JavaScript, would need a Node subprocess. (repo)
• TatSu -- EBNF-leaning, not a clean fit.

pe is the only Python library I found whose surface-syntax matches what the spec already writes.

Tradeoffs

• New runtime dependency on pe (small, pure Python, last release Jun 2024 v0.6.0; you'd want to check it's still maintained when you commit to it).
• Need a one-time audit of the four PEG-bearing docs to confirm pe parses every form they use (the U+XXXX thing being the known gap).
• Visual-diff every existing railroad diagram against the new pipeline before cutting over.

Want me to prototype the swap on a doc_build branch -- replacing filter_railroad.py + peg_to_peg.py + the pegen/ import with pe + a small operator-dispatch walker -- and verify it round-trips the four PEG docs to
identical (or visually-equivalent) SVGs?

Sources:

• goodmami/pe -- Python PEG library, native standard-PEG syntax with inspectable AST
• pe operators -- the AST node classes (Sequence, Choice, Optional, Star, Plus, And, Not, Class, Literal, Nonterminal, ...)
• tabatkins/railroad-diagrams (Python port) -- renderer doc_build is already using
• dundalek/GrammKit -- JS reference for "what a PEG-to-railroad pipeline looks like"
• textX/Arpeggio CleanPEG -- ruled out (uses =, regex char classes)
• erikrose/parsimonious -- ruled out (uses =, ~"regex")

❯ Another claude agent reported that argpeggio had a dedicated "classic" ford style grammar fronted. Can you confirm this? Is this the one that you reported "uses = not <-, and uses r'regex' not [a-z] for char classes"?

● Confirmed -- and I was partly wrong

The other agent is right that arpeggio has two PEG frontends, and one of them is Ford-style:

┌──────────────────────────┬─────────────────────────────────────────┬─────────┬────────┬────────────┬─────────────────┐
│ Frontend │ Import │ Rule op │ Choice │ Char class │ Rule terminator │
├──────────────────────────┼─────────────────────────────────────────┼─────────┼────────┼────────────┼─────────────────┤
│ Standard PEG (Ford 2004) │ -- │ <- │ / │ [a-z] │ none │
├──────────────────────────┼─────────────────────────────────────────┼─────────┼────────┼────────────┼─────────────────┤
│ arpeggio.peg ("classic") │ from arpeggio.peg import ParserPEG │ <- │ / │ r'[a-z]' │ ; (mandatory) │
├──────────────────────────┼─────────────────────────────────────────┼─────────┼────────┼────────────┼─────────────────┤
│ arpeggio.cleanpeg │ from arpeggio.cleanpeg import ParserPEG │ = │ / │ r'[a-z]' │ none │
├──────────────────────────┼─────────────────────────────────────────┼─────────┼────────┼────────────┼─────────────────┤
│ pe │ from pe._parse import loads │ <- │ / │ [a-z] │ none │
└──────────────────────────┴─────────────────────────────────────────┴─────────┴────────┴────────────┴─────────────────┘

So my earlier claim "arpeggio uses = not <-" was wrong -- that's only true of the cleanpeg variant. The classic arpeggio.peg does use <-. I
conflated the two frontends and that other agent caught a real error.

What I was right about: neither arpeggio frontend accepts bare [a-z] character classes. You have to write them as regex matches with an explicit
r'' prefix in both. I just re-checked the meta-grammar in arpeggio/peg.py -- the match rule there is str_match | regex_match | rule_crossref, and
regex_match is the form that uses r"...". Bare [...] is not a token at all.

There's also a second mismatch with the spec for arpeggio.peg: every rule must be terminated with ;, which the spec PEG doesn't write.

What this means for the spec

If you went with arpeggio.peg, you'd still need a preprocessing pass before parsing:

1. [A-Z] -> r'[A-Z]' (regex-prefix every char class)
2. Append ; to every rule line
3. Plus the U+XXXX substitution that any choice will need

That's smaller and safer than today's peg_to_peg.py (no Unicode-lookalike substitutions, no flatten-all-newlines), but it's still some translation
-- mostly because of the char-class issue, which is the most common construct in the spec.

pe remains the only library I've actually verified accepts the spec's PEG verbatim (modulo U+XXXX), because it natively understands [a-z] as a
character class without a regex wrapper. So my recommendation doesn't change, but the reason I rejected arpeggio shifts: it's the char-class
syntax and the rule terminator, not the rule operator.

Sources (re-checked):

• Arpeggio grammars docs -- explicitly mentions both arpeggio.peg (with <- and ;) and arpeggio.cleanpeg (with =)
• arpeggio/peg.py source -- meta-grammar shows rule = rule_name, "<-", ordered_choice, ";" and char classes via regex_match (r"...")
• goodmami/pe -- the library I verified accepts spec PEG natively

❯ Does arpeggio's grammar require a ";" terminator on EVERY rule, or only if multiple rules are fed as input at once?

● Empirically confirmed -- I just installed arpeggio 2.0.3 and tried it: every rule, including the only rule when
there's just one. It's a per-rule terminator, not a separator.

=== 1 rule, no ; === FAILED (Expected '?'|'*'|'+'|...|';' at end)
=== 1 rule, with ; === OK
=== 2 rules, no ; === FAILED (chokes after first rule's body)
=== 2 rules, ; on last === FAILED (same -- first rule still missing ;)
=== 2 rules, ; on every === OK
=== 2 rules, ; on first === FAILED (second rule missing ;)

The error message gives the structural answer: Expected '?' or '*' or '+' or '#' or '&' or '!' or 'r'' or 'r"' or
rule_crossref or '(' or str_match or '/' or ';' -- the ; is what tells the parser "this rule's ordered_choice is
done." Without it the parser tries to keep extending the body into whatever comes next, so it fails somewhere in
or after that rule, regardless of how many rules there are.

This matches the meta-grammar: in arpeggio/peg.py, rule() returns rule_name, LEFT_ARROW, ordered_choice, ";" --
the ; is part of the rule production itself, not a between-rules separator.

So if you went with arpeggio.peg, you'd append ; to every PEG line in the spec (still mechanical, still small),
not just to "rule boundaries" or "between rules." Doesn't change the recommendation -- pe is still the only
library I've found that takes the spec verbatim -- but the other agent's instinct that arpeggio.peg is Ford-style
was right; it's the surrounding requirements (; and regex char classes) that keep it from being a drop-in.

Source:

• arpeggio/peg.py meta-grammar -- rule = rule_name, LEFT_ARROW, ordered_choice, ";" confirms ; is part of every
  rule production
• empirical: arpeggio 2.0.3, six minimal grammars tried in test_arpeggio.py

========================================================================================================================
(Claude session from doc_build)

❯ Our current peg_to_peg.py has some shortcomings - for instance, if fed a text block with multiple rules, separated by newlines, it strips the newlines.
Investigate PyPI modules that would do "proper" standard-peg to pegen grammar conversion; if none exist that do this "out of the box", find one that would aid, and make our solution for robust.

● Findings

No out-of-the-box converter exists on PyPI for standard-PEG → pegen. (GitHub code search returns 0 hits; no PyPI package advertises this.)

Best aid: pe (v0.6.0, 2025-06, MIT, pure Python). It is purpose-built around Bryan Ford's standard PEG — its self-hosted parser explicitly defines Definition <- Identifier '<-' Expression and the spec page says "Any pe
grammar without extended features should parse exactly as a standard PEG grammar." It exposes pe.Grammar.definitions as Dict[str, Definition(op, args)] with a typed Operator enum
(DOT/LIT/CLS/SYM/OPT/STR/PLS/AND/NOT/SEQ/CHC/...) — easy to walk.

Runners-up: Arpeggio ships a peg/cleanpeg metagrammar but peg.py requires a trailing ; per rule (doesn't match the spec's input). parsimonious uses = not <-. PyPI pegen itself is stale (v0.3.0, 2023) — keep vendoring
from CPython.

There is also a separate, well-understood bug: filter_railroad.py:74 does "".join(code.split("\n")) before calling the converter. That alone is what made the two-rule AsciiStart/AsciiContinue block silently fail
recently. It should be removed regardless of what we do with the converter.

Two paths

Path A — minimum-change robustness fix (recommended first): drop the "".join(...) flatten in filter_railroad.py:74, and rewrite peg_to_peg.py as a pe-based parse → walk the typed AST → emit pegen syntax. Removes ~70
lines of regex-and-Unicode-lookalike rewrites; downstream pegen + gen_svg.convert_node stay untouched. Still preserves the ↵ line-break marker, U+xxxx escape, and [...]-as-char-class hacks via the new visitor. Adds one
pure-Python dep (pe).

Path B — bigger rewrite: drop both peg_to_peg.py and the vendored pegen/. Use pe to parse standard PEG directly into its AST, retarget gen_svg.convert_node to dispatch on pe.Operator instead of pegen node types. Cleaner
long-term but ~3 MB of vendored code goes away and every existing diagram needs visual verification.

Tradeoff: A is contained and fixes the reported class of bug; B is the right end state if we expect more grammar churn but is a separate, larger effort that needs a visual-diff audit of all ~190 existing diagrams.

Want me to file this as a beads issue and start on Path A?

❯ The reason why the "".join(code.split("\n")) exists is because: a) the code assumes that the input block only has a single rule, and b) it DOES allow that rule to span multiple lines. I don't want to do ANY pre-parsing of the input, nor make assumptions about many rules may be in it. The entire pre-parsing should be eliminated.

● Understood — eliminate all the pre-parsing, let pe do the work, walk its AST and emit pegen.

Two things in the input aren't standard PEG and need a decision before I start:

1. ↵ line-break marker (U+21B5). It appears mid-rule in the spec and is downstream-meaningful (filter_railroad.py:35 looks for it as a NonTerminal). It is not a legal PEG identifier character, so pe will reject it.
   Options: (a) configure/override pe's identifier production to admit it; (b) keep it textually as a '↵' literal that the AST walker recognises and re-emits as the bare identifier ↵ for pegen. (b) is cleaner — it stays as
   a parsed terminal, no library forking, no input rewrite.
2. U+0041 codepoint syntax. Current converter has special handling for U+([0-9A-Fa-f]{4,8}). This is also non-standard PEG. Same options apply; same recommendation — let pe see it as part of an identifier-like or literal
   token, then transform during emit.

If both of those land in the AST walker rather than as text rewrites, the converter becomes: pe.Grammar parse → recursive emit(definition) per Operator → emit name: rhs lines. No splitlines, no regex line scan, no
assumptions about rule count.

filter_railroad.py:74's "".join(code.split("\n")) also goes — pass code through unchanged.

Want me to file this as a beads task and start on it (with the AST-walker approach for ↵ and U+xxxx)?

❯ What are the most stable / widely used solutions for parsing standard PEG grammars?

● Most stable / widely used Python PEG parsers

Filtered by what actually accepts standard Ford-style PEG (<-, /, [...]):

┌─────────────────────────┬───────────────────────────────────────┬───────┬───────────────────┬─────────────┬─────────────┐
│ Library │ Ford syntax OOTB? │ Stars │ Last release │ Monthly DLs │ Maintenance │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ Arpeggio (arpeggio.peg) │ Yes — dedicated frontend │ 287 │ 2025-09 (v2.0.3) │ ~4.7M │ Active │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ pe │ Yes — is the default dialect │ 50 │ 2025-06 (v0.6.0) │ ~505 │ Active │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ TatSu │ Near-Ford (= not <-) │ 440 │ 2026-03 (v5.18.0) │ ~1.57M │ Very active │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ canopy │ Yes — but it's a code generator │ 434 │ 2026-01 (v3.3.2) │ ~28K │ Light │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ pegen (PyPI) │ No (CPython dialect, :) │ 196 │ 2023-11 (v0.3.0) │ ~17K │ Stale │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ Lark │ No (own EBNF, :) │ 5,866 │ 2025-10 │ ~65M │ Very active │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ parsimonious │ No (=, ~"regex") │ 1,912 │ 2025-11 │ ~6.2M │ Light │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ pyparsing │ No (combinator API, not grammar text) │ 2,466 │ 2026-04 │ ~338M │ Very active │
├─────────────────────────┼───────────────────────────────────────┼───────┼───────────────────┼─────────────┼─────────────┤
│ pyPEG2 │ Partial (Python-class DSL) │ — │ 2015 │ ~75K │ Dead │
└─────────────────────────┴───────────────────────────────────────┴───────┴───────────────────┴─────────────┴─────────────┘

Sources: PyPI JSON (e.g. https://pypi.org/pypi/pe/json, https://pypi.org/pypi/Arpeggio/json), pypistats (https://pypistats.org/api/packages//recent), GitHub REST (https://github.com/goodmami/pe,
https://github.com/textX/Arpeggio, https://github.com/neogeny/TatSu, https://github.com/lark-parser/lark).

The honest picture

If "stable / widely used" is the dominant criterion, the top three Python parser libraries overall are Lark, pyparsing, parsimonious — but none of them ingest Ford PEG syntax directly; using them would mean writing a
meta-grammar of standard PEG ourselves (the prior chat report sketched ~20 lines for parsimonious).

If "ingests Ford PEG out of the box" is required, the field shrinks to two real options:

1. Arpeggio is the conservative pick: ~10,000× more downloads than pe, larger user base, longer history, ships arpeggio.peg specifically for Ford syntax. Tradeoff: its Ford frontend (arpeggio/peg.py) requires a ;
   terminator on each rule, which the spec doesn't use — so we'd either need to fork the metagrammar (small change) or feed it cleanpeg (= instead of <-, also wrong). Generic parse-tree + visitor, more boilerplate to walk.
2. pe is the natural fit: standard PEG is the default dialect, no terminator required, typed Definition(op, args) AST with an Operator enum that walks cleanly. Tradeoff: niche library — 50 stars, ~500 monthly downloads,
   single maintainer. If pe goes unmaintained, we'd need to swap libraries; the AST-walker code itself wouldn't change much.

Recommendation

Given the converter is small and isolated, pe remains the best functional fit despite the bus-factor concern — arpeggio is more popular but doesn't accept the spec's exact syntax without modification, which puts us back
in the business of pre-parsing.

If bus factor is the dealbreaker, the realistic alternative isn't arpeggio — it's writing a ~20-line PEG meta-grammar and feeding it to lark or parsimonious. That's still strictly less ad-hoc than the current
peg_to_peg.py and uses libraries with millions of downloads.

✻ Baked for 6m 13s

❯ What parsing solution does github.com/PixarAnimationStudios/OpenUSD use for parsing PEG grammars? What grammar "flavor" does it use? (ie, does it use "clossing Ford PEG" style grammar, or something else?) Presumably it's using some c / c++ library; is there a python wrapper or binding available for whatever library this is?

● OpenUSD's PEG parsing

Library: PEGTL (taocpp), vendored at pxr/base/pegtl/ (v3.2.7, with TAO_PEGTL_* macros renamed to PXR_PEGTL_*). Used by multiple parsers under
pxr/usd/sdf/:

• pathParser.{h,cpp} — SdfPath syntax
• textFileFormatParser.{h,cpp} — .usda text format
• pathPatternParser.h, predicateExpressionParser.h, variableExpressionParser.cpp, booleanExpressionParsing.cpp

Sources: https://github.com/PixarAnimationStudios/OpenUSD/tree/dev/pxr/base/pegtl,
https://github.com/PixarAnimationStudios/OpenUSD/blob/dev/pxr/usd/sdf/pathParser.h, https://github.com/taocpp/PEGTL.

Grammar "flavor": PEGTL grammars are not a text DSL — they are nested C++ template structs:

struct Slash : PEGTL_NS::one<'/'> {};
struct PrimName : Utf8Identifier {};
struct Number : PEGTL_NS::sor<...> {};

There is no .peg / .grammar text file anywhere in the OpenUSD repo. The grammar lives entirely in C++ headers. So the AOUSD spec's Rule <- expr /
expr notation is a separate, hand-authored, human-readable normative re-statement — not source-compatible with the actual PEGTL parser. The two
are kept in sync by hand. The AOUSD spec's flavor is closest to Ford's original PEG paper (and incidentally matches cpp-peglib's text dialect,
which is a different library that PEGTL is not).

Python bindings:

• USD itself: yes, via boost::python (pxr package) — but that exposes parsed objects (Sdf.Path, Sdf.Layer, etc.), not the PEGTL machinery
  underneath.
• PEGTL itself: no Python bindings exist. It's a header-only C++ template library; Python wrappers would require either binding each grammar
  instance individually (PEGTL grammars are types, not data) or reimplementing the PEG combinator surface in Python. The taocpp project has not
  shipped one and PyPI has nothing for it.

Implication for our use case

OpenUSD is not a source we can borrow a parser from — its grammar isn't text and it has no PEG-text frontend. The AOUSD spec's grammar fragments
are an independent artifact, written in a Ford-PEG flavor that needs its own parser on the Python side. Among the options we've evaluated, the
closest text-flavor match for what's already in the spec is pe (Ford syntax exactly) or cpp-peglib (same text flavor but it's C++, not Python).
None of OpenUSD's stack helps here.