Implement semantic predicate accountability hooks

.conformance-review/rust-test-stg-honest-reference-gap.md

@@ -0,0 +1,126 @@
+# `Rust.test.stg` — honest reference & runtime-gap analysis
+
+> **Migration status (July 2026): the render-then-compile pipeline is
+> implemented and is the harness's only pipeline — the legacy
+> template-recognition path (pattern-matching descriptor markup and
+> simulating its output) has been deleted.** `antlr4-runtime-testsuite`
+> renders each descriptor grammar through `Rust.test.stg` with the real
+> StringTemplate engine (`tools/stg-render/RenderGrammar.java` via the
+> ANTLR jar) and generates with `antlr4-rust-gen --actions embedded`,
+> which splices the
+> rendered Rust action/predicate bodies verbatim after `$`-attribute
+> translation (`src/bin_support/embedded.rs` — the Rust analog of ANTLR's
+> `ActionTranslator`). The four capability axes below are now generated:
+> an output sink (`self.output()`), typed context views with positional
+> accessors and public attribute fields (`FromRuleContext` downcast),
+> typed attrs snapshots replacing the int-only `int_return` map, and
+> `@members` as real struct fields / impl items. Listener traits and a
+> typed walker bridge cover the listener suite. Validating the reference
+> against the real ST engine also surfaced blind-authoring defects; each
+> correction is documented in `Rust.test.stg.design-notes.md` under
+> "Reference corrections". The historical analysis below is preserved
+> as-written.
+
+Companion to `Rust.test.stg` and `Rust.test.stg.design-notes.md` in this
+directory. This document records **how** the reference `.stg` was produced and
+**what diffing it against our actual runtime reveals**.
+
+## What a `.test.stg` is, and why we want an honest one
+
+ANTLR's `runtime-testsuite` descriptors are **target-neutral**: their grammars
+contain StringTemplate calls like `<writeln("$e.v")>`, `<Cast("BinaryContext",
+"$ctx")>`, `<SubContextLocal(...)>`. Before generating a parser, ANTLR renders
+the descriptor grammar through the *current target's* `templates/<Lang>.test.stg`
+group — a pure `.g4`-text → `.g4`-text StringTemplate render
+(`new ST(group, descriptor.grammar).render()`, `RuntimeTests.prepareGrammars`).
+The rendered `.g4` then contains real, target-language action code that the
+target's code generator compiles. Every official target (Java, Go, Python3,
+C#, C++, Swift, Dart, TypeScript, JavaScript, PHP) ships one such file.
+
+We have none. Our conformance harness instead **recognizes a subset of template
+markup inside the generator** (`src/bin_support/templates.rs`) rather than
+rendering the `.stg` into `.g4`. Producing a real `Rust.test.stg` is the
+framework-blessed path — but only if it is *honest*: it must describe an ideal,
+idiomatic Rust target, not be reverse-engineered from whatever our current
+runtime happens to expose (that would re-introduce fixture-fitting).
+
+## How this reference was produced (blind methodology)
+
+`Rust.test.stg` + `Rust.test.stg.design-notes.md` were authored by a fresh agent
+**deliberately blind to this runtime**. It was allowed to read only the other
+targets' `.test.stg` files (Java as the canonical/complete reference; Go as the
+reference for getter/exported-name conventions) plus idiomatic Rust. It was
+**forbidden** from reading `src/`, the code-generation template
+(`tool/resources/org/antlr/v4/tool/templates/codegen/Rust/Rust.stg`), any
+generated parser, or `src/bin_support/templates.rs`.
+
+The point: force a specification of what an *ideal* Rust ANTLR target should
+expose, uncontaminated by our current shortcuts. The design notes therefore
+include a 21-capability **"Runtime API surface this assumes"** checklist — a
+ready-made gap backlog — plus an honest "least certain" section flagging the
+five renderings most likely to diverge from any real runtime.
+
+The result covers all **70** templates with signatures matching `Java.test.stg`
+exactly (it even caught that a hand-listed checklist of 69 omitted `StringType`).
+
+## What diffing the ideal against our runtime reveals
+
+Examining the blind reference against what we actually generate (from a kept
+`target/antlr-runtime-testsuite/LeftRecursion_MultipleAlternativesWithCommonLabel_1`
+work dir) shows a **systematic, four-axis divergence**:
+
+| The ideal `.stg` assumes | Our runtime actually does |
+| --- | --- |
+| Output via an injected `self.output()` sink (`&mut dyn Write`) | `println!` to **stdout** |
+| Typed context structs (`EContext`, `BinaryContext`) with positional child accessors `ctx.e(0)` / `ctx.e_all()` | **Zero** context structs; `fn e()` is a *rule method* returning `ParseTree` |
+| Rule return attributes as public fields (`ctx.v`) | Values kept in a runtime `int_return` map keyed by rule-index (`src/tree.rs`) |
+| `.downcast_ref::<BinaryContext>()` for labeled-alternative access | No labeled-alt context types to downcast to |
+
+### Architectural crux: render vs. recognize
+
+ANTLR renders `descriptor.grammar` through the `.stg` into a `.g4` full of real
+target code — no dependency on the generated parser. Our harness does **not**
+render: the `.g4` it writes still contains raw `<Cast(...)>` / `<writeln(...)>`
+markup, and the *generator* pattern-matches template bodies. So "use
+`Rust.test.stg` the ANTLR way" means render `.stg` → `.g4` **and then compile
+it** — which our generator cannot do today, because it emits none of the four
+capabilities above.
+
+## Implications
+
+- **Committing this as a reference/spec** (what this directory is) is cheap and
+  valuable: it is the honest definition of an ideal Rust target plus a concrete,
+  itemized gap backlog.
+- **Actually using it during conformance runs the ANTLR way** is a large,
+  multi-phase migration: add a StringTemplate render step to the harness *and*
+  grow the generator to emit the assumed API (output sink, typed context
+  structs, positional accessors, downcast). It is effectively the work of
+  "becoming a real ANTLR Rust target," which is a much larger scope than the
+  original goal (making `--sem-unknown=error` a viable default).
+- **Trap to avoid:** adapting the `.stg` to render into our *current* idiom
+  (`println!`, `int_return` map) so it "works" today. That re-introduces exactly
+  the fixture-fitting removed when the `RuleValue` re-evaluator was deleted. The
+  reference must stay honest.
+
+## The five least-certain renderings (most likely to diverge)
+
+From the design notes, ranked by risk — these are where a real Rust target is
+most likely to need bespoke work:
+
+1. **`RuleInvocationStack`** (and debug-formatted list prints): Rust `{:?}` on
+   `Vec<String>` yields quoted `["a", "b"]`, not the Java-style `[a, b]` the
+   descriptors byte-compare against. Every other target needed a bespoke
+   formatter (Go `PrintArrayJavaStyle`, Python `str_list`). **Highest byte-match
+   risk.**
+2. **`PositionAdjustingLexerDef` / `PositionAdjustingLexer`**: the entire
+   lexer-override model (`base_next_token`/`base_emit`, swappable interpreter,
+   codepoint char indexing) is invented; a trait/hook-based customization point
+   would reshape it.
+3. **`Result` / `Production`**: bet on public return-attribute fields vs. getters
+   or `Option`-wrapped returns (Go is the one target that diverged to getters).
+4. **`Cast`**: assumes `Any`-style `downcast_ref` on context trait objects; if
+   contexts are enums the idiomatic form is `match`/`if let`.
+5. **Listener structs calling `self.output()`**: inside a listener method `self`
+   is the listener, not the recognizer, so an ideal Rust listener likely needs
+   the output sink injected into its struct (as Java/C# thread a `TextWriter`
+   into `LeafListener`).

AGENTS.md

@@ -72,7 +72,23 @@ The harness reads `antlr4-upstream/runtime-testsuite` and the same ANTLR jar fet
 cargo run --release --quiet --bin antlr4-runtime-testsuite
 ```
 
-Defaults to `ANTLR4_JAR=/tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar` and `ANTLR4_RUNTIME_TESTSUITE=/tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite`. Override with `--antlr-jar`/`--descriptors` or env vars. Expected outcome: `summary: 357 passed, 0 failed, 0 skipped, 357 run`. Wall-clock ≈ 10–15 minutes on Apple Silicon, ≈ 30 minutes on the GitHub Linux runner.
+Defaults to `ANTLR4_JAR=/tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar` and `ANTLR4_RUNTIME_TESTSUITE=/tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite`. Override with `--antlr-jar`/`--descriptors` or env vars. Wall-clock ≈ 10–15 minutes on Apple Silicon, ≈ 30 minutes on the GitHub Linux runner.
+
+### The rendered (embedded-actions) pipeline
+
+The harness runs descriptors the way every official ANTLR target does:
+each descriptor grammar is rendered through
+`.conformance-review/Rust.test.stg` with the real StringTemplate engine
+(`tools/stg-render/RenderGrammar.java`, executed via the ANTLR jar and the
+Java single-file source launcher), so its actions/predicates become real
+Rust code. The rendered grammar feeds both the ANTLR tool and
+`antlr4-rust-gen --actions embedded`, which splices the bodies verbatim
+after `$`-attribute translation (`src/bin_support/embedded.rs`) and
+generates typed context views, per-rule attrs structs, members
+fields/methods, listener traits, and recognizer facades. `--stg PATH`
+overrides the template group. (An earlier template-recognition pipeline,
+which simulated action output instead of executing it, was replaced by
+this one before ever shipping.)
 
 ### Run a subset while iterating
 

CLAUDE.md

@@ -37,6 +37,16 @@ cargo run --release --bin antlr4-rust-gen -- \
 The output crate must depend on this runtime (`antlr-rust-runtime = { path = ... }`).
 Both the kotlin-parity dumper and the parse-bench runner are examples.
 
+Every run also writes a `semantics.json` manifest into `--out-dir` listing each
+semantic predicate/action coordinate and its disposition. `--sem-unknown
+error|hook|assume-true|assume-false`, `--sem-patterns`, and
+`--require-full-semantics` control untranslated coordinates (default
+`assume-true`, deprecated; see the README "Semantic Predicates and Actions"
+section and issue #9).
+Generated parsers emit SemIR tables, `with_hooks(tokens, hooks)`, and typed
+hook adapters for bare helper predicates; lexer callers can route closure hooks
+through `LexerSemCtx` and the shared `SemanticHooks` trait.
+
 ## Kotlin parser parity perf benchmark
 
 Reproduces the timings against the Kotlin grammar from `antlr/grammars-v4`.
@@ -101,7 +111,23 @@ The harness reads `antlr4-upstream/runtime-testsuite` and the same ANTLR jar fet
 cargo run --release --quiet --bin antlr4-runtime-testsuite
 ```
 
-Defaults to `ANTLR4_JAR=/tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar` and `ANTLR4_RUNTIME_TESTSUITE=/tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite`. Override with `--antlr-jar`/`--descriptors` or env vars. Expected outcome: `summary: 357 passed, 0 failed, 0 skipped, 357 run`. Wall-clock ≈ 10–15 minutes on Apple Silicon, ≈ 30 minutes on the GitHub Linux runner.
+Defaults to `ANTLR4_JAR=/tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar` and `ANTLR4_RUNTIME_TESTSUITE=/tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite`. Override with `--antlr-jar`/`--descriptors` or env vars. Wall-clock ≈ 10–15 minutes on Apple Silicon, ≈ 30 minutes on the GitHub Linux runner.
+
+### The rendered (embedded-actions) pipeline
+
+The harness runs descriptors the way every official ANTLR target does:
+each descriptor grammar is rendered through
+`.conformance-review/Rust.test.stg` with the real StringTemplate engine
+(`tools/stg-render/RenderGrammar.java`, executed via the ANTLR jar and the
+Java single-file source launcher), so its actions/predicates become real
+Rust code. The rendered grammar feeds both the ANTLR tool and
+`antlr4-rust-gen --actions embedded`, which splices the bodies verbatim
+after `$`-attribute translation (`src/bin_support/embedded.rs`) and
+generates typed context views, per-rule attrs structs, members
+fields/methods, listener traits, and recognizer facades. `--stg PATH`
+overrides the template group. (An earlier template-recognition pipeline,
+which simulated action output instead of executing it, was replaced by
+this one before ever shipping.)
 
 ### Run a subset while iterating
 

README.md

@@ -228,15 +228,149 @@ See [docs/kotlin-build.md](docs/kotlin-build.md) for the Kotlin smoke workflow.
 See [docs/runtime-testsuite.md](docs/runtime-testsuite.md) for the upstream
 runtime-testsuite harness.
 
+### Semantic Predicates and Actions: the Compatibility Boundary
+
+ANTLR grammars may embed **target-language** semantic predicates and actions
+(`{isTypeName()}?`, `{this.count++;}`). The serialized ATN records *where*
+they occur, but not executable code, so a metadata-first runtime cannot run
+arbitrary grammar-embedded snippets. The boundary is:
+
+- **Target-agnostic grammars** — no embedded code, or only built-in lexer
+  commands (`skip`, `channel(...)`, `mode(...)`, `type(...)`) — are fully
+  supported.
+- **Recognized predicate/action shapes** — a library of common idioms
+  (constant predicates, lookahead text/type checks, integer member counters,
+  column predicates, and the upstream testsuite's action templates) — are
+  translated into SemIR by `antlr4-rust-gen` when the grammar source is
+  passed via `--grammar`.
+- **User pattern files** — `--sem-patterns file.toml` can add exact predicate
+  rewrites, helper-call rewrites, and per-coordinate `hook` /
+  `assume-true` / `assume-false` / `error` dispositions without changing the
+  generator.
+- **Everything else is not silently guessed.** Each generator run writes a
+  `semantics.json` manifest next to the generated modules listing every
+  predicate/action coordinate with its grammar source span, body, and
+  disposition (`translated`, `hooked`, `assume-true`, `assume-false`,
+  `ignored`, `synthetic`, or `error`). A `synthetic` action is one ANTLR
+  inserts itself (e.g. during left-recursion elimination); it has no
+  grammar-author source, is a runtime no-op, and is exempt from the `error`
+  gate — only actions the author actually wrote in the grammar can fail it.
+
+Unknown coordinates are governed by `--sem-unknown`:
+
+```bash
+antlr4-rust-gen --lexer L.interp --parser P.interp --grammar G.g4 \
+    --out-dir src/generated --sem-unknown error
+```
+
+- `assume-true` (current default, deprecated): unknown predicates pass,
+  unknown actions are no-ops — the historical behavior. A future minor
+  release changes the default to `error`.
+- `hook`: unknown parser predicates are routed to `SemanticHooks` and fail if
+  the hook does not handle them.
+- `assume-false`: unknown predicates fail, removing the guarded alternatives.
+- `error`: generation fails, naming each coordinate:
+
+  ```text
+  unsupported semantic predicate: rule=s(0) pred_index=0 at 2:4: {isTypeName()}
+  ```
+
+At runtime the same policy exists as
+`ParserRuntimeOptions::unknown_predicate_policy`
+(`UnknownSemanticPolicy::{AssumeTrue, AssumeFalse, Error}`); under `Error`,
+evaluating an unknown predicate coordinate fails the parse with
+`AntlrError::Unsupported` instead of producing a tree whose shape silently
+depended on a guess.
+
+Generated parsers also expose a parser-side hook escape hatch:
+`MyParser::with_hooks(tokens, hooks)`, where `hooks` implements
+`SemanticHooks`. Unknown parser predicates are offered to
+`SemanticHooks::sempred` before the fallback policy is applied, and unhandled
+parser action events are offered to `SemanticHooks::action` after the committed
+parse path is selected. Predicate hooks may run speculatively during
+prediction, so they must be replay-safe.
+
+For bare helper-call predicates, generated parsers also emit a typed hook
+adapter (`MyParserHooks` plus `MyParserTypedHooks<T>`) that maps stable
+manifest coordinates to named Rust methods. Lexer callers can use
+`LexerSemCtx` with `atn::lexer::next_token_with_semantic_hooks` or the
+compiled-DFA variant to route lexer predicates/actions through the same
+`SemanticHooks` trait.
+
+Use `--require-full-semantics` in CI when every coordinate must be either
+translated or explicitly hooked; policy fallbacks fail generation.
+
+#### Embedded target-language actions are not portable — including in official ANTLR
+
+A grammar that embeds a **target-language** action (a `{ ... }` block of
+Java/C#/etc. code, rather than a portable lexer command) is only usable with
+the language it was written for. This is a limitation of ANTLR itself, not of
+this runtime: **the official ANTLR tool does not translate embedded actions
+between targets — it copies the source text verbatim into the generated code.**
+
+For example, the official
+[Kotlin/kotlin-spec](https://github.com/Kotlin/kotlin-spec) `KotlinLexer.g4`
+contains a Java-only action:
+
+```antlr
+RCURL: '}' { if (!_modeStack.isEmpty()) { popMode(); } };
+```
+
+Generating a **Go** parser from it with the official tool
+(`antlr4 -Dlanguage=Go KotlinLexer.g4`) emits the Java verbatim:
+
+```go
+func (l *KotlinLexer) RCURL_Action(localctx antlr.RuleContext, actionIndex int) {
+	switch actionIndex {
+	case 0:
+		if !_modeStack.isEmpty() { // undefined in Go — does not compile
+			popMode()              // undefined in Go — does not compile
+		}
+	}
+}
+```
+
+The generated Go **fails to compile** (`undefined: _modeStack`, `undefined:
+popMode`), and ANTLR offers no supported way to fix it beyond hand-editing the
+grammar — the grammar even carries a comment telling non-Java users to replace
+the snippet manually. Every non-Java ANTLR target has this gap.
+
+This runtime does better in two ways:
+
+1. It recognizes a **library of common embedded idioms** (e.g. the guarded
+   `popMode()` above) and maps them to the equivalent portable operation, so
+   many real grammars generate as-is.
+2. For anything it does not recognize, `--sem-unknown=error` fails **loudly**
+   at generation time, naming the coordinate, instead of silently emitting
+   uncompilable or no-op code. The fix is to express the action as a portable
+   lexer command (`-> popMode`, `-> pushMode(X)`, `-> type(X)`,
+   `-> channel(HIDDEN)`), add a `--sem-patterns` rewrite, or route it through a
+   `SemanticHooks` implementation.
+
+Portable lexer commands and the recognized idioms are the target-agnostic
+subset; prefer them when authoring grammars intended for multiple runtimes.
+
+Grammars whose `{ ... }` blocks are already **Rust** can skip translation
+entirely: `antlr4-rust-gen --actions embedded --grammar Foo.g4` splices the
+bodies verbatim (after `$`-attribute translation) into the generated parser,
+inline at their ATN action/predicate coordinates. This is the mode the
+conformance harness uses after rendering descriptor grammars through
+`Rust.test.stg` (see below).
+
 ## Runtime Testsuite
 
 On the maintainer checkout, where the ANTLR jar and upstream runtime-testsuite
 live under `/tmp/antlr-cleanroom`, run the full sweep with:
 
 ```bash
-cargo run --quiet --bin antlr4-runtime-testsuite
+cargo run --release --quiet --bin antlr4-runtime-testsuite
 ```
 
+The harness runs descriptors the way every official ANTLR target does: each
+descriptor grammar is rendered through `.conformance-review/Rust.test.stg`
+with the real StringTemplate engine, so its actions and predicates become real
+Rust code that is compiled and executed inline.
+
 Run a specific descriptor:
 
 ```bash