feat: support script-driven generators

[gennaroprota]

This adds the second flavor of addon-defined output generator, complementing the data-driven Handlebars generators from #1197: a generator backed entirely by a user Lua or JavaScript script, with no C++ and no compilation. The script owns the whole emit loop through a generate(corpus, output) entry point, so it can produce output shapes the per-page generators cannot, such as a single artifact aggregated across every symbol (a search index, for example).

The design is recorded in the #1216 design comment.

Changes

• Source: A new src/lib/Gen/script/ module: ScriptGenerator (a Generator whose build() owns the emit loop and invokes the script's generate(corpus, output)), OutputSink (the file-writing API exposed to the script as output.write, resolved under the output directory and forbidden from escaping it), and the Lua and JavaScript runners. The manifest parser and the addon-directory walk are factored into a shared src/lib/Gen/GeneratorManifest.{hpp,cpp}, so the data-driven and script-driven discovery passes read the same mrdocs-generator.yml; src/lib/Gen/hbs/DataDrivenGenerators.{cpp,hpp} now consume it and skip directories that declare a script: entry. script::discoverScriptGenerators is wired into src/tool/GenerateAction.cpp and the test runner alongside the data-driven pass. The generator option's help text in src/lib/ConfigOptions.json now mentions the script-driven flavor. A latent Lua-bridge bug is fixed in src/lib/Support/Lua.cpp: domValue_push now marshals Undefined (to nil) and SafeString (to a string) rather than aborting, so reading an absent field (for example the global namespace's empty name) no longer crashes a Lua script.
• Tests: A unit suite drives both runners against a synthetic corpus, exercises the output writer's path safety (writes under the root; rejects absolute and escaping paths), and checks that discovery installs a ScriptGenerator. A regression test reads a symbol with no name field, covering the Undefined-to-nil marshalling that a real corpus needs.
• Documentation: A new "Script-driven generators" section in docs/modules/ROOT/pages/generators.adoc, and the regenerated docs/mrdocs.schema.json.
• Breaking changes: None. The script-driven flavor is additive; data-driven discovery and output are unchanged.

Testing

• src/test/lib/Gen/script/ is the unit-level coverage: discovery, the output writer's path-escaping guards, and both the Lua and JavaScript runners producing the expected aggregated file.
• A script-driven generator's output is an arbitrary aggregated tree, which the per-file golden harness (one expected file per source) cannot check; the mechanism is therefore covered by the unit suite rather than by goldens.
• The Undefined-to-nil regression test guards the Lua-bridge fix, which also benefits the feat: support Lua and JavaScript extensions #1196 corpus-transform scripts that read symbol fields.
• No CI workflow changes are needed.

Documentation

• A new "Script-driven generators" section in docs/modules/ROOT/pages/generators.adoc covers the generate(corpus, output) entry point, the output.write API, the script: discovery marker, and a search-index example.
• The generator option's help text (src/lib/ConfigOptions.json) and the generated docs/mrdocs.schema.json now mention script-driven generators.

[github-actions]

🧾 Changes by Scope

Scope	Lines Δ%	Lines Δ	Lines +	Lines -	Files Δ	Files +	Files ~	Files ↔	Files -
🛠️ Source	77%	1181	1024	157	14	9	5	-	-
🧪 Unit Tests	18%	278	278	-	2	1	1	-	-
📄 Docs	5%	72	71	1	2	-	2	-	-
Total	100%	1531	1373	158	18	10	8	-	-

Legend: Files + (added), Files ~ (modified), Files ↔ (renamed), Files - (removed)

🔝 Top Files

• src/test/lib/Gen/script/ScriptGenerator.cpp (Unit Tests): 270 lines Δ (+270 / -0)
• src/lib/Gen/GeneratorManifest.cpp (Source): 195 lines Δ (+195 / -0)
• src/lib/Gen/hbs/DataDrivenGenerators.cpp (Source): 182 lines Δ (+31 / -151)

Generated by 🚫 dangerJS against 75e8432

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.12%. Comparing base (224c795) to head (75e8432).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1218   +/-   ##
========================================
  Coverage    82.12%   82.12%           
========================================
  Files           33       33           
  Lines         3149     3149           
  Branches       734      734           
========================================
  Hits          2586     2586           
  Misses         387      387           
  Partials       176      176           

Flag	Coverage Δ
bootstrap	82.12% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cppalliance-bot]

An automated preview of the documentation is available at https://1218.mrdocs.prtest2.cppalliance.org/index.html

If more commits are pushed to the pull request, the docs will rebuild at the same URL.

2026-06-03 13:29:05 UTC

[alandefreitas]

Nice! Let me split this review into two points:

On the contribution

• In the documentation example, in what directory would output.write write?
• How can the user see the configuration values and so on?
• How can the extension receive its own parameters?
• The extensions model we had envisioned (and discussed a lot in the last PR) is that a single extensions would contain all kinds of functionalities. The entry point is described as generate (in the same file, right?) but I don't see the word "generate" anywhere in the example script of the documentation. Why is that? Is it just my lack of skills with lua? The transform_corpus example does say function transform_corpus.
• As a side note, could you please put this new documentation content on a new page? I'll reorganize it later in nav.adoc. I'm working on documentation right now, and this will have its own page.

On the use of AI

I think this is a little out of hand, to be honest. 😅 And it's creating a bottleneck rather than being useful at this point.

The design, and how each of its open questions was resolved, is recorded in the #1216 design comment

I don't find this very useful at all. Let me try to explain what I find useful first:

The way to make contributions more useful and less extractive is by having a human in the loop, adding human entropy, and being able to quickly answer questions about the human decisions. The https://llvm.org/docs/AIToolPolicy.html explains this well. Even the PR summary becomes less and less useful if there's no human in the loop, because it's harder and harder to read without intentionality. Only humans know what decisions were made by humans and why. Letting AI do it because AI doesn't know what was on the human's mind. AI doesn't know what you see as important or not. And the reviewer won't know it if AI generates the report. So that's the only part of the description that is actually useful to the contribution. Everything else could be generated by the reviewer just asking Claude to do it the same way. The reviewer was getting a better reply because Claude would make the answer according to the preferences of the reviewer.

I think the word loop here is also very important. When asked something like "why did you do that", humans operate on improving the design of things that are highest ROI. Or, at the very least, humans would answer with the things that have the highest ROI first. AI doesn't know what has the highest ROI to humans, so it tends to create very long answers that are completely inappropriate for the audience, doesn't use the internal vocabulary, includes metaphors, doesn't order things by internal importance, or any of that. AI answers are low ROI: return is low because they don't answer what's important first and how to phrase it to the audience, because they can't, and investment is high because you have to read this very long text that gives you no information.

And, of course, there's value in the human taking accountability over the code that was generated. Because without that, the "contribution" becomes about throwing this work on the reviewer. The human should take accountability for the entire code being reviewed and be able to answer questions with a human response immediately and with confidence. It's very hard to pin down a single metric for that because it's very easy to perceive it when it's not there. If that's not the case, and the answers also come from AI, then there's no contribution here because, again, the reviewer could just ask AI the question directly and avoid the friction.

I know we're excited about AI because it generates a lot of code for us, but trying to bypass the human contribution and answering things with AI only creates more bottlenecks. We have 8 open PRs right now because code generation is faster, but CI and reviews are the bottleneck. We can't skip review; we can't merge code that's not good in this project, since it's not a new project. Even if we trust AI, we still have the specification to check. So only humans in the loop can break this bottleneck by helping prioritize with human entropy. Otherwise, we'll just have more and more blocked PRs, and IA will not help us move things forward.

Of course, AI can still help us with text-related tasks. As I type this, Grammarly is making grammar suggestions. But this is the gist of what I'm saying is human. I often also ask Claude to reorganize text into sections differently, and so on. But the content and its ROI are still accountable, reviewed, and aligned with the vocabulary we use internally.

For instance, AI fluff paragraphs like:

D3 - One context, one call. Build the script context once, evaluate the script once, and call generate once. This matches corpus transforms (a single whole-corpus call), not helpers (many per-node calls), so the existing single-shot extension machinery applies almost unchanged.

just wastes me half a minute to read without understanding. Then, it takes much more time because I have to reverse engineer these distant AI catch phrases ("One context, one call"), metaphors (AI never writes things descriptively, which is very inappropriate in technical contexts), expressions that are not part of the project vocabulary ("single-shot extension"), and these unnecessarily long and complex sentence hierarchies. The whole document would take 15 minutes to read, and it's mostly fluff before getting into the PR. And content-wise, I'm sure no human would say this is high ROI content in the context of this PR. And the way AI works (predicting the next word retroactively) makes it the opposite of a design document because it's retroactively justifying the decisions. And even if the text were good, it doesn't reflect our internal vocabulary at all, which is really weird. And even if all of that were false, the lack of human entropy (to communicate what's on the human's mind) would still mean the reviewer could get a better version of it by asking Claude locally.

And the document is about 50x longer. Meaning it's all this pain times 50. We know that, in practice, most people read the first 10 words, notice it's AI, and ignore it completely. We need a human in the loop somewhere.