feat(test): expose the platform test run's report as a retrievable artifact

[blarghmatey]

What are the relevant tickets?

N/A — follow-up to the review of #111, where a --junitxml flag was written inside the container but unreachable (the container was evaluated for stdout(), so no File/Directory was ever returned) and was removed rather than shipped broken.

Description (What does it do?)

platform test returned pytest stdout and nothing else, so a run's result was unarchivable and — more importantly — unattributable. Because a plugin that ships no tests collects nothing and never fails the run, "this plugin's suite passed" and "this plugin shipped no suite" are indistinguishable in the exit code. That distinction is exactly what tells us whether a plugin's [tests] extra has landed upstream yet (see mitodl/open-edx-plugins#830), and it is knowable only after collection.

• The run now always writes a report. The in-container driver passes --junitxml, then parses the result and writes report.xml, summary.json and summary.md to /openedx/reports. summary.json attributes every test case to a target — edx-platform, or the specific plugin package it came from — and names contributing_plugins / silent_plugins.
• New platform test-report function (lehrer build test-report) returns that directory instead of stdout, so ... export --path ./reports works. It runs the pytest exec with expect=ReturnType.ANY on purpose: a report you can only retrieve when everything passed is useless — a failing suite is precisely when you want it.
• test keeps its exit-code-as-gate contract, unchanged: no expect= override, so a non-zero pytest run still fails the dagger call. It also echoes the summary table at the end of its stdout.
• Both entry points share one _prepare_test_run helper, so the artifact can never describe a different run than the gate executed.

Two supporting cleanups, both load-bearing for the above:

• The two pytest paths (with/without --include-plugins) are folded into the single driver script. That is what makes the report unconditional rather than a privilege of the plugin-inclusive path. With --no-include-plugins the script degrades to a plain edx-platform run and now stays silent about plugins instead of printing a misleading no installed plugin packages to scan.
• test now resolves its cell via the existing _resolve_cell, dropping a hand-rolled duplicate of the resolution that check_deployment / verify_settings / regenerate_aqueduct_settings already share.

The summarizer lives in src/lehrer/core/junit_report.py — stdlib-only, no Dagger imports — and is injected into the container as a file, so the code that runs next to pytest is the very same code the host-side unit tests cover.

How can this be tested?

Automated (15 new tests, uv run pytest — 230 pass):

• tests/core/test_junit_report.py covers the summarizer: attribution by classname, unattributable cases falling back to edx-platform rather than vanishing from the totals, plugins that collected nothing still getting a row, and the JSON/Markdown rendering.
• tests/core/test_plugin_tests.py executes the generated driver for real in a subprocess against a stub pytest that writes a known JUnit XML — the report tail only runs after pytest.main() returns, so reading the source can't check it. This asserts the artifacts are written, that a failing exit code survives summarizing (a bug here would launder a red suite into green), and that an unparseable report prints REPORT SUMMARY FAILED without turning a passing suite red.

Manual:

dagger functions                        # module loads
dagger call platform test-report --help # function registered, Directory return

A real end-to-end run needs a full platform image build (tens of minutes) and was not run here; the container-independent half is covered by the subprocess test above.

lehrer build test-report --cell mit-ol/master/mitxonline \
  --custom-settings ./deployments/mit-ol/settings \
  export --path ./reports

Additional Context

Reviewer questions worth weighing:

1. expect=ANY on test-report. This deliberately swallows pytest's exit code — the function's whole contract is "give me the artifact regardless". A caller who wants both a gate and an artifact runs test and test-report as separate steps. Flagging it because it's the one place a failing suite does not fail the call.
2. No CI job runs platform test yet, so nothing publishes this today. That's intentional scope: adding a full test tier to CI is its own decision (wall-clock and cost), and this PR only removes the blocker that made it impossible to archive results.
3. python -c is retained rather than running the driver as a file: it keeps sys.path[0] as the empty string, so pytest and edx-platform's conftests resolve imports relative to the /openedx/edx-platform workdir. Running a script file would put the driver's own directory there instead.

[Copilot]

Pull request overview

Adds exportable JUnit and per-plugin reports for platform test runs.

Changes:

• Generates XML, JSON, and Markdown test reports.
• Adds platform test-report and CLI forwarding.
• Adds summarizer and driver tests.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/lehrer/core/junit_report.py	Summarizes JUnit results by target.
src/lehrer/core/plugin_tests.py	Generates reports after pytest.
src/lehrer/core/platform.py	Adds report-returning test API.
src/lehrer/cli/build.py	Exposes test-report.
tests/core/test_junit_report.py	Tests report summarization.
tests/core/test_plugin_tests.py	Tests generated driver reporting.
README.md	Documents the command.
deployments/mit-ol/build.md	Documents report export and contents.