Reproducible twin-validation reports: empty-context guard (done), arm keying, report-data export, ergonomics

[johnjosephhorton]

Tracking issue for making the twin-validation report reproducible for any study (from a dogfooding session on Pew W154 microdata). Full plan in the driving study's REPRODUCIBILITY_PLAN.md.

Correctness guards

• F8: guard multi-heldout empty context — --heldout-questions <all items> silently produced covariate-only twins (empty observed_answers) with no warning, a different/weaker experiment. Now errors with a --allow-empty-context opt-in. (done: branch feat/report-data-and-guards)
• twin-validate collapses arms by model label — twin_bootstrap.py:_group_rows keys predictions by (model_label, question), so two jobs with the same model but different strategy/context merge and can't be separated. Key by job_id/approach.
• Lead outputs with skill scores — mostly already true (--metric defaults to nll); optional: default to nll_vs_empirical.

Structured report-data export

• twin-validate --report-data report_data.json emitting: skill scores (per model×question×reference, with CIs), baseline diagnostics (features, XGBoost hyperparameters, training rows, skill vs uniform), per-question marginals (real vs twin vs baseline), and example predictions. Today all of this must be reconstructed by hand from bootstrap.json + raw predictions + reading twin_baseline.py + parsing zwill_report.html.

Ergonomics

• Separate progress from JSON on run commands (edsl-run prints Initializing.../Progress: to stdout before the JSON envelope, so json.load(stdout) fails).
• Standardize comma-separated multi-value flags (--jobs, --models, --heldout-questions).

Also

• First-class "hold each item out in turn, others as context" mode (currently requires hand-scripting one export per item).

The F8 guard is landed; the rest are in progress on the same branch.