Add the E001 canonical evidence runner#7
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a repository-controlled, reproducible “canonical evidence” generator for E001 Stage A (Python 3.13), plus a dedicated GitHub Actions workflow to run the frozen 72-run matrix twice, validate determinism, and publish a deterministic evidence archive.
Changes:
- Introduces
tools/generate_e001_evidence.pyto run/repeat the E001 matrix, write raw traces + summaries, and emit provenance/manifest metadata. - Adds a GitHub Actions workflow to run the generator on
main, package results into a deterministic.tar.gz, and upload it as an artifact. - Extends the simulator test suite with a new test that exercises the evidence generator end-to-end.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tools/generate_e001_evidence.py | New evidence generator that runs the matrix twice, writes artifacts, and emits provenance/validation metadata. |
| tests/test_e001_simulator.py | Adds a test that validates evidence generation output shape and determinism checks. |
| .github/workflows/e001-canonical-evidence.yml | New workflow to run evidence generation on Python 3.13 and upload a deterministic archive + checksum artifact. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| manifest = matrix_manifest(first) | ||
| manifest["determinism_mismatches"] = mismatches | ||
| manifest["source_commit"] = source_commit | ||
|
|
| configuration_bytes = CONFIGURATION_PATH.read_bytes() | ||
| specification_bytes = SPECIFICATION_PATH.read_bytes() | ||
| (inputs_dir / CONFIGURATION_PATH.name).write_bytes(configuration_bytes) | ||
| _write_json(output / "matrix_manifest.json", manifest) |
| "invocation": ( | ||
| "PYTHONPATH=src python tools/generate_e001_evidence.py " | ||
| f"--output build/e001-canonical --source-commit {source_commit}" | ||
| ), |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0269ace94
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| paths: | ||
| - .github/workflows/e001-canonical-evidence.yml | ||
| - experiments/E001/config/** | ||
| - src/superloop_e001/** | ||
| - tools/generate_e001_evidence.py |
There was a problem hiding this comment.
When a push to main changes only experiments/E001/EXPERIMENT__E001__THREE_RING_BOUNDED_FLOW__v0.1__2026-07-13.md, this paths filter does not start the canonical workflow, even though tools/generate_e001_evidence.py reads and hashes that specification into provenance. That leaves the latest main-branch protocol without a new canonical artifact unless someone remembers to run the workflow manually; include the spec file or a broader experiments/E001/*.md pattern in the push paths.
Useful? React with 👍 / 👎.
| "validation": { | ||
| "determinism_mismatches": mismatches, | ||
| "invalid_run_ids": manifest["invalid_run_ids"], | ||
| "run_count": manifest["run_count"], |
There was a problem hiding this comment.
This records run_count, but the validation below never compares it with expected_run_count/72. If a future change to the matrix constants or run_matrix() accidentally drops or adds runs while the remaining runs are deterministic and invariant-clean, this generator still exits successfully and publishes an archive that is not the complete canonical matrix; add an explicit run-count check before returning.
Useful? React with 👍 / 👎.
What changed
Why
The local workspace provides Python 3.12, while ADR 0001 declares Python 3.13 for E001 Stage A. Canonical evidence therefore needs a repository-controlled runner that creates a traceable artifact from an exact main-branch commit.
Validation
ca25a9f9030124ded4bd3d6d2d1905ef6d5a095b916767d09f211706f9f82dc5The local checksum is a packaging test only. The canonical checksum will be produced by the Python 3.13 main-branch workflow after this PR merges.