Fix PyPI canonical release recovery

[Fieldnote-Echo]

Summary

• add a canonical Python dist recovery path that uses the current build for new versions and verified PyPI-served files when PyPI already owns an immutable version
• rewire release assets, PyPI publish/verify, smoke testing, and provenance so v0.3.0 recovery can go green without pretending rebuilt wheels are the PyPI files
• hash-pin local wheel/sdist installs and pin the new release shape in invariant tests and docs

Validation

• python3 -m py_compile tests/release_pypi_canonical_dist.py tests/release_pypi_canonical_dist_tests.py tests/release_publish_invariants.py
• python3 tests/release_pypi_canonical_dist_tests.py
• python3 YAML parse for .github/workflows/release.yml and .github/workflows/python.yml
• bash tests/release_publish_invariants.sh
• bash tests/release_signed_release_invariants.sh
• bash -n tests/release_environment_settings.sh
• bash tests/release_environment_settings.sh
• live PyPI 0.3.0 canonicalization probe: downloaded existing files, verified PyPI SHA256 digests, and matched PyPI JSON
• cargo package -p ordvec --locked --list
• cargo publish -p ordvec --dry-run --locked

[qodo-code-review]

Review Summary by Qodo

Implement PyPI canonical dist recovery for immutable version releases

✨ Enhancement 🐞 Bug fix

Walkthroughs

Description

• Add canonical Python dist recovery path for PyPI immutable versions
  - Uses current build for new versions
  - Downloads and verifies PyPI-served files for existing versions
• Implement release_pypi_canonical_dist.py with canonicalize/verify commands
• Rewire release workflow to select canonical dist before attestation
• Update invariant tests to validate new pypi-canonical-dist job
• Hash-pin wheel/sdist installs in smoke tests and build workflows

Diagram

flowchart LR
  build["build-wheels<br/>build-sdist<br/>build-crate"]
  canonical["pypi-canonical-dist<br/>canonicalize"]
  attest["attest<br/>provenance"]
  release["release-assets-draft"]
  publish["publish-pypi<br/>publish-crate"]
  
  build -->|raw artifacts| canonical
  canonical -->|canonical dist| attest
  canonical -->|canonical dist| release
  attest --> release
  release --> publish
  canonical -->|source flag| publish

Loading

File Changes

1. tests/release_pypi_canonical_dist.py ✨ Enhancement +222/-0

New canonical Python dist selection module

• New module implementing canonical PyPI dist handling for release workflow
• canonicalize() command: uses current build for new versions, downloads/verifies PyPI files for
 existing versions
• verify() command: polls PyPI JSON endpoint to confirm hashes match canonical dist
• Helper functions for SHA-256 computation, PyPI payload fetching, and file validation
• Supports recovery mode where PyPI already owns immutable version bytes

tests/release_pypi_canonical_dist.py

---

2. tests/release_pypi_canonical_dist_tests.py 🧪 Tests +116/-0

Unit tests for canonical dist module

• New unit test suite for release_pypi_canonical_dist.py
• Tests missing PyPI release uses current build
• Tests existing PyPI release uses verified remote bytes
• Tests filename drift rejection between local and remote
• Mocks PyPI payload fetching for deterministic testing

tests/release_pypi_canonical_dist_tests.py

---

3. tests/release_publish_invariants.py ✨ Enhancement +111/-47

Add canonical dist job validation to invariants

• Add check_pypi_canonical_dist() function to validate new workflow job
• Verify job needs both build-wheels and build-sdist
• Validate canonicalize step reads built-dist and writes canonical-dist
• Ensure exactly one pypi-canonical-dist artifact upload
• Update check_publish_pypi() to require pypi-canonical-dist dependency
• Change publish-pypi to download canonical dist instead of raw artifacts
• Add verification step validation for PyPI hash checking
• Add helper functions has_need() and contains_text()

tests/release_publish_invariants.py

---

View more (5)

4. tests/release_signed_release_invariants.sh 📝 Documentation +26/-10

Update release graph invariants for canonical dist

• Update release graph documentation to include pypi-canonical-dist job
• Clarify that raw artifacts are distinct from canonical dist
• Note that attestation/provenance are crate-only in recovery mode
• Add validation that pypi-canonical-dist runs canonicalize command
• Add validation that publish-pypi consumes canonical dist artifact
• Update publish-pypi validation to check for verify command instead of inline hash check

tests/release_signed_release_invariants.sh

---

5. .github/workflows/python.yml ✨ Enhancement +20/-1

Hash-pin Python package installations

• Replace direct wheel install with hash-pinned installation
• Generate SHA-256 digest of wheel and create requirements file
• Use --require-hashes --no-index --no-deps flags for verified install
• Apply same pattern to sdist installation

.github/workflows/python.yml

---

6. .github/workflows/release.yml ✨ Enhancement +200/-92

Integrate canonical dist selection into release workflow

• Add new pypi-canonical-dist job that canonicalizes Python dist
• Job downloads wheels and sdist, runs canonicalize command, uploads canonical dist
• Outputs source flag indicating 'build' or 'pypi' origin
• Update smoke-linux-aarch64-wheel to depend on pypi-canonical-dist instead of build-wheels
• Update smoke test to download canonical dist and hash-pin wheel installation
• Update attest job to conditionally attest crate-only or all artifacts based on source
• Update combine-hashes to limit SLSA subjects to crate when source is 'pypi'
• Update release-assets-draft to collect canonical dist separately
• Update publish-pypi to skip upload when source is 'pypi' and verify instead
• Replace inline hash verification with call to release_pypi_canonical_dist.py verify
• Update workflow documentation to explain recovery mode behavior
• Hash-pin all wheel/sdist installations throughout workflow

.github/workflows/release.yml

---

7. Cargo.toml ⚙️ Configuration changes +2/-0

Exclude new test files from crate package

• Exclude new test files from crate package
• Add tests/release_pypi_canonical_dist.py and tests/release_pypi_canonical_dist_tests.py to
 exclude list

Cargo.toml

---

8. RELEASING.md 📝 Documentation +32/-16

Document canonical dist recovery in release guide

• Document canonical Python dist selection in release pipeline
• Explain new behavior: current build for new versions, verified PyPI bytes for existing versions
• Clarify that recovery mode limits attestation/SLSA subjects to crate only
• Update publish-pypi description to cover both upload and verification modes
• Expand step 6 to describe canonical dist selection before attestation
• Clarify step 7 behavior for recovery mode verification

RELEASING.md

---

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

1. Fragile /tmp temp path ✓ Resolved 🐞 Bug ☼ Reliability

Description

The wheel install steps in python.yml and release.yml write requirement files to /tmp, even
though those steps run on windows-latest under shell: bash; this relies on Git Bash’s /tmp
mapping and can break if the runner shell/environment changes or /tmp isn’t available/writable.

Code

.github/workflows/release.yml[R310-320]

Evidence

release.yml runs the wheel test step on windows-latest as part of the build matrix, and that
step writes to /tmp. python.yml also includes a Windows job and writes to /tmp in the wheel
install step.

.github/workflows/release.yml[254-320]
.github/workflows/python.yml[53-119]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Workflow steps write temporary requirements files to hardcoded `/tmp/...`, but these steps run on Windows runners too (via `shell: bash`). This is fragile and can break if `/tmp` is not present/writable or if the shell changes.

### Issue Context
This affects both CI (`python.yml`) and the release workflow (`release.yml`) wheel test steps.

### Fix Focus Areas
- .github/workflows/python.yml[97-119]
- .github/workflows/release.yml[296-320]
- .github/workflows/release.yml[349-372]
- .github/workflows/release.yml[420-442]

### Suggested fix
- Replace `/tmp/ordvec-*-requirements.txt` with a path under `${RUNNER_TEMP}` (available on all runners) or with `mktemp`.
- Example (bash):
 - `REQ_FILE="${RUNNER_TEMP}/ordvec-wheel-requirements.txt"`
 - Redirect to `"$REQ_FILE"` and pass `-r "$REQ_FILE"`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Brittle aarch64 wheel match ✓ Resolved 🐞 Bug ☼ Reliability

Description

smoke-linux-aarch64-wheel selects the wheel using the hardcoded glob
*manylinux_2_17_aarch64*.whl, so if the manylinux policy tag changes (e.g.,
manylinux_2_28_aarch64) the job will fail to find a wheel and abort the release pipeline.

Code

.github/workflows/release.yml[R353-356]

Evidence

The smoke job downloads the consolidated pypi-canonical-dist and then uses a fixed
manylinux_2_17_aarch64 substring to find the aarch64 wheel, which will break if the platform tag
changes.

.github/workflows/release.yml[327-359]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The aarch64 smoke test filters wheels using a specific manylinux substring. This is brittle against toolchain/manylinux tag changes and can cause releases to fail even when a valid aarch64 wheel exists.

### Issue Context
The smoke job now downloads `pypi-canonical-dist`, which can contain multiple wheels; selection must be robust but not over-specific.

### Fix Focus Areas
- .github/workflows/release.yml[327-360]

### Suggested fix
- Use a version-agnostic match such as `*aarch64*.whl` and then assert exactly one match.
- If multiple matches are possible in the future (e.g., manylinux + musllinux), add a small Python filter that prefers `manylinux` (or matches the platform tag more formally) rather than hardcoding `manylinux_2_17`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[gemini-code-assist]

Code Review

This pull request introduces a canonical PyPI distribution handling mechanism for the release workflow, allowing the recovery path to download and verify immutable files already published on PyPI instead of relying on raw rebuilt artifacts. It adds a new script release_pypi_canonical_dist.py along with unit tests and updates the release invariants and documentation. The review feedback focuses on improving the robustness of the new script against transient network or CDN propagation errors. Specifically, the reviewer suggests raising exceptions instead of calling fail() immediately in fetch_pypi_payload and pypi_dist_map so that the retry loop in verify() can function as intended, with corresponding updates to canonicalize and verify to handle these exceptions gracefully.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!