Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 177 additions & 9 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,12 @@ jobs:
SHA: ${{ github.sha }}
run: |
set -euo pipefail
MAIN_SHA="$(gh api "repos/${REPO}/commits/main" --jq '.sha')"
echo "current main sha: ${MAIN_SHA}"
if [ "$SHA" != "$MAIN_SHA" ]; then
echo "::error::release tag points at ${SHA}, but current main is ${MAIN_SHA}. Move the tag to the current protected main HEAD and re-run."
exit 1
fi
# Require a SUCCESSFUL push run for this SHA *on main* for each workflow.
# Filtering on branch as well as head_sha stops a green run for the same
# commit on an unrelated branch from satisfying the gate.
Expand Down Expand Up @@ -698,7 +704,13 @@ jobs:
python3 tests/release_pypi_canonical_dist.py canonicalize \
--version "$VERSION" \
--built-dir built-dist \
--out-dir canonical-dist
--out-dir canonical-dist \
--expected-wheels 4 \
--expected-sdists 1 \
--required-wheel-tag x86_64 \
--required-wheel-tag aarch64 \
--required-wheel-tag macosx \
--required-wheel-tag win_amd64
- name: Upload the canonical Python dist
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down Expand Up @@ -745,7 +757,13 @@ jobs:
--project ordvec-manifest \
--version "$VERSION" \
--built-dir built-dist \
--out-dir canonical-dist
--out-dir canonical-dist \
--expected-wheels 4 \
--expected-sdists 1 \
--required-wheel-tag x86_64 \
--required-wheel-tag aarch64 \
--required-wheel-tag macosx \
--required-wheel-tag win_amd64
- name: Upload the canonical ordvec-manifest Python dist
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down Expand Up @@ -1058,12 +1076,96 @@ jobs:
exit 1
fi
echo "OK: byte-identity verified ($A_SHA)"
- name: Check for existing ordvec .crate recovery
id: crate_recovery
env:
VERSION: ${{ needs.guard.outputs.version }}
run: |
set -euo pipefail
ATTESTED="${RUNNER_TEMP}/attested/ordvec-${VERSION}.crate"
[ -f "$ATTESTED" ] || { echo "::error::attested .crate missing at $ATTESTED"; exit 1; }
A_SHA=$(sha256sum "$ATTESTED" | cut -d' ' -f1)
API_URL="https://crates.io/api/v1/crates/ordvec/${VERSION}/download"
STATIC_URL="https://static.crates.io/crates/ordvec/ordvec-${VERSION}.crate"
CRATES_IO_USER_AGENT="ordvec-release-verify/${VERSION} (https://github.com/Fieldnote-Echo/ordvec)"
EXISTING="${RUNNER_TEMP}/existing-ordvec.crate"
API_STATUS_FILE="${RUNNER_TEMP}/existing-ordvec-api-status.txt"
STATIC_STATUS_FILE="${RUNNER_TEMP}/existing-ordvec-static-status.txt"
already_present=false

rm -f "$EXISTING" "$API_STATUS_FILE" "$STATIC_STATUS_FILE"
API_CURL_EXIT=0
curl -sSL --retry 3 --retry-delay 2 --retry-all-errors --connect-timeout 10 --max-time 60 \
--user-agent "$CRATES_IO_USER_AGENT" \
--write-out "%{http_code}" \
--output "$EXISTING" \
"$API_URL" > "$API_STATUS_FILE" || API_CURL_EXIT=$?
API_STATUS="$(cat "$API_STATUS_FILE")"
if [ "$API_CURL_EXIT" -ne 0 ]; then
echo "::error::could not determine crates.io status while checking ordvec ${VERSION} at $API_URL (curl exit ${API_CURL_EXIT}). Refusing recovery."
exit 1
fi
case "$API_STATUS" in
200)
already_present=true
;;
404)
rm -f "$EXISTING"
;;
*)
echo "::error::unexpected crates.io status ${API_STATUS} while checking ordvec ${VERSION} at $API_URL. Refusing recovery."
exit 1
;;
esac

if [ "$already_present" != true ]; then
STATIC_CURL_EXIT=0
curl -sSL --retry 3 --retry-delay 2 --retry-all-errors --connect-timeout 10 --max-time 60 \
--user-agent "$CRATES_IO_USER_AGENT" \
--write-out "%{http_code}" \
--output "$EXISTING" \
"$STATIC_URL" > "$STATIC_STATUS_FILE" || STATIC_CURL_EXIT=$?
STATIC_STATUS="$(cat "$STATIC_STATUS_FILE")"
if [ "$STATIC_CURL_EXIT" -ne 0 ]; then
echo "::error::could not determine crates.io status while checking ordvec ${VERSION} at $STATIC_URL (curl exit ${STATIC_CURL_EXIT}). Refusing recovery."
exit 1
fi
case "$STATIC_STATUS" in
200)
already_present=true
;;
404)
rm -f "$EXISTING"
;;
*)
echo "::error::unexpected crates.io status ${STATIC_STATUS} while checking ordvec ${VERSION} at $STATIC_URL. Refusing recovery."
exit 1
;;
esac
fi

if [ "$already_present" = true ]; then
E_SHA=$(sha256sum "$EXISTING" | cut -d' ' -f1)
echo "attested: $A_SHA"
echo "crates.io-served: $E_SHA"
if [ "$A_SHA" != "$E_SHA" ]; then
echo "::error::crates.io already serves ordvec ${VERSION}, but the served .crate is not byte-identical to the SLSA-attested artifact ($E_SHA != $A_SHA). Refusing recovery."
exit 1
fi
echo "already_published=true" >> "$GITHUB_OUTPUT"
echo "::notice::crates.io already serves byte-identical ordvec ${VERSION}; skipping upload and verifying served bytes."
else
echo "already_published=false" >> "$GITHUB_OUTPUT"
echo "Both crates.io recovery endpoints returned 404 for ordvec ${VERSION}; proceeding with publish."
fi
# Mint the short-lived crates.io credential immediately before publish so
# the ephemeral token's exposure window is minimal. No stored secret.
- name: Mint a short-lived crates.io credential (OIDC)
if: steps.crate_recovery.outputs.already_published != 'true'
id: auth
uses: rust-lang/crates-io-auth-action@bbd81622f20ce9e2dd9622e3218b975523e45bbe # v1.0.4
- name: cargo publish
if: steps.crate_recovery.outputs.already_published != 'true'
env:
CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
run: cargo publish -p ordvec --locked
Expand Down Expand Up @@ -1338,8 +1440,62 @@ jobs:
STATIC_URL="https://static.crates.io/crates/ordvec-manifest/ordvec-manifest-${VERSION}.crate"
CRATES_IO_USER_AGENT="ordvec-release-verify/${VERSION} (https://github.com/Fieldnote-Echo/ordvec)"
EXISTING="${RUNNER_TEMP}/existing-ordvec-manifest.crate"
if curl -fsSL --user-agent "$CRATES_IO_USER_AGENT" "$API_URL" -o "$EXISTING" \
|| curl -fsSL --user-agent "$CRATES_IO_USER_AGENT" "$STATIC_URL" -o "$EXISTING"; then
API_STATUS_FILE="${RUNNER_TEMP}/existing-ordvec-manifest-api-status.txt"
STATIC_STATUS_FILE="${RUNNER_TEMP}/existing-ordvec-manifest-static-status.txt"
already_present=false

rm -f "$EXISTING" "$API_STATUS_FILE" "$STATIC_STATUS_FILE"
API_CURL_EXIT=0
curl -sSL --retry 3 --retry-delay 2 --retry-all-errors --connect-timeout 10 --max-time 60 \
--user-agent "$CRATES_IO_USER_AGENT" \
--write-out "%{http_code}" \
--output "$EXISTING" \
"$API_URL" > "$API_STATUS_FILE" || API_CURL_EXIT=$?
API_STATUS="$(cat "$API_STATUS_FILE")"
if [ "$API_CURL_EXIT" -ne 0 ]; then
echo "::error::could not determine crates.io status while checking ordvec-manifest ${VERSION} at $API_URL (curl exit ${API_CURL_EXIT}). Refusing recovery."
exit 1
fi
case "$API_STATUS" in
200)
already_present=true
;;
404)
rm -f "$EXISTING"
;;
*)
echo "::error::unexpected crates.io status ${API_STATUS} while checking ordvec-manifest ${VERSION} at $API_URL. Refusing recovery."
exit 1
;;
esac

if [ "$already_present" != true ]; then
STATIC_CURL_EXIT=0
curl -sSL --retry 3 --retry-delay 2 --retry-all-errors --connect-timeout 10 --max-time 60 \
--user-agent "$CRATES_IO_USER_AGENT" \
--write-out "%{http_code}" \
--output "$EXISTING" \
"$STATIC_URL" > "$STATIC_STATUS_FILE" || STATIC_CURL_EXIT=$?
STATIC_STATUS="$(cat "$STATIC_STATUS_FILE")"
if [ "$STATIC_CURL_EXIT" -ne 0 ]; then
echo "::error::could not determine crates.io status while checking ordvec-manifest ${VERSION} at $STATIC_URL (curl exit ${STATIC_CURL_EXIT}). Refusing recovery."
exit 1
fi
case "$STATIC_STATUS" in
200)
already_present=true
;;
404)
rm -f "$EXISTING"
;;
*)
echo "::error::unexpected crates.io status ${STATIC_STATUS} while checking ordvec-manifest ${VERSION} at $STATIC_URL. Refusing recovery."
exit 1
;;
esac
fi

if [ "$already_present" = true ]; then
E_SHA=$(sha256sum "$EXISTING" | cut -d' ' -f1)
echo "attested: $A_SHA"
echo "crates.io-served: $E_SHA"
Expand All @@ -1351,7 +1507,7 @@ jobs:
echo "::notice::crates.io already serves byte-identical ordvec-manifest ${VERSION}; skipping upload and verifying served bytes."
else
echo "already_published=false" >> "$GITHUB_OUTPUT"
echo "No existing ordvec-manifest ${VERSION} .crate found on crates.io; proceeding with publish."
echo "Both crates.io recovery endpoints returned 404 for ordvec-manifest ${VERSION}; proceeding with publish."
fi
- name: Validate manifest publish dry-run
if: steps.manifest_crate_recovery.outputs.already_published != 'true'
Expand Down Expand Up @@ -1398,7 +1554,7 @@ jobs:

publish-manifest-pypi:
name: publish ordvec-manifest to PyPI
needs: [guard, pypi-manifest-canonical-dist, release-manifest-assets-draft]
needs: [guard, pypi-manifest-canonical-dist, release-manifest-assets-draft, publish-manifest-crate]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
environment:
Expand Down Expand Up @@ -1437,11 +1593,17 @@ jobs:
python3 tests/release_pypi_canonical_dist.py verify \
--project ordvec-manifest \
--version "$VERSION" \
--dist-dir dist
--dist-dir dist \
--expected-wheels 4 \
--expected-sdists 1 \
--required-wheel-tag x86_64 \
--required-wheel-tag aarch64 \
--required-wheel-tag macosx \
--required-wheel-tag win_amd64

publish-pypi:
name: publish to PyPI
needs: [guard, pypi-canonical-dist, release-assets-draft]
needs: [guard, pypi-canonical-dist, release-assets-draft, publish-crate]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
environment:
Expand Down Expand Up @@ -1479,7 +1641,13 @@ jobs:
set -euo pipefail
python3 tests/release_pypi_canonical_dist.py verify \
--version "$VERSION" \
--dist-dir dist
--dist-dir dist \
--expected-wheels 4 \
--expected-sdists 1 \
--required-wheel-tag x86_64 \
--required-wheel-tag aarch64 \
--required-wheel-tag macosx \
--required-wheel-tag win_amd64

publish-github-release:
name: un-draft the GitHub Release (only after all registry publishes succeed)
Expand Down
51 changes: 29 additions & 22 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@
> push.

`ordvec` (the Rust crate), `ordvec-manifest` (the lockstep manifest verifier
crate), and `ordvec` on PyPI (the PyO3 wheel built from `ordvec-python/`) are
released by **pushing a `vMAJOR.MINOR.PATCH` tag** to a commit on `main`. The
release workflow handles build, canonical Python artifact selection,
crate), `ordvec` on PyPI (the PyO3 wheel built from `ordvec-python/`), and
`ordvec-manifest` on PyPI (the PyO3 wheel built from
`ordvec-manifest-python/`) are released by **pushing a `vMAJOR.MINOR.PATCH` tag**
to current `main` HEAD. The release workflow handles build, canonical Python artifact selection,
attestation, SLSA provenance, Release-asset attach, and un-draft automatically;
only the registry environment approvals are manual.

Expand All @@ -21,13 +22,13 @@ The unified `release.yml`:
- triggers on **tag push** (`v[0-9]*.[0-9]*.[0-9]*`); a strict-SemVer guard
step rejects pre-release / leading-zero / non-SemVer tags so they wake the
workflow but skip every job below the gate;
- runs a **`require-ci-green`** gate confirming the per-commit CI is green on
`main` for the tagged SHA — `ci.yml`, `python.yml`, `fuzz.yml`, `codeql.yml`,
`actionlint.yml`, `zizmor.yml`
(a *successful* run for that exact SHA on `main`);
- runs a **`require-ci-green`** gate confirming the tag points at current `main`
HEAD and that per-commit CI is green on `main` for that SHA — `ci.yml`,
`python.yml`, `fuzz.yml`, `codeql.yml`, `actionlint.yml`, `zizmor.yml` (a
*successful* run for that exact SHA on `main`);
- publishes via **OIDC trusted publishing** (no long-lived crates.io / PyPI
tokens in the repo) for both Rust crates and the Python distribution;
- canonicalizes the Python dist before attestation and release upload: for a
tokens in the repo) for both Rust crates and both Python distributions;
- canonicalizes each Python dist before attestation and release upload: for a
new PyPI version it uses the current run's wheels/sdist; if PyPI already owns
that immutable version during recovery, it downloads the exact PyPI-served
files, verifies their SHA-256 digests from PyPI JSON, and uses those bytes as
Expand Down Expand Up @@ -69,7 +70,7 @@ The unified `release.yml`:
`cargo package -p ordvec-manifest --locked` and byte-compares that output to
the attested artifact before minting its own OIDC token;
- **un-drafts the GitHub Release ONLY after `publish-crate`,
`publish-manifest-crate`, AND `publish-pypi` succeed**
`publish-manifest-crate`, `publish-pypi`, AND `publish-manifest-pypi` succeed**
(`publish-github-release` is the sole un-draft point). If any publish fails
or is skipped, the Release stays DRAFT — no public Release ever exists for a
version the registries refused;
Expand Down Expand Up @@ -116,7 +117,10 @@ filename. Until a record is updated, the corresponding gated publish fails
requires an initial owner bootstrap before a new crate's Trusted Publisher can
be configured, do that explicit maintainer-approved bootstrap before tagging.
- **PyPI** → `ordvec` → Publishing → GitHub publisher: `workflow = release.yml`,
`environment = pypi`.
`environment = pypi`, project URL `https://pypi.org/p/ordvec`.
- **PyPI** → `ordvec-manifest` → Publishing → GitHub publisher:
`workflow = release.yml`, `environment = pypi`, project URL
`https://pypi.org/p/ordvec-manifest`.

### Tag and branch protection

Expand Down Expand Up @@ -188,7 +192,7 @@ filename. Until a record is updated, the corresponding gated publish fails
and accept only the stable release tag pattern. Separately verify the
registry Trusted Publisher records by hand: crates.io must point both
`ordvec` and `ordvec-manifest` to `release.yml` / `crates-io`, and PyPI must
point `ordvec` to `release.yml` / `pypi`.
point both `ordvec` and `ordvec-manifest` to `release.yml` / `pypi`.
6. Get the maintainer's explicit go to publish.
7. Push the version tag from `main` (signed):

Expand All @@ -198,18 +202,20 @@ filename. Until a record is updated, the corresponding gated publish fails
```

`release.yml` triggers automatically. It builds the core `.crate`, wheels,
and sdist; selects the canonical Python dist (current build for a new PyPI
version, verified PyPI bytes for an existing immutable version); attests the
files this run can honestly attest (GitHub attestation store +
and sdist for both Python packages; selects the canonical Python dists
(current build for a new PyPI version, verified PyPI bytes for an existing
immutable version); attests the files this run can honestly attest (GitHub
attestation store +
`*.sigstore.json`); generates SLSA `*.intoto.jsonl`; and stages the core and
Python assets on the GitHub Release — **as a DRAFT**. After `publish-crate`
succeeds, it builds, attests, generates SLSA provenance for, and stages the
lockstep `ordvec-manifest` `.crate`, then pauses at the manifest registry
environment gate.
8. **Approve each publish environment pause** in the Actions UI. There are
three registry publish jobs: `publish-crate`, `publish-manifest-crate`, and
`publish-pypi`. The two crates.io jobs use the same `crates-io` environment
and may require separate approvals; PyPI uses the `pypi` environment.
four registry publish jobs: `publish-crate`, `publish-manifest-crate`,
`publish-pypi`, and `publish-manifest-pypi`. The two crates.io jobs use the
same `crates-io` environment and may require separate approvals; the two PyPI
jobs use the `pypi` environment and may also require separate approvals.
Required-reviewer approval is what authorises each registry push.
- `publish-crate` and `publish-manifest-crate` first sha256-compare their
repackaged `.crate` to the SLSA-attested artifact — if either diverges
Expand All @@ -219,10 +225,11 @@ filename. Until a record is updated, the corresponding gated publish fails
un-drafts the GitHub Release automatically. If one gate fails, the Release
stays DRAFT — investigate and re-run from a fixed workflow rather than
approving another registry into a partial state.
- `publish-pypi` either uploads the fresh canonical dist or, if PyPI already
serves that version, skips upload and verifies the existing files. In both
modes it compares every PyPI-served wheel/sdist SHA-256 digest against the
canonical `dist/` files before the GitHub Release can un-draft.
- `publish-pypi` and `publish-manifest-pypi` either upload their fresh
canonical dist or, if PyPI already serves that version, skip upload and
verify the existing files. In both modes they compare every PyPI-served
wheel/sdist SHA-256 digest against the canonical `dist/` files before the
GitHub Release can un-draft.
9. Verify each published artifact and its provenance:
- crates.io / docs.rs for `ordvec` and `ordvec-manifest`;
- PyPI (confirm the post-publish hash-verification log, optionally
Expand Down
Loading
Loading