Supply chain hardening: Docker image cosigning, dependency lockdown, and CI pinning#365
Supply chain hardening: Docker image cosigning, dependency lockdown, and CI pinning#365OlivierTrudeau wants to merge 3 commits into
Conversation
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThis PR introduces cryptographic image signature verification for Aurora's container artifacts using Sigstore Cosign. CI workflows now sign published images and Helm charts after pushing to GHCR, a new verification script validates signatures during deployment, and the Makefile orchestrates signing/verification in build and deployment flows. Documentation describes manual and automatic verification, plus Kubernetes policy enforcement. ChangesImage Signing and Verification System
Sequence DiagramsequenceDiagram
participant CI as GitHub Actions
participant GHCR as Container Registry<br/>(GHCR)
participant Cosign as Sigstore Cosign
participant Local as Local Build/Deploy
participant K8s as Kubernetes
CI->>GHCR: Build & Push Images
CI->>Cosign: Sign Image Digest (OIDC)
Cosign-->>GHCR: Signature Stored
Local->>GHCR: Pull Signed Images
Local->>Cosign: Verify Signatures
Cosign-->>Local: ✓ Verified
Local->>K8s: Deploy via Helm
Local->>Local: Build Images Locally
Local->>Cosign: Sign Local Images
Cosign-->>Local: ✓ Signed
Local->>K8s: Deploy via Helm
K8s->>Cosign: (Optional) Enforce Signatures<br/>via policy-controller
Cosign-->>K8s: ✓ Admission Allowed
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 6 minutes and 18 seconds.Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/publish-images.yml:
- Around line 227-234: The workflow inspects the bare IMAGE_NAME (which defaults
to :latest) and thus grabs the wrong manifest; change the docker buildx
imagetools inspect call that sets DIGEST to target the explicitly tagged image
reference used when pushing (the same image reference used in the "Inspect and
verify multi-arch manifest" step) instead of the bare IMAGE_NAME, so that DIGEST
reflects the newly created tag (e.g., inspect "${IMAGE_NAME}:${IMAGE_TAG}" or
the exact image variable from the earlier push), and keep the cosign sign
invocation using "${IMAGE_NAME}@${DIGEST}" so cosign signs the correct manifest.
In `@Makefile`:
- Around line 341-345: The deploy-build → deploy failure is because cosign
keyless signatures produced locally don’t match the GitHub Actions OIDC
issuer/identity restrictions enforced by scripts/cosign-verify.sh; either make
signing CI-only by gating the cosign sign steps in the Makefile (the cosign sign
lines for aurora-server, aurora-frontend, aurora-terminal under deploy-build) so
they only run when GITHUB_ACTIONS (or another CI-only env flag) is true, or add
a local signing path and verification policy: create a separate local-sign
branch in the Makefile (e.g., deploy-build-local) and/or a second verification
mode in scripts/cosign-verify.sh that accepts a local issuer/identity-regexp or
a SKIP_LOCAL_VERIFICATION flag, and update deploy to choose the appropriate
verification path based on the same CI/local flag.
- Around line 339-345: The Makefile currently computes SERVER_DIGEST,
FRONTEND_DIGEST, and TERMINAL_DIGEST by hashing the raw manifest bytes which can
include an extra newline; instead, query the registry for the canonical digest
and use that for cosign. Replace the docker buildx imagetools inspect --raw |
sha256sum pipeline for SERVER_DIGEST, FRONTEND_DIGEST and TERMINAL_DIGEST with a
direct lookup such as docker buildx imagetools inspect IMAGE:TAG --format
'{{.Manifest.Digest}}' (or crane digest IMAGE:TAG) to set those variables before
calling cosign sign --yes IMAGE@$$SERVER_DIGEST (and the frontend/terminal
equivalents).
In `@scripts/cosign-verify.sh`:
- Around line 11-14: The script currently exits 0 when cosign is missing which
silently bypasses verification; change this to fail closed by exiting non-zero
unless an explicit opt-out is provided (e.g., an env var SKIP_COSIGN=true or a
--allow-unverified/--skip-cosign flag). Update the cosign presence check (the if
! command -v cosign ... block) to: 1) detect an opt-out (check $SKIP_COSIGN or
parse a --skip-cosign flag), 2) if no opt-out, print a clear error and exit with
a non-zero status, and 3) if opt-out is present, print a conspicuous warning
including why it is unsafe and proceed. Ensure all messages reference cosign and
the opt-out variable/flag so callers can audit the behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 17cbb153-4d25-4f7b-b0c5-1a76a93ef5c2
⛔ Files ignored due to path filters (1)
client/bun.lockis excluded by!**/*.lock,!**/*.lock
📒 Files selected for processing (8)
.github/workflows/publish-helm.yml.github/workflows/publish-images.ymlMakefileclient/.npmrcclient/Dockerfilescripts/cosign-verify.shwebsite/docs/deployment/image-verification.mdwebsite/sidebars.ts
| SERVER_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-server:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \ | ||
| FRONTEND_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-frontend:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \ | ||
| cosign sign --yes $$IMAGE_REGISTRY/aurora-server@$$SERVER_DIGEST; \ | ||
| cosign sign --yes $$IMAGE_REGISTRY/aurora-frontend@$$FRONTEND_DIGEST; \ | ||
| if [ "$$ENABLE_POD_ISOLATION" = "true" ]; then \ | ||
| TERMINAL_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-terminal:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \ | ||
| cosign sign --yes $$IMAGE_REGISTRY/aurora-terminal@$$TERMINAL_DIGEST; \ |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Is the SHA256 of docker buildx imagetools inspect --raw IMAGE:TAGoutput guaranteed to equal the registry manifest digest, or shoulddocker buildx imagetools inspect IMAGE:TAG --format '{{.Manifest.Digest}}'/crane digest be used when signing with Cosign?
💡 Result:
No, the SHA256 of docker buildx imagetools inspect --raw IMAGE:TAG output is not guaranteed to equal the registry manifest digest. The --raw flag outputs the original unformatted JSON manifest bytes fetched from the registry, but the command may add a trailing newline character when printing to stdout. This extra byte alters the hash, making a direct SHA256 (e.g., sha256sum) mismatch the true registry digest. To reliably obtain the exact registry manifest digest: - Use docker buildx imagetools inspect IMAGE:TAG --format '{{.Manifest.Digest}}' (computes and displays the digest directly). - Or crane digest IMAGE:TAG (fetches via HEAD or GET and returns the precise SHA256 digest). For Cosign signing, use one of these digest commands, as Cosign requires the exact registry manifest digest (e.g., cosign sign IMAGE@sha256:digest). Relying on hashing --raw output risks incorrect digests due to the potential trailing newline, as noted in Docker Buildx issue #181 where developers recommend stripping it (e.g., docker buildx imagetools inspect --raw IMAGE | perl -pe 'chomp if eof' | sha256sum).
Citations:
- 1: https://docs.docker.com/reference/cli/docker/buildx/imagetools/inspect/
- 2: docker buildx imagetools inspect --raw output is lacking Digest part docker/buildx#181
- 3: https://linuxcommandlibrary.com/man/crane-digest
- 4: https://docs.docker.com/reference/cli/docker/buildx/imagetools/inspect
- 5: https://github.com/google/go-containerregistry/blob/v0.21.3/pkg/crane/digest.go
Replace manifest digest derivation with direct registry lookup.
Lines 339-345 compute digests by piping docker buildx imagetools inspect --raw output to sha256sum. This approach is unreliable: the --raw flag outputs manifest JSON bytes, but a trailing newline is appended when writing to stdout. This extra byte changes the hash, causing it to diverge from the actual registry manifest digest. Cosign would then sign a non-existent digest.
Use docker buildx imagetools inspect IMAGE:TAG --format '{{.Manifest.Digest}}' to retrieve the digest directly from the registry (or alternatively, crane digest IMAGE:TAG).
Suggested fix
- SERVER_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-server:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \
- FRONTEND_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-frontend:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \
+ SERVER_DIGEST=$$(docker buildx imagetools inspect $$IMAGE_REGISTRY/aurora-server:$$GIT_SHA --format '{{.Manifest.Digest}}'); \
+ FRONTEND_DIGEST=$$(docker buildx imagetools inspect $$IMAGE_REGISTRY/aurora-frontend:$$GIT_SHA --format '{{.Manifest.Digest}}'); \
cosign sign --yes $$IMAGE_REGISTRY/aurora-server@$$SERVER_DIGEST; \
cosign sign --yes $$IMAGE_REGISTRY/aurora-frontend@$$FRONTEND_DIGEST; \
if [ "$$ENABLE_POD_ISOLATION" = "true" ]; then \
- TERMINAL_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-terminal:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \
+ TERMINAL_DIGEST=$$(docker buildx imagetools inspect $$IMAGE_REGISTRY/aurora-terminal:$$GIT_SHA --format '{{.Manifest.Digest}}'); \
cosign sign --yes $$IMAGE_REGISTRY/aurora-terminal@$$TERMINAL_DIGEST; \
fi; \🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Makefile` around lines 339 - 345, The Makefile currently computes
SERVER_DIGEST, FRONTEND_DIGEST, and TERMINAL_DIGEST by hashing the raw manifest
bytes which can include an extra newline; instead, query the registry for the
canonical digest and use that for cosign. Replace the docker buildx imagetools
inspect --raw | sha256sum pipeline for SERVER_DIGEST, FRONTEND_DIGEST and
TERMINAL_DIGEST with a direct lookup such as docker buildx imagetools inspect
IMAGE:TAG --format '{{.Manifest.Digest}}' (or crane digest IMAGE:TAG) to set
those variables before calling cosign sign --yes IMAGE@$$SERVER_DIGEST (and the
frontend/terminal equivalents).
| cosign sign --yes $$IMAGE_REGISTRY/aurora-server@$$SERVER_DIGEST; \ | ||
| cosign sign --yes $$IMAGE_REGISTRY/aurora-frontend@$$FRONTEND_DIGEST; \ | ||
| if [ "$$ENABLE_POD_ISOLATION" = "true" ]; then \ | ||
| TERMINAL_DIGEST=$$(docker buildx imagetools inspect --raw $$IMAGE_REGISTRY/aurora-terminal:$$GIT_SHA | sha256sum | awk '{print "sha256:"$$1}'); \ | ||
| cosign sign --yes $$IMAGE_REGISTRY/aurora-terminal@$$TERMINAL_DIGEST; \ |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
When cosign sign --yes IMAGE@DIGESTis run locally outside GitHub Actions, what certificate issuer and subject are embedded in the Fulcio certificate? Willcosign verify --certificate-identity-regexp='https://github.com/Arvo-AI/aurora/.*' --certificate-oidc-issuer='https://token.actions.githubusercontent.com' IMAGE:TAG accept those locally generated signatures?
💡 Result:
Certificate embedded in the Fulcio (keyless) certificate when you run cosign sign --yes IMAGE@DIGEST locally: 1) Certificate issuer - The OIDC token issuer (“iss” in the OIDC JWT) is embedded/checked via Fulcio’s certificate custom field, and for GitHub Actions workloads it is https://token.actions.githubusercontent.com. 2) Certificate subject - For GitHub Actions, the subject is the workflow identity URI placed into the certificate’s Subject Alternative Name (SAN) as a URI, of the form: https://github.com///.github/workflows/<WORKFLOW_FILE>@refs/heads/ (Fulcio includes the GitHub Actions workflow identity in the certificate SAN.) Will cosign verify --certificate-identity-regexp='https://github.com/Arvo-AI/aurora/.*' --certificate-oidc-issuer='https://token.actions.githubusercontent.com' accept those locally generated signatures? - Yes, provided the locally generated signature was created using a GitHub Actions OIDC identity with certificate-oidc-issuer https://token.actions.githubusercontent.com and its certificate SAN subject matches the GitHub workflow-identity URI pattern. Key caveat about “locally outside GitHub Actions” - Running cosign locally does not inherently mint a GitHub Actions OIDC token; cosign obtains an OIDC identity token from ambient providers or from configured flags/tokens. Therefore, if the OIDC token used locally is not a GitHub Actions token (i.e., its iss is not https://token.actions.githubusercontent.com), the resulting Fulcio certificate will not have that issuer and cosign verify with --certificate-oidc-issuer=https://token.actions.githubusercontent.com will fail. What you should check - Inspect the certificate identity/issuer in the verification output or by running cosign verify with verbose/identity flags; the SAN URI subject and issuer must match the expectations above.
Citations:
- 1: https://docs.sigstore.dev/certificate_authority/oidc-in-fulcio
- 2: Add Github workflow run information to the signing certificate sigstore/fulcio#624
- 3: https://docs.sigstore.dev/cosign/verifying/verify/
- 4: https://github.com/sigstore/cosign/blob/29361993239764ae63c3017198cc80ff5816c08f/pkg/cosign/verify.go
- 5: https://github.com/sigstore/cosign/blob/7ba521444f9fcfdf2e1e5936c05834597674e6c9/cmd/cosign/cli/fulcio/fulcio.go
- 6: "cosign verify-blob" or "cosign verify" with local certificate and chain always asks for oidc provider sigstore/cosign#3572
🏁 Script executed:
# Check the Makefile to understand the deploy-build and deploy targets
git ls-files | grep -i makefileRepository: Arvo-AI/aurora
Length of output: 67
🏁 Script executed:
# Read the relevant sections of the Makefile
sed -n '330,370p' MakefileRepository: Arvo-AI/aurora
Length of output: 2356
🏁 Script executed:
# Check if cosign-verify.sh exists and examine its verification policy
find . -name "cosign-verify.sh" -type f 2>/dev/nullRepository: Arvo-AI/aurora
Length of output: 85
🏁 Script executed:
# Read the cosign-verify.sh script
cat -n scripts/cosign-verify.shRepository: Arvo-AI/aurora
Length of output: 1390
deploy-build → deploy flow will fail for local execution.
The deploy-build target signs images with cosign sign --yes, but immediately afterward, the deploy target runs scripts/cosign-verify.sh which enforces verification against GitHub Actions OIDC (https://token.actions.githubusercontent.com) and GitHub workflow identity patterns only.
When cosign signs locally outside GitHub Actions, it obtains an OIDC token from a different provider/context, producing a Fulcio certificate with a different issuer and subject. The verification in cosign-verify.sh (lines 26–27) will reject these locally-signed images:
--certificate-identity-regexp="https://github.com/Arvo-AI/aurora/.*"
--certificate-oidc-issuer="https://token.actions.githubusercontent.com"
Choose one model: either make deploy-build CI-only (run signing in GitHub Actions exclusively), or add a separate local signing path with its own verification policy.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Makefile` around lines 341 - 345, The deploy-build → deploy failure is
because cosign keyless signatures produced locally don’t match the GitHub
Actions OIDC issuer/identity restrictions enforced by scripts/cosign-verify.sh;
either make signing CI-only by gating the cosign sign steps in the Makefile (the
cosign sign lines for aurora-server, aurora-frontend, aurora-terminal under
deploy-build) so they only run when GITHUB_ACTIONS (or another CI-only env flag)
is true, or add a local signing path and verification policy: create a separate
local-sign branch in the Makefile (e.g., deploy-build-local) and/or a second
verification mode in scripts/cosign-verify.sh that accepts a local
issuer/identity-regexp or a SKIP_LOCAL_VERIFICATION flag, and update deploy to
choose the appropriate verification path based on the same CI/local flag.
| if ! command -v cosign >/dev/null 2>&1; then | ||
| echo "WARNING: cosign not installed, skipping signature verification." | ||
| echo "Install cosign to verify image provenance: https://docs.sigstore.dev/cosign/system_config/installation/" | ||
| exit 0 |
There was a problem hiding this comment.
Don't silently bypass verification when cosign is missing.
These deployment paths are presented as enforcement, but this block returns success when the verifier is absent. That makes "don't install cosign" an easy bypass for the new supply-chain check. Fail closed here, or require an explicit opt-out flag for unverified deployments.
Suggested fix
if ! command -v cosign >/dev/null 2>&1; then
- echo "WARNING: cosign not installed, skipping signature verification."
- echo "Install cosign to verify image provenance: https://docs.sigstore.dev/cosign/system_config/installation/"
- exit 0
+ if [ "${ALLOW_UNVERIFIED_IMAGES:-0}" = "1" ]; then
+ echo "WARNING: cosign not installed, skipping signature verification."
+ exit 0
+ fi
+ echo "ERROR: cosign is required for image signature verification." >&2
+ exit 1
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if ! command -v cosign >/dev/null 2>&1; then | |
| echo "WARNING: cosign not installed, skipping signature verification." | |
| echo "Install cosign to verify image provenance: https://docs.sigstore.dev/cosign/system_config/installation/" | |
| exit 0 | |
| if ! command -v cosign >/dev/null 2>&1; then | |
| if [ "${ALLOW_UNVERIFIED_IMAGES:-0}" = "1" ]; then | |
| echo "WARNING: cosign not installed, skipping signature verification." | |
| exit 0 | |
| fi | |
| echo "ERROR: cosign is required for image signature verification." >&2 | |
| exit 1 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/cosign-verify.sh` around lines 11 - 14, The script currently exits 0
when cosign is missing which silently bypasses verification; change this to fail
closed by exiting non-zero unless an explicit opt-out is provided (e.g., an env
var SKIP_COSIGN=true or a --allow-unverified/--skip-cosign flag). Update the
cosign presence check (the if ! command -v cosign ... block) to: 1) detect an
opt-out (check $SKIP_COSIGN or parse a --skip-cosign flag), 2) if no opt-out,
print a clear error and exit with a non-zero status, and 3) if opt-out is
present, print a conspicuous warning including why it is unsafe and proceed.
Ensure all messages reference cosign and the opt-out variable/flag so callers
can audit the behavior.
|
1 participant
Summary
publish-images.ymlafter multi-arch manifest creation. Sign Helm chart OCI artifacts inpublish-helm.ymlusing cosign + crane for digest resolution.--frozen-lockfile --ignore-scriptsin the production Dockerfile builder stage to prevent lockfile drift and block postinstall hook attacks (axios RAT vector). Addclient/.npmrcwithignore-scripts=truefor local dev parity.make prod-prebuiltandmake deployverify image signatures before running, failing hard if cosign detects invalid or missing signatures. Verification logic centralized inscripts/cosign-verify.sh.make deployfromdeploy-buildso K8s deployments can use prebuilt images. Add--no-buildtomake prod-prebuiltto prevent accidental rebuilds. Add cosign signing tomake deploy-buildfor self-hosted registry pushes.image-verification.mddocs page covering manual verification, automatic Makefile verification, and Sigstore policy-controller setup for K8s runtime enforcement.All GitHub Actions (existing and newly added) are pinned by full commit SHA.
Test plan
docker build --target builder -f client/Dockerfile ./clientsucceeds with synced lockfiledocker buildfails with--frozen-lockfilewhen lockfile is out of syncbun installinclient/skips postinstall scripts (.npmrc)scripts/cosign-verify.shexits 1 on unsigned imagesscripts/cosign-verify.shwarns and exits 0 when cosign is not installedwebsite/docs build succeeds with new pagepublish-images.ymlcosign signing succeeds (triggerworkflow_dispatchon this branch)cosign verifysucceeds locally after CI signing completespublish-helm.ymlsigning (verified on firstv*.*.*tag after merge)Made with Cursor
Summary by CodeRabbit
New Features
Chores