Skip to content

ci: skip eval jobs deterministically on fork PRs #2345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andrey-esipov wants to merge 3 commits into
base: main
Choose a base branch
from
+13 −2
Open

ci: skip eval jobs deterministically on fork PRs #2345

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
47db5d0
ci: skip eval jobs deterministically on fork PRs
andrey-esipov Jul 25, 2026
0f9904c
ci: keep build-image running on forks to avoid startup failure
andrey-esipov Jul 25, 2026
a60522f
ci: correct rationale comment on build-image fork behavior
andrey-esipov Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/evals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,19 +45,30 @@ jobs:
- if: steps.check.outputs.exists == 'false'
run: cp package.json bun.lock .github/docker/

# A fork PR's GITHUB_TOKEN only has `packages: read`, so pushing fails.
# Still BUILD (validates Dockerfile.ci changes), just don't publish. This
# job intentionally keeps no `if:` so fork PRs still get one real, honest
# green check here instead of a run where every job is grey.
- if: steps.check.outputs.exists == 'false'
uses: docker/build-push-action@v6
with:
context: .github/docker
file: .github/docker/Dockerfile.ci
push: true
push: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
tags: |
${{ steps.meta.outputs.tag }}
${{ env.IMAGE }}:latest

# Fork PRs never receive repository secrets (ANTHROPIC_API_KEY et al), so every
# API-calling eval fails at SDK auth before a model runs. Skip deterministically
# rather than leaving the outcome to Docker-cache luck: a warm cache let these
# run and fail, a cold one made build-image fail its push and the shards skip.
# Same-repo PRs, pushes, and workflow_dispatch keep full coverage. Fork work
# gets real coverage via a trusted base-repo branch.
evals:
runs-on: ${{ matrix.suite.runner || 'ubicloud-standard-8' }}
needs: build-image
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
container:
image: ${{ needs.build-image.outputs.image-tag }}
credentials:
Expand Down Expand Up @@ -263,7 +274,7 @@ jobs:
report:
runs-on: ubicloud-standard-8
needs: evals
if: always() && github.event_name == 'pull_request'
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 5
permissions:
contents: read
Expand Down
Loading