Skip to content

Shard sublibrary-project-tests matrix under GitHub's 256-job cap#107

Merged
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:fix-sublibrary-matrix-256-shard
Jul 5, 2026
Merged

Shard sublibrary-project-tests matrix under GitHub's 256-job cap#107
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:fix-sublibrary-matrix-256-shard

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown

Problem

sublibrary-project-tests.yml feeds the entire affected-sublibrary matrix into a single strategy.matrix on a job that calls a reusable workflow (uses: tests.yml@v1). GitHub caps a matrix at 256 jobs; above that it instantiates no cells and fails the run with nothing scheduled (no annotation, only detect shows up as a job).

A monorepo-wide change that touches many sublibraries at once trips this. Concretely, OrdinaryDiffEq.jl #3777 (a QA/ExplicitImports conversion editing ~54 lib/*/test/qa/*) made detect compute a 266-cell projects-matrix (incl. 56 QA cells). detect succeeded and has_changes=true, but the test job produced zero jobs and the run failed — so QA never ran for any sublibrary, silently. Smaller PRs (e.g. an 87-cell master push) spawn fine, including their QA cells, which is why this went unnoticed.

Fix

Split the computed projects-matrix into 200-cell shards in detect and run one test-N reusable-workflow job per shard (4 shards → 800-cell capacity), each gated by has_N. Every per-job strategy.matrix now stays well under 256.

  • Small affected sets are unchanged: everything lands in shard_1; shards 2–4 skip via has_N == 'false'.
  • A >800-cell total errors loudly asking for another shard, rather than silently dropping jobs.

Validation (local)

Replayed the exact detect shell logic against the real 266-cell #3777 matrix (extracted from the failing run's detect log):

shard_1: len=200 (44 QA)  shard_2: len=66 (12 QA)  shard_3/4: len=0
union == original (order-preserving) = true; 56 QA cells preserved

Also verified the 4-cell (single-sublib) and empty-matrix cases behave as before. YAML parses. No external consumer references this workflow's removed matrix/has_changes outputs (grouped-tests.yml has its own separate detect).

Takes effect for OrdinaryDiffEq #3777 (and every monorepo using @v1) once merged and v1 is retagged.

Please ignore until reviewed by @ChrisRackauckas.

🤖 Generated with Claude Code

The affected-sublibrary test matrix is fed into a single strategy.matrix on a
job that calls a reusable workflow (uses: tests.yml). GitHub caps a matrix at
256 jobs; above that it instantiates no cells and fails the run with nothing
scheduled. A monorepo-wide change that touches many sublibraries at once (e.g. a
QA/ExplicitImports conversion editing every lib/*/test/qa) can push the affected
set past 256 (OrdinaryDiffEq.jl #3777 computed 266 cells, incl. 56 QA cells),
so the whole test job silently produced zero jobs and QA never ran.

Split the computed projects-matrix into 200-cell shards in detect and run one
test-N reusable-workflow job per shard, each with has_N gating. Small affected
sets are unchanged (everything lands in shard_1; shards 2-4 skip). A >800-cell
total errors loudly asking for another shard rather than silently dropping jobs.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review July 5, 2026 05:53
@ChrisRackauckas ChrisRackauckas merged commit 22690c4 into SciML:master Jul 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants