Skip to content

ci: intersect flat --solvers list with each problem in the benchmark matrix#94

Open
andrinr wants to merge 1 commit into
mainfrom
ci/benchmark-per-problem-solver-filter
Open

ci: intersect flat --solvers list with each problem in the benchmark matrix#94
andrinr wants to merge 1 commit into
mainfrom
ci/benchmark-per-problem-solver-filter

Conversation

@andrinr

@andrinr andrinr commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Problem

benchmark-run.yml runs a matrix over problems and applied the same flat inputs.solvers list to every leg's mosaic run (--problems "${{ matrix.problem }}" --solvers "$SOLVERS"). A cross-domain list breaks the legs whose problem lacks some of those solvers:

Run SOLVERS="deal.II,FEniCS,Firedrake,JAX-FEM,torch-fem"
Unknown solver 'torch-fem'.
Available solvers: FEniCS, Firedrake, JAX-FEM, TopOpt.jl, deal.II
Error: Process completed with exit code 1.

torch-fem is thermal-only (structural-mesh has TopOpt.jl instead), and mosaic run (_helpers.py:471) rejects a name absent from the problem it runs against. One bad leg fails the whole dispatch — which is exactly what blocks verifying a thermal-only change (e.g. #85) whenever the matrix also includes structural-mesh.

Fix

  • Add ​.github/scripts/intersect-solvers.py — prints the intersection of a flat solver list with one problem's own solver set (case-insensitive, canonical casing preserved), mirroring the existing filter-problems-by-solver.py.
  • In the Run step, intersect SOLVERS with matrix.problem before calling mosaic run; skip the leg when the intersection is empty.

Each leg now runs exactly the requested solvers it actually has:

leg requested runs
thermal-mesh deal.II,FEniCS,Firedrake,JAX-FEM,torch-fem all five
structural-mesh same FEniCS,Firedrake,JAX-FEM,deal.II (torch-fem dropped)
(no overlap) same leg skipped with a ::notice::

The pull-solver-images.py step already skips out-of-problem solvers (pull-solver-images.py:96), so only the run step needed this.

Verification

  • intersect-solvers.py compiles; YAML validates; the intersection logic is unit-tested (cases above).
  • Full end-to-end needs a benchmark dispatch (Docker + GHCR images), which I can't run here — the matrix legs are otherwise unchanged.

🤖 Generated with Claude Code

benchmark-run.yml runs a matrix over problems and passed the same flat
inputs.solvers list to every leg's `mosaic run`. A cross-domain list (e.g.
deal.II,FEniCS,Firedrake,JAX-FEM,torch-fem) then fails on the structural-mesh
leg, because torch-fem is thermal-only and `mosaic run` rejects a solver name
absent from the problem it runs against (TopOpt.jl is structural's torch-fem
counterpart). One bad leg fails the whole dispatch.

Add .github/scripts/intersect-solvers.py and, in the Run step, intersect the
requested solvers with the current problem's own solver set before invoking
mosaic; skip the leg when the intersection is empty. Each leg now runs exactly
the requested solvers it actually has. The pull-solver-images.py step already
skips out-of-problem solvers, so only the run step needed this.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@PasteurBot

Copy link
Copy Markdown
Contributor

📊 View the full benchmark results

No benchmarks ran for this PR, so there is no status report.


@dionhaefner

Copy link
Copy Markdown
Contributor

@andrinr Not quite understanding what bug is being fixed here. How can I trigger the error above?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants