perf: upgrade INS.jl tesseract to v5.0.0 (native ν gradient, in-place forward, float32) by agdestein · Pull Request #103 · pasteurlabs/mosaic

agdestein · 2026-07-10T16:41:45Z

Follow-up to agdestein/IncompressibleNavierStokes.jl#197 and the companion wording PR (#102). This upgrades the INS.jl tesseract to the just-released v5.0.0 and fixes three things that skewed the reported numbers. Every INS.jl symbol used by this tesseract is unchanged in v5.0.0, so the upgrade itself is just the version pin.

Suggested label: benchmark:solver — INS.jl numbers change; please re-run the ins_jl cells.

Changes that affect published numbers

VJP: native viscosity gradient replaces the finite-difference fallback.
INS.jl v5.0.0's diffusion rrule provides the ν cotangent (the diffusive term is linear in ν, so it is exact). ns_vjp now takes grad_v0, grad_dt, and grad_nu from a single Zygote pullback. This removes two full forward rollouts from every VJP call — the FD fallback ran unconditionally, even though vjp_cost (jax.grad w.r.t. v0) never requested the ν gradient, inflating all reported INS.jl VJP times by ~25–36%.
Forward pass: in-place solve_unsteady instead of the AD-ready rollout.
apply has no gradient tape to build, so it now uses the library's in-place time stepper (RKMethods.RK44, which applies the same per-stage BC + projection as the projected RHS used for the VJP primal). Measured at the 2-D cost point (N=256, 100 steps, CPU): 2.49 s → 0.56 s, with interiors agreeing to round-off (≤4e-14 relative in a Float64 A/B test). The VJP primal keeps the non-mutating rollout — see the validation section for the cross-check.
The solve now actually runs in Float32 (internal_dtype: float64 → float32).
Previously the Setup was built from Float64 coordinates, and INS.jl's non-mutating operators allocate their outputs from the setup grid — so the first projection silently promoted the state (and the whole Zygote tape) to Float64, despite the Float32 casts at every interface: Float64 compute cost with Float32 I/O precision. (A bare 0.5 literal in the collocated↔staggered helpers had the same effect; they are now eltype-preserving.) Building the grid in Float32 makes forward + VJP run in single precision end-to-end, like the other float32 solvers. Note this will move INS.jl's FD-gradient-error numbers to the float32 noise floor of its peers — the previous best-in-class values partly reflected the hidden double precision.
(If you'd rather keep INS.jl in genuine Float64 and disclose the dtype difference instead, that is a one-line revert of the LinRange change — happy to go either way.)

Changes with identical results (pure speedup)

Removed the per-stage add_ghosts/strip_ghosts wrapping in the RK4 rollout used for the VJP primal. create_right_hand_side applies the periodic BCs to its input and projects its output, so the ghost refresh between stages is redundant — the state's ghost entries go stale but are never read, and are stripped at the end. Bitwise-identical interiors; ~19% off the non-mutating rollout time plus fewer Zygote tape nodes.

Not changed

The channel/obstacle code paths (excluded from all experiments) are untouched.
Zygote pin (0.7.10) and Julia version (1.11.3) unchanged; INS.jl v5.0.0 requires only Julia ≥ 1.10.
Time integration stays RK4. A 3-stage third-order scheme would shave ~25% off both passes; that is a numerics/tuning decision best made separately — happy to send it as a follow-up if wanted.

Validation

Run against INS.jl v5.0.0 outside the container (calling the ns_solver.jl functions directly), 2-D (N=32) and 3-D (N=16):

in-place apply vs non-mutating VJP-primal rollout: rel. diff 2.2e-4 (2-D) / 1.0e-4 (3-D) — Float32 round-off accumulation (the same A/B in Float64 gives ≤4e-14);
grad_v0 (random direction) and grad_dt vs central finite differences: rel. err 1.1e-3–2.5e-3 (Float32 FD noise floor);
grad_nu vs central FD: 3e-2 at ε=1% shrinking monotonically to 5.4e-3 at ε=20% — the FD is noise-limited by Float32 loss quantization (identical FD values at ε=5% and 10%) and converges toward the AD value; the upstream rrule is additionally verified exactly (linear in ν) with ChainRulesTestUtils;
TGV kinetic-energy decay 0.9562 vs analytic 0.9608 at N=64/20 steps (the known ~2.4e-3 spatial + interpolation error of this solver in the baseline);
no silent promotion: every array in forward and pullback is Float32.

Expected effects of the benchmark:solver re-run: forward times ↓ ~4–6× plus the float32 gain, VJP times ↓ ~25–36% plus the float32 gain, FD-gradient errors move to the float32 floor, physics/agreement results unchanged within tolerance.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gdif1Cu1pdCJENe2hNUaF1

- Pin IncompressibleNavierStokes 5.0.0 (was 4.0.0). - ns_vjp: take grad_nu from the same Zygote pullback as grad_v0/grad_dt; v5's diffusion rrule provides the viscosity cotangent (exact, the term is linear in nu). Removes the finite-difference fallback and its two extra forward rollouts per VJP call (~25-36% of reported VJP time, which vjp_cost never requested). - ns_apply: forward-only rollouts use the in-place solve_unsteady stepper (RK44, same per-stage BC + projection as the projected RHS); 4-6x faster than the non-mutating AD-ready rollout at identical interiors. The VJP primal keeps the non-mutating path. - Build the Setup from Float32 coordinates and make the collocated<-> staggered helpers eltype-preserving. Previously the Float64 grid (and a bare 0.5 literal) silently promoted the state and the Zygote tape to Float64 after the first projection, despite Float32 casts at every interface; internal_dtype now truthfully float32. - Drop the redundant per-stage ghost wrapping in the RK4 rollout: create_right_hand_side applies periodic BCs to its input and projects its output, so the refresh was dead work (~19% of the rollout plus Zygote tape nodes); interiors are bitwise identical. Validated against v5.0.0 in 2D and 3D: in-place vs non-mutating forward agree to Float32 round-off (2e-4; 4e-14 in a Float64 A/B), grad_v0 and grad_dt match central FD to ~1e-3, grad_nu converges to the AD value as the FD step grows (noise-limited by Float32 loss quantization), and TGV kinetic-energy decay matches the analytic rate at the solver's known spatial error. Channel/obstacle code paths (excluded from experiments) are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gdif1Cu1pdCJENe2hNUaF1

PasteurBot · 2026-07-10T16:41:53Z

CLA signatures confirmed

All contributors have signed the Contributor License Agreement.
_{Posted by the CLA Assistant Lite bot.}

agdestein · 2026-07-10T16:53:55Z

@PasteurBot recheck

Following up on the invitation in agdestein/IncompressibleNavierStokes.jl#197 — thanks for including INS.jl! This first PR only corrects a few statements about the solver; the companion upgrade PR is #103. Suggested label: `benchmark:none` — strings and metadata only, no behavior change. ## Changes - **`INS_JL_NO_OBSTACLE` exclusion text.** The current text says the cylinder "cannot be represented in INS.jl" and that its "spectral/LU pressure projection is also periodic-only". Neither is accurate: - `psolver_direct` (LU) assembles the pressure Laplacian for any supported BC combination — it is the general-BC fallback that `default_psolver` selects for non-periodic setups. `psolver_transform` (DCT) supports Dirichlet BCs, as this repo's own channel code uses. Only `psolver_spectral` (FFT) is periodic-only. - Brinkman volume penalization is a momentum forcing term, and a *finite* penalization `−(χ/η)·u` added to the RHS before projection is smooth and stable. What diverges is this tesseract's *hard-zero masking after each projection* (an infinite-penalization clamp the projection cannot see). INS.jl also natively supports the channel configuration (`DirichletBC(f)` inflow, `PressureBC` outflow, periodic lateral BCs). The exclusion itself stays (that route isn't implemented in the tesseract); only the attribution is corrected. - **`discretization: FD` → `FV`.** INS.jl is a staggered finite-volume (energy-conserving) discretization; `FV` is among the documented options for this field. - **"CPU-only" description.** The wrapper runs on CPU, but the library's kernels are KernelAbstractions.jl and run on CUDA (including the spectral pressure solver and the differentiable path). Reworded so readers don't take the cost tables as a solver property; `uses_gpu: false` is kept as a property of this tesseract. No results change; the generated solver/results pages pick these up on the next docs build. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Gdif1Cu1pdCJENe2hNUaF1 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Dion Häfner <dion.haefner@simulation.science>

Fork PRs (e.g. #103) fail benchmark CI because they cannot access the private mosaic/* GHCR packages: the read-only fork token is denied `packages: write` (build push fails) and cannot read the private package either (every image pull returns `manifest unknown`). Both the changed solver's head-SHA image and the unchanged solvers' base-SHA images are unreachable, so the benchmark run jobs fail with "no image found". Split benchmark CI along a trust boundary, mirroring the existing docs-publish workflow_run pattern: - benchmark.yml (on pull_request, fork context, read-only, no secrets) now only plans and hands off. `plan-gate` (renamed from the old `bench-ok` merge-gate job) enforces the label policy and seeds the `bench-ok` commit status: failure on a policy violation, immediate success when nothing will run, else pending. Seeds are best-effort so a fork token that can't write a status doesn't cascade. A new `handoff` job uploads the validated plan outputs as an artifact. - benchmark-execute.yml (new, on workflow_run of "Benchmark", trusted base-repo context) recovers + validates the handoff, then builds and pushes the changed solver image to private GHCR, runs the benchmarks (pulling the private images), renders the docs preview, and posts the authoritative `bench-ok` commit status plus the PR comment / RTD preview. Its `finalize` job always resolves a seeded pending — posting `error` if the handoff can't be recovered on a successful planner run — so the gate can never deadlock in `pending`. The status step is gated on the planner having concluded success, so it never clobbers a correct terminal status set by plan-gate. - build-tesseracts.yml gains `ref`/`image_sha` inputs so the trusted run builds the PR's HEAD and tags the image to match what the benchmark run pulls; the fork-tolerant `continue-on-error` push is removed (both remaining callers hold packages:write). - docs-publish.yml is retired: workflow_run cannot chain, so its RTD + PR-comment steps fold into benchmark-execute's finalize job. The reusable build/benchmark calls drop `secrets: inherit`: the jobs that build and run fork-authored solver code see only the ephemeral GITHUB_TOKEN, keeping RTD_TOKEN / the bot PAT out of reach. Branch protection keeps requiring the `bench-ok` context unchanged — it is now a commit status rather than a job, posted only by the two paths above, so a PR cannot merge until real benchmarks pass. workflow_run changes only take effect once on main, so the fork path must be verified by re-running a fork PR after merge. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

PasteurBot · 2026-07-21T15:35:56Z

📊 View the full benchmark results

The rendered docs preview has every plot for this run (forward accuracy,
gradients, cost, optimization) merged with existing baseline results on main. The summary below reports pass/fail status.

Status diff vs base

Legend · ✅ ok · 🟠 anom · ❌ fail · · missing · 🚫 excluded (permanent — out of score denominator) · ⚪ excluded (work-to-do) · * stale — result predates current benchmark run

4 regression(s) · 1 improvement(s) · 0 other transition(s) · 0 new row(s) · 0 removed row(s) · score 0.98 → 0.94

🔴 Regressions

✅→❌ ns-3d-grid · cost/spatial_cost · INS.jl — all entries failed
✅→❌ ns-3d-grid · cost/temporal_cost · INS.jl — all entries failed
✅→❌ ns-grid · cost/spatial_cost · INS.jl — all entries failed
✅→❌ ns-grid · cost/temporal_cost · INS.jl — all entries failed

🟢 Improvements

🟠→✅ ns-grid · cost/temporal_cost · PhiFlow

Full Mosaic status

Mosaic status

Legend · ✅ ok · 🟠 anom · ❌ fail · · missing · 🚫 excluded (permanent — out of score denominator) · ⚪ excluded (work-to-do) · * stale — result predates current benchmark run

Each solver is run against every experiment in the suite. ok = produced valid results; fail = crashed or returned invalid data; anom = ran successfully but tripped an automated quality check (e.g. poor gradient accuracy, outlier wall-clock time, or diverged optimisation). Thresholds are defined per-problem in the problem config.

problem	ok	anom	fail	excl (perm)	stale	score
`ns-3d-grid`	88	0	2	9	13	🟢 0.94
`ns-grid`	101	6	2	16	15	🟢 0.92
`structural-mesh`	45	0	0	5	0	🟢 1.00
`thermal-mesh`	55	0	0	7	8	🟢 0.96
overall	289	6	4	37	36	🟢 0.94

Failures & anomalies

❌ ns-3d-grid · cost/spatial_cost · INS.jl — all entries failed
❌ ns-3d-grid · cost/temporal_cost · INS.jl — all entries failed
🟠 ns-grid · forward/agreement/tgv · PhiFlow — phiflow's double CenteredGrid↔StaggeredGrid resampling gives 4.18% amplitude damping (ratio=0.9582); cosine=0.9999924 (pattern correct); arithmetic-average output conversion fix worsened error 9×; upstream library change required
🟠 ns-grid · forward/agreement/tgv · XLB — automatic k=9 sub-steps reduce Ma 0.88→0.098 (81× Ma² reduction); errors drop from 0.216-0.278 → 0.026-0.031 (11-24× peers); remaining floor is O(dx²) LBM spatial discretization at N=64, not reducible by further sub-stepping (tested k=9..27); valid=True
🟠 ns-grid · forward/baseline · INS.jl — staggered MAC grid double-interpolation: collocated TGV IC -> staggered faces -> collocated output gives sin^2(pi/N) round-trip error at all N; 35-40x above collocated peers
🟠 ns-grid · forward/baseline · jax-cfd — staggered MAC grid double-interpolation: collocated TGV IC -> staggered faces -> collocated output gives sin^2(pi/N) round-trip error at all N; 35-40x above collocated peers
🟠 ns-grid · forward/baseline · XLB — irreducible O(Ma²) LBM compressibility error floor: at fixed dt=0.01, Ma=u·dt/dx grows with N; at N=128 Ma~0.2 giving ~0.007 error floor (230× peers); anomalous at all N
🟠 ns-grid · forward/tgv_nu_sweep · XLB — same root cause as forward/agreement/tgv — automatic k=9 sub-stepping reduces Ma 0.88→0.098 but residual O(dx²) LBM spatial discretization gives 11-24× peer errors at all nu values (0.0001–0.05); 0.0309 at nu=0.05 is 12.0× peer median; not reducible by further sub-stepping (tested k=9..27); valid=True
❌ ns-grid · cost/spatial_cost · INS.jl — all entries failed
❌ ns-grid · cost/temporal_cost · INS.jl — all entries failed

ns-3d-grid — 16 experiment(s)

experiment	`Exponax`	`INS.jl`	`OpenFOAM`	`PhiFlow`	`PICT`	`Warp-NS`	`XLB`
`forward/agreement`	✅	✅*	✅	✅	✅	✅	✅
`forward/baseline`	✅	✅*	✅	✅	✅	✅	✅
`forward/physical_laws/vs_N`	✅	✅*	✅	✅	✅	✅	✅
`forward/physical_laws/vs_nu`	✅	✅*	✅	✅	✅	✅	✅
`forward/physical_laws/vs_steps`	✅	✅*	✅	✅	✅	✅	✅
`cost/spatial_cost`	✅	❌	✅	✅	✅	✅	✅
`cost/temporal_cost`	✅	❌	✅	✅	✅	✅	✅
`cost/vjp_cost/by_N`	✅	✅*	🚫	✅	✅	✅	✅
`cost/vjp_cost/by_steps`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/fd_check`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/horizon_sweep_limits`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/jacobian_svd`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_nu01`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_steps20`	✅	✅*	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_steps40`	✅	✅*	🚫	✅	✅	✅	✅
`optimization/recovery_constant_ic_bfgs_proj`	✅	✅	🚫	✅	✅	✅	✅

ns-grid — 20 experiment(s)

experiment	`INS.jl`	`jax-cfd`	`OpenFOAM`	`PhiFlow`	`PICT`	`Warp-NS`	`XLB`
`forward/agreement/multimode`	✅*	✅	✅	✅	✅	✅	✅
`forward/agreement/tgv`	✅*	✅	✅	🟠	✅	✅	🟠
`forward/baseline`	🟠	🟠	✅	✅	✅	✅	🟠
`forward/cylinder`	🚫	🚫	✅	✅	✅	🚫	✅
`forward/physical_laws/vs_N`	✅*	✅	✅	✅	✅	✅	✅
`forward/physical_laws/vs_nu`	✅*	✅	✅	✅	✅	✅	✅
`forward/physical_laws/vs_steps`	✅*	✅	✅	✅	✅	✅	✅
`forward/tgv_nu_sweep`	✅*	✅	✅	✅	✅	✅	🟠
`cost/spatial_cost`	❌	✅	✅	✅	✅	✅	✅
`cost/temporal_cost`	❌	✅	✅	✅	✅	✅	✅
`cost/vjp_cost/by_N`	✅*	✅	🚫	✅	✅	✅	✅
`cost/vjp_cost/by_steps`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/fd_check`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/horizon_sweep`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/jacobian_svd`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_nu01`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_steps20`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/jacobian_svd_steps40`	✅*	✅	🚫	✅	✅	✅	✅
`gradient/param_sweep`	✅*	✅	🚫	✅	✅	✅	✅
`optimization/drag_opt`	🚫	🚫	🚫	✅	✅	🚫	✅

structural-mesh — 10 experiment(s)

experiment	`deal.II`	`FEniCS`	`Firedrake`	`JAX-FEM`	`TopOpt.jl`
`forward/agreement`	✅	✅	✅	✅	✅
`forward/baseline`	✅	✅	✅	✅	✅
`forward/physical_laws`	✅	✅	✅	✅	✅
`cost/spatial_cost`	✅	✅	✅	✅	✅
`cost/temporal_cost`	✅	✅	✅	✅	✅
`cost/vjp_cost/by_N`	🚫	✅	✅	✅	✅
`cost/vjp_cost/by_steps`	🚫	✅	✅	✅	✅
`gradient/fd_check`	🚫	✅	✅	✅	✅
`gradient/param_sweep`	🚫	✅	✅	✅	✅
`optimization/topopt`	🚫	✅	✅	✅	✅

thermal-mesh — 14 experiment(s)

experiment	`deal.II`	`FEniCS`	`Firedrake`	`JAX-FEM`	`torch-fem`
`forward/agreement`	✅	✅	✅	✅	✅
`forward/baseline`	✅	✅	✅	✅	✅
`forward/physical_laws`	✅	✅	✅	✅	✅
`forward/source_baseline`	✅	✅	✅	✅	✅
`forward/source_linearity`	✅	✅	✅	✅	✅
`cost/spatial_cost`	✅	✅	✅	✅	✅
`cost/temporal_cost`	✅	✅	✅	✅	✅
`cost/vjp_cost/by_N`	🚫	✅	✅	✅	✅
`cost/vjp_cost/by_steps`	🚫	✅	✅	✅	✅
`gradient/fd_check`	🚫	✅*	✅*	✅	✅
`gradient/param_sweep`	🚫	✅*	✅*	✅	✅
`gradient/source_fd_check`	🚫	✅*	✅*	✅	✅
`gradient/source_width_sweep`	🚫	✅*	✅*	✅	✅
`optimization/conductivity_recovery_bfgs`	🚫	✅	✅	✅	✅

## Summary #113 broke `main`: its trusted `workflow_run` executor checked out the PR's fork code (`ref: <head SHA>`) to build it, but `actions/checkout@v6` refuses a fork PR checkout in a `workflow_run` context by default (the "pwn request" guard), so the build job failed for every solver-changing PR. Opting in (`allow-unsafe-pr-checkout: true`) would run fork-authored `build_base.sh`/Dockerfile with the base repo's `packages: write` token — exactly the vulnerability the guard exists to prevent. This PR **reverts #113** and replaces it with a much smaller, fork-safe flow that never executes fork code in a trusted context and never requires a fork to push to (or privately read from) GHCR. ## How it works now Everything stays on the `pull_request` trigger: - **Build (PR path):** builds the changed solver, `docker save`s it, and uploads a `solver-image-<name>` artifact instead of pushing to GHCR. The GHCR push path is retained only for push-to-main / dispatch (which publishes the baseline `:latest` / `:<sha>` images). - **Run:** `docker load`s those artifacts back to `<name>:latest` — the exact local tag `mosaic run --no-build` resolves — then pulls only the *unchanged* solvers from **public** GHCR at the PR's base SHA, skipping the loaded ones (`pull-solver-images.py --skip-images`). - `bench-ok` stays a normal job-based merge gate (**no branch-protection change**). Docs RTD/preview + PR comment continue via the existing `docs-publish.yml` (`workflow_run`). Fork code only ever runs in the untrusted `pull_request` context (read-only token, no secrets) — the build produces an artifact, the run only consumes images — so no trusted `workflow_run`, commit-status gating, or handoff is needed. ## Required setup (done) The `mosaic/*` GHCR packages must be **public-read** so fork PRs can pull base images and warm the build `cache-from`. This has been flipped to public. Write stays maintainer-only (unchanged). ## Verification - `actionlint` clean (only pre-existing legacy shellcheck/`head.ref` nits, none in new code). - Local proof of the tag seam: `docker save <name>:latest` → `docker load` reproduces exactly `<name>:latest`. - Two rounds of adversarial review; the second confirmed the tag chain against `config.py`/`runner.py` and found no correctness bugs. - `workflow_run`/artifact behavior can only be fully exercised once on `main`; after merge, re-run a fork PR (e.g. #103) to confirm end-to-end. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

dionhaefner · 2026-07-22T12:45:15Z

Seems like I finally fixed CI for forks 💦

@agdestein Does the new INS.jl Tesseract build for you locally? Seeing failures like this in the log:

Tesseract container flamboyant_montalcini failed to start:
Traceback (most recent call last):
  File "/python-env/lib/python3.11/site-packages/filelock/_unix.py", line 85, in _acquire_native
    fd = os.open(self.lock_file, open_flags, open_mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/app/julia_env/lock.pid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python-env/lib/python3.11/site-packages/tesseract_core/runtime/core.py", line 114, in load_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tesseract/tesseract_api.py", line 7, in <module>
    import juliacall
  File "/python-env/lib/python3.11/site-packages/juliacall/__init__.py", line 333, in <module>
    init()
  File "/python-env/lib/python3.11/site-packages/juliacall/__init__.py", line 200, in init
    CONFIG['exepath'] = juliapkg.executable()
                        ^^^^^^^^^^^^^^^^^^^^^
  File "/python-env/lib/python3.11/site-packages/juliapkg/deps.py", line 646, in executable
    resolve()
  File "/python-env/lib/python3.11/site-packages/juliapkg/deps.py", line 464, in resolve
    lock.acquire(timeout=3)
  File "/python-env/lib/python3.11/site-packages/filelock/_api.py", line 1057, in acquire
    self._poll_until_acquired(
  File "/python-env/lib/python3.11/site-packages/filelock/_api.py", line 1218, in _poll_until_acquired
    self._acquire_with_fork_tracking()
  File "/python-env/lib/python3.11/site-packages/filelock/_api.py", line 1276, in _acquire_with_fork_tracking
    self._acquire()
  File "/python-env/lib/python3.11/site-packages/filelock/_unix.py", line 70, in _acquire
    missing_flock = self._acquire_native()
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/python-env/lib/python3.11/site-packages/filelock/_unix.py", line 99, in _acquire_native
    fd = os.open(self.lock_file, open_flags & ~os.O_CREAT, open_mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/app/julia_env/lock.pid'

This hints at a permissions error in /app/julia_env but I wonder if it occurs locally as well.

`import juliacall` resolves juliacall's Julia environment on first use: juliapkg adds PythonCall to /app/julia_env, precompiles it into /app/julia_depot, and guards all of it with a lock file at /app/julia_env/lock.pid (mode 644). Left to itself, that first import is Tesseract's own `tesseract-runtime check`, which the generated Dockerfile runs as uid 9999 *after* every custom build step -- so the lock file and ~700 depot entries end up owned by 9999, past the reach of our `chmod -R 777 /app`. `tesseract serve` starts the container as the *host* uid (1001 on the runner, 1000 on a laptop), which then cannot open the lock file: PermissionError: [Errno 13] Permission denied: '/app/julia_env/lock.pid' The API import fails, and every call reports "Tesseract container ... failed to start". The build-time check itself passes, because it is the one context that runs as 9999 -- so the image builds green and is dead on arrival. Not specific to v5.0.0: this bites any change that invalidates the cached check layer, and topopt-jl has the same latent shape. Do the warm-up as root instead, mirroring the init in tesseract_api.py, and chmod in the same RUN (a separate layer would copy up every file the warm-up just created, +130 MB). The build-time check then finds the environment resolved and juliapkg skips straight past it: nothing under /app is left owned by 9999, and the check layer drops from 134 MB to 1.4 MB. Verified locally by building both ways: as uid 1000 the old image fails with the traceback above and `Tesseract.from_image()` raises "Tesseract failed to start", while the new image passes `check` as uid 1000/4321/9999 and completes apply + vector_jacobian_product unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…art it Same defect as the ins-jl tesseract in the previous commit, and the same one-line shape of fix. juliacall resolves its Julia environment on first import, taking a lock at /app/julia_env/lock.pid (mode 644) and precompiling the PythonCall extensions into /app/julia_depot. That first import is Tesseract's `tesseract-runtime check`, which the generated Dockerfile runs as uid 9999 after every custom build step -- past our `chmod -R 777 /app` -- while `tesseract serve` starts the container as the host uid, which then cannot open the lock file. This image is currently alive only because nothing has invalidated its cached check layer; any change to this tesseract would have shipped a container that dies on every call with PermissionError: [Errno 13] Permission denied: '/app/julia_env/lock.pid' Verified locally by building both ways: the current config produces an image whose `check` fails as uid 1000 with the traceback above, with 203 files under /app owned by 9999 (including the SciMLBase/MLCore PythonCall extension caches), and `Tesseract.from_image()` raises "Tesseract failed to start". With this commit: 1 file (Pkg's usage log), `check` passes as uid 1000/4321/9999, and apply returns compliance=62957.645. Check layer 38.7 MB -> 1.32 MB; image 4.35 GB -> 4.33 GB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

agdestein · 2026-07-25T11:08:26Z

Yes, it reproduces locally, and it turns out not to be specific to v5.0.0.

Root cause. tesseract_api.py does import juliacall, and juliacall resolves its Julia
environment on first import: juliapkg adds PythonCall to /app/julia_env, precompiles it into
/app/julia_depot, and guards all of it with a lock file at /app/julia_env/lock.pid
(mode 644 — filelock's 0o666 & ~umask).

That "first import" is Tesseract's own sanity check,
RUN _TESSERACT_IS_BUILDING=1 /tesseract/entrypoint.sh tesseract-runtime check, which
Dockerfile.base runs as USER 9999:9999 after all custom_build_steps. So the lock file
and ~700 depot entries are owned by 9999, and my RUN chmod -R 777 /app — which runs earlier —
never sees them. The template's own follow-up chmod covers /tesseract, /home/tesseract-user
and /tmp, but not /app.

At runtime, Tesseract.from_image() → engine.serve(), which defaults to
user = f"{os.getuid()}:{os.getgid()}" (engine.py:702). Mosaic never passes user, so the
container runs as the host uid — 1001 on the runner, 1000 on my laptop, never 9999. Hence
EACCES on lock.pid, on both the O_CREAT open and the non-O_CREAT fallback.

The image builds green because the build-time check is the one context that runs as 9999, so it
ships dead on arrival. Local runs hit it too — tesseract serve on my machine fails identically.

Fix: do the warm-up as root in a build step that mirrors the API's Julia init, and chmod in
the same RUN. juliapkg then finds the environment resolved and skips straight past it during
the check.

A/B locally (built both ways, then docker run --user … and Tesseract.from_image):

	before	after
`lock.pid`	`-rw-r--r-- 9999:9999`	`-rwxrwxrwx 0:0`
files under `/app` not root-owned	717	1 (Pkg usage log)
`check` as uid 1000	`PermissionError`	✅
`from_image()` + apply + VJP	`Tesseract failed to start`	✅
build-time check layer	134 MB	1.4 MB
image size	2.56 GB	2.55 GB

TopOpt.jl has the same defect

Worth flagging since it's the other juliacall tesseract: topopt-jl is broken in exactly the
same way, and is only alive because nothing has invalidated its cached check layer yet. I built
it from main to confirm — check fails as uid 1000 with the identical
PermissionError: … '/app/julia_env/lock.pid', 203 files under /app owned by 9999 (including
the SciMLBasePythonCallExt / MLCorePythonCallExt precompile caches), and from_image()
raises "Tesseract failed to start". So any PR that touches that tesseract would have shipped a
dead container, seemingly out of nowhere.

I've included the same one-line fix for it here. After it: 1 file left, check passes as uid
1000/4321/9999, apply returns compliance=62957.645, check layer 38.7 MB → 1.32 MB, image
4.35 GB → 4.33 GB. Happy to split it into its own PR if you'd rather keep this one to INS.jl —
it does mean TopOpt.jl gets rebuilt and re-benchmarked.

The ins_jl cells still need the re-run — the ❌/stale entries in the table above are all this
failure, not solver behaviour.

`mosaic run -s "INS.jl,TopOpt.jl" -p ns-grid` exits 1 with "Unknown solver 'TopOpt.jl'" even though the name is perfectly valid -- it just belongs to structural-mesh. The flat -s CSV is a union set: _apply_solver_filter applies it per problem and documents that "a name that doesn't exist on the current problem is fine (it presumably belongs to another problem)", leaving typo detection to the up-front check. But that check scoped itself to the problems in -p, so the two disagree the moment a run is sharded by problem while -s carries the full cross-problem set -- which is exactly what the benchmark workflow does: one job per (suite, problem), each invoked with `--solvers "<every changed solver>"`. A PR touching tesseracts in two problem domains therefore fails *every* benchmark job before doing any work, in both directions (ns-grid rejects TopOpt.jl, structural-mesh rejects INS.jl). Validate against every registered problem instead, which is what the function's own docstring already claimed ("must exist on at least one problem"). A genuine typo still fails fast, and its "did you mean" suggestion now draws on the full solver list rather than only the problems being run. `problem_list` is no longer needed, so drop it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

agdestein · 2026-07-25T12:23:02Z

Heads-up on the red benchmark jobs — that one isn't the tesseracts, it's a harness bug this PR happens to be the first to trigger. Both images build fine now; every job then dies up-front with

Run SOLVERS="INS.jl,TopOpt.jl"
Unknown solver 'TopOpt.jl'.
Available solvers: Exponax, INS.jl, OpenFOAM, PICT, PhiFlow, Warp-NS, XLB

_validate_solver_csv (mosaic/benchmarks/cli/_helpers.py) checks flat -s names against the solvers of the problems in -p. The benchmark workflow shards one job per (suite, problem) but passes each of them the full changed-solver list, so --problems ns-grid --solvers "INS.jl,TopOpt.jl" sees TopOpt.jl as a typo. It fails in both directions — structural-mesh rejects INS.jl the same way — so a PR touching tesseracts in two problem domains loses every benchmark job before any work happens.

The check contradicts the filter it guards: _apply_solver_filter documents that "a name that doesn't exist on the current problem is fine (it presumably belongs to another problem)", and _validate_solver_csv's own docstring says names must exist on at least one problem — it just looked at the wrong set. Fixed in c1d4053 by validating against every registered problem. A real typo still fails fast, and the "did you mean" suggestion now draws on the whole solver list instead of only the problems being run:

$ mosaic run -p ns-grid -s "INS.jl,TopOptt.jl"
Unknown solver 'TopOptt.jl'. Did you mean 'TopOpt.jl'?

Worth noting the workflow-side alternative doesn't work as-is: --solvers also accepts a per-problem map (ns-grid=INS.jl;structural-mesh=TopOpt.jl), but .github/scripts/pull-solver-images.py splits that flag on , only, so emitting the map form from the workflow would silently pull no images. Happy to move this commit to its own PR against the harness if you'd rather keep this one to the tesseracts.

Local checks on the fix: full test suite 478 passed, ruff clean, and all three sharded invocations (ns-grid, ns-3d-grid, structural-mesh × -s "INS.jl,TopOpt.jl") now get past validation.

agdestein mentioned this pull request Jul 10, 2026

doc: correct INS.jl solver description and exclusion rationale #102

Merged

dionhaefner added the benchmark:solver Benchmark only the modified solver label Jul 10, 2026

Merge branch 'main' into ins-jl-v5-upgrade

5e0d73b

Merge branch 'main' into ins-jl-v5-upgrade

c9e7cf4

dionhaefner mentioned this pull request Jul 21, 2026

ci: make fork-PR benchmarks work via trusted workflow_run #113

Merged

Merge branch 'main' into ins-jl-v5-upgrade

91a1ef9

dionhaefner mentioned this pull request Jul 22, 2026

ci: fork-safe benchmarks via image artifacts (replaces #113) #115

Merged

Merge branch 'main' into ins-jl-v5-upgrade

6829d14

agdestein and others added 2 commits July 25, 2026 13:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf: upgrade INS.jl tesseract to v5.0.0 (native ν gradient, in-place forward, float32)#103

perf: upgrade INS.jl tesseract to v5.0.0 (native ν gradient, in-place forward, float32)#103
agdestein wants to merge 8 commits into
pasteurlabs:mainfrom
agdestein:ins-jl-v5-upgrade

agdestein commented Jul 10, 2026 •

edited

Loading

Uh oh!

PasteurBot commented Jul 10, 2026 •

edited

Loading

Uh oh!

agdestein commented Jul 10, 2026

Uh oh!

PasteurBot commented Jul 21, 2026 •

edited

Loading

Mosaic status

Failures & anomalies

Uh oh!

dionhaefner commented Jul 22, 2026

Uh oh!

agdestein commented Jul 25, 2026

Uh oh!

agdestein commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

agdestein commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes that affect published numbers

Changes with identical results (pure speedup)

Not changed

Validation

Uh oh!

PasteurBot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLA signatures confirmed

Uh oh!

agdestein commented Jul 10, 2026

Uh oh!

PasteurBot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 View the full benchmark results

Status diff vs base

🔴 Regressions

🟢 Improvements

Mosaic status

Failures & anomalies

Uh oh!

dionhaefner commented Jul 22, 2026

Uh oh!

agdestein commented Jul 25, 2026

TopOpt.jl has the same defect

Uh oh!

agdestein commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

agdestein commented Jul 10, 2026 •

edited

Loading

PasteurBot commented Jul 10, 2026 •

edited

Loading

PasteurBot commented Jul 21, 2026 •

edited

Loading