Skip to content

Raise reconcile_state! kernels to fix GPU segfault#408

Open
xkykai wants to merge 3 commits into
mainfrom
glw/reactant-gpu-reconcile-raise
Open

Raise reconcile_state! kernels to fix GPU segfault#408
xkykai wants to merge 3 commits into
mainfrom
glw/reactant-gpu-reconcile-raise

Conversation

@xkykai

@xkykai xkykai commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Closes #407.

On the Reactant GPU backend, reconcile_state!'s plain @jit lowers the coupled model's kernel launch through EnzymeXLA's kernel-call ABI rewrite (rewriteKernelCallABIgpu.LaunchFuncOp), which segfaults on the large coupled update_state! kernel. This happens at model construction, before any physics runs.

This compiles reconcile_state! with raise=true, so the kernels are raised to StableHLO instead of lowered to a GPU launch, avoiding the crashing path. It's also the lowering the differentiable examples already use for the gradient, so the forward and adjoint passes share the same path.

 function reconcile_state!(model::ReactantESM)
-    @jit Oceananigans.initialize!(model.interfaces.exchanger, model)
-    @jit Oceananigans.TimeSteppers.update_state!(model)
+    @jit raise=true Oceananigans.initialize!(model.interfaces.exchanger, model)
+    @jit raise=true Oceananigans.TimeSteppers.update_state!(model)
     return nothing
 end

Verification (NVIDIA Tesla T4, Reactant 0.2.268)

  • The MWE in Coupled model segfaults on the Reactant GPU backend in reconcile_state! #407 no longer segfaults; the coupled model builds on GPU.
  • The full differentiable dry-layer slab example runs end-to-end on GPU (build + reverse-mode gradient of the 841-step loop): ∂T/∂𝒫̇ = -1.02e3 K/(kg m⁻² s⁻¹) ≈ -0.047 K/mm, matching the CPU-documented ~0.04 K/mm sign and magnitude.
  • Runtime is unaffected (XLA re-vectorizes the raised StableHLO); the only cost is a somewhat longer compile.

🤖 Generated with Claude Code

@dkytezab @glwagner

On the Reactant GPU backend, reconcile_state!'s plain `@jit` lowers the coupled
model's kernel launch through EnzymeXLA's kernel-call ABI rewrite
(rewriteKernelCallABI -> gpu.LaunchFuncOp), which segfaults on the large coupled
update_state! kernel. Compile with `raise=true` so the kernels are raised to
StableHLO instead of lowered to a GPU launch, avoiding the crashing path. This is
also the lowering the differentiable examples use for the gradient.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ext/NumericalEarthReactantExt.jl 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@dkytezab

dkytezab commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

I this is a reasonably fine workaround for the moment, but we should find the underlying cause of the segfault and address it there.

Added a TODO comment regarding issue resolution for kernel launch segfaults in the reconcile_state! function.
@xkykai

xkykai commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

ok, will close this once tests pass as a temporary workaround.
I've added a TODO so if the resolution of EnzymeAD/Reactant.jl#3026 solves this problem we can roll back the fix then.

@xkykai

xkykai commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

for the record tests fail right now because of EnzymeAD/Reactant.jl#3028

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coupled model segfaults on the Reactant GPU backend in reconcile_state!

3 participants