Skip to content

fix(web): prevent SIGILL crash under sustained gRPC stream by adding WASM yield and backpressure#12784

Open
SBALAVIGNESH123 wants to merge 1 commit into
rerun-io:mainfrom
SBALAVIGNESH123:fix/web-sigill-grpc-stream-backpressure
Open

fix(web): prevent SIGILL crash under sustained gRPC stream by adding WASM yield and backpressure#12784
SBALAVIGNESH123 wants to merge 1 commit into
rerun-io:mainfrom
SBALAVIGNESH123:fix/web-sigill-grpc-stream-backpressure

Conversation

@SBALAVIGNESH123
Copy link
Copy Markdown

Fixes #12723

Root Cause

The gRPC read loop in
e_grpc_client/src/read.rs runs as a tight async loop via wasm_bindgen_futures::spawn_local. Under sustained load, stream.try_next().await resolves instantly (Poll::Ready) on every iteration, starving the browser event loop.

Simultaneously, the
e_quota_channel WASM send path bypasses backpressure entirely — logging a debug warning but sending into an unbounded crossbeam channel regardless of byte budget.

The combination causes unbounded WASM linear memory growth. When memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and Chromium translates the WASM trap to SIGILL, killing the renderer process.

Firefox survives because SpiderMonkey traps the unreachable instruction at the VM level rather than forwarding it as an OS signal.

Changes

1. Periodic yield in gRPC read loop (WASM only)

Every 32 messages, yield to the browser event loop via setTimeout(0). This gives the browser a tick for GC, rendering, and the channel consumer to drain queued messages. When the channel queue exceeds 128 messages, pause ingestion entirely until the consumer catches up.

Uses the same web_sys::Window::set_timeout + js_sys::Promise + wasm_bindgen_futures::JsFuture pattern already established in
e_backoff/src/lib.rs.

2. Hard ceiling on WASM channel send path

When the byte budget is exceeded by 2×, drop the message with a warn_once! instead of sending anyway into the unbounded crossbeam channel. This prevents unbounded memory growth while still allowing small bursts (up to 2× capacity).

Previously, the WASM path logged a debug_once! and sent unconditionally — the byte quota was purely advisory.

Files Changed

File Change
crates/store/re_grpc_client/src/read.rs Add yield_to_browser() helper + periodic yield + consumer backpressure check
crates/utils/re_quota_channel/src/sync/mod.rs Add 2× hard ceiling with message drop on WASM send path
crates/store/re_grpc_client/Cargo.toml Add js-sys and web-sys WASM dependencies (both already in workspace)

Testing

Tested with the reproduction script from #12723:

�ash cargo run --package rerun-cli --no-default-features --features web_viewer -- \ --serve-web --memory-limit 1500MiB

With a sustained 250K point-cloud stream at ~8.75 MB/row. Before fix: Chrome crashes within 30–120 seconds with SIGILL. After fix: Chrome survives indefinitely, with graceful degradation under extreme load.

…WASM yield and backpressure

Fixes rerun-io#12723

The gRPC read loop in re_grpc_client/src/read.rs runs as a tight async
loop via wasm_bindgen_futures::spawn_local. Under sustained load,
stream.try_next().await resolves instantly (Poll::Ready) on every
iteration, starving the browser event loop.

Simultaneously, the re_quota_channel WASM send path bypasses backpressure
entirely - logging a debug warning but sending into an unbounded crossbeam
channel regardless of byte budget.

The combination causes unbounded WASM linear memory growth. When
memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and
Chromium translates the WASM trap to SIGILL, killing the renderer.
Firefox survives because SpiderMonkey traps the unreachable instruction
at the VM level rather than forwarding it as an OS signal.

Changes:

1. Periodic yield in gRPC read loop (WASM only): Every 32 messages,
   yield to the browser via setTimeout(0). This gives the event loop
   a tick for GC, rendering, and channel consumer drain. When the
   channel queue exceeds 128 messages, pause ingestion entirely until
   the consumer catches up.

2. Hard ceiling on WASM channel: When byte budget is exceeded by 2x,
   drop the message with a warning instead of sending anyway. This
   prevents unbounded memory growth while still allowing small bursts.
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chromium SIGILL crash in web viewer under sustained gRPC live-stream load

1 participant