fix(web): prevent SIGILL crash under sustained gRPC stream by adding WASM yield and backpressure#12784
Open
SBALAVIGNESH123 wants to merge 1 commit into
Conversation
…WASM yield and backpressure Fixes rerun-io#12723 The gRPC read loop in re_grpc_client/src/read.rs runs as a tight async loop via wasm_bindgen_futures::spawn_local. Under sustained load, stream.try_next().await resolves instantly (Poll::Ready) on every iteration, starving the browser event loop. Simultaneously, the re_quota_channel WASM send path bypasses backpressure entirely - logging a debug warning but sending into an unbounded crossbeam channel regardless of byte budget. The combination causes unbounded WASM linear memory growth. When memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and Chromium translates the WASM trap to SIGILL, killing the renderer. Firefox survives because SpiderMonkey traps the unreachable instruction at the VM level rather than forwarding it as an OS signal. Changes: 1. Periodic yield in gRPC read loop (WASM only): Every 32 messages, yield to the browser via setTimeout(0). This gives the event loop a tick for GC, rendering, and channel consumer drain. When the channel queue exceeds 128 messages, pause ingestion entirely until the consumer catches up. 2. Hard ceiling on WASM channel: When byte budget is exceeded by 2x, drop the message with a warning instead of sending anyway. This prevents unbounded memory growth while still allowing small bursts.
Contributor
There was a problem hiding this comment.
Hi! Thanks for opening this pull request.
Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12723
Root Cause
The gRPC read loop in
e_grpc_client/src/read.rs runs as a tight async loop via wasm_bindgen_futures::spawn_local. Under sustained load, stream.try_next().await resolves instantly (Poll::Ready) on every iteration, starving the browser event loop.
Simultaneously, the
e_quota_channel WASM send path bypasses backpressure entirely — logging a debug warning but sending into an unbounded crossbeam channel regardless of byte budget.
The combination causes unbounded WASM linear memory growth. When memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and Chromium translates the WASM trap to SIGILL, killing the renderer process.
Firefox survives because SpiderMonkey traps the unreachable instruction at the VM level rather than forwarding it as an OS signal.
Changes
1. Periodic yield in gRPC read loop (WASM only)
Every 32 messages, yield to the browser event loop via setTimeout(0). This gives the browser a tick for GC, rendering, and the channel consumer to drain queued messages. When the channel queue exceeds 128 messages, pause ingestion entirely until the consumer catches up.
Uses the same web_sys::Window::set_timeout + js_sys::Promise + wasm_bindgen_futures::JsFuture pattern already established in
e_backoff/src/lib.rs.
2. Hard ceiling on WASM channel send path
When the byte budget is exceeded by 2×, drop the message with a warn_once! instead of sending anyway into the unbounded crossbeam channel. This prevents unbounded memory growth while still allowing small bursts (up to 2× capacity).
Previously, the WASM path logged a debug_once! and sent unconditionally — the byte quota was purely advisory.
Files Changed
Testing
Tested with the reproduction script from #12723:
�ash cargo run --package rerun-cli --no-default-features --features web_viewer -- \ --serve-web --memory-limit 1500MiBWith a sustained 250K point-cloud stream at ~8.75 MB/row. Before fix: Chrome crashes within 30–120 seconds with SIGILL. After fix: Chrome survives indefinitely, with graceful degradation under extreme load.