Scope cluster reply serialization failures and peer-delivered defects to their own request

[marbemac]

Type

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

If a volatile entity reply can't be serialized, the failure currently takes down the whole runner-to-runner connection. RunnerServer dies with the Reply.serialize error, RpcServer broadcasts it as a connection-level Defect frame (killing every in-flight request on that connection), and Runners.makeRpc treats the delivered defect as RunnerUnavailable. Sharding.sendOutgoing then re-sends the same request id every sendRetryInterval, and since the entity already processed it, the caller gets AlreadyProcessingMessage for a request that actually succeeded — only the reply couldn't be encoded.

Each step is reasonable on its own; it's the combination that turns one unencodable reply into a connection-wide failure and a retry loop. This PR changes all three steps, so no two of them can recombine this way:

• RunnerServer now responds to the affected request with a defect reply (ReplyWithContext.fromDefect) when reply serialization fails, the same way request serialization failures are already handled in Runners. The Stream handler also ends its queue, since the fallback reply is terminal.
• Runners.makeRpc only converts transport failures (RpcClientError) to RunnerUnavailable. Defects delivered by the peer resolve the request with the real error instead of being re-sent. Persisted requests still map to RunnerUnavailable, because their reply can be recovered from storage.
• The runner protocol servers (RunnerServer.layer, HttpRunner.toHttpEffect{,Websocket}) set disableFatalDefects: true, so other handler defects also stay scoped to a single request. From what I can find nothing depends on connection-fatal defects on this channel (a Defect frame doesn't close the socket, and reconnection is driven by socket errors), and the entity level already works this way — but if connection-fatal defects are intentional here, this change can be dropped without affecting the other two.

Tests: test/cluster/Runners.test.ts covers both fixes at the unit level (also the first tests for the RunnerUnavailable / AlreadyProcessingMessage paths), SocketRunner.test.ts adds a two-runner repro over real sockets checking that a sibling in-flight request survives, and RpcServer.test.ts gets a disableFatalDefects isolation test.

Related

Same direction as #2457, which did this for unknown request tags.

Summary by CodeRabbit

• Bug Fixes
  • Isolated reply serialization failures to the affected request instead of terminating the shared runner connection.
  • Ensured concurrent requests continue processing when a sibling request fails.
  • Improved handling of peer and transport failures, including clearer recovery behavior for persisted requests.
  • Preserved active streams when unrelated requests encounter fatal defects or routing errors.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3d0b2d7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 28 packages

Name	Type
effect	Patch
@effect/opentelemetry	Patch
@effect/platform-browser	Patch
@effect/platform-bun	Patch
@effect/platform-node-shared	Patch
@effect/platform-node	Patch
@effect/vitest	Patch
@effect/ai-anthropic	Patch
@effect/ai-openai-compat	Patch
@effect/ai-openai	Patch
@effect/ai-openrouter	Patch
@effect/atom-react	Patch
@effect/atom-solid	Patch
@effect/atom-vue	Patch
@effect/sql-clickhouse	Patch
@effect/sql-d1	Patch
@effect/sql-libsql	Patch
@effect/sql-mssql	Patch
@effect/sql-mysql2	Patch
@effect/sql-pg	Patch
@effect/sql-pglite	Patch
@effect/sql-sqlite-bun	Patch
@effect/sql-sqlite-do	Patch
@effect/sql-sqlite-node	Patch
@effect/sql-sqlite-react-native	Patch
@effect/sql-sqlite-wasm	Patch
@effect/docgen	Patch
@effect/openapi-generator	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 548e5d52-2c46-47e0-a7df-3725452da27d

📥 Commits

Reviewing files that changed from the base of the PR and between 4ac7e8b and 3d0b2d7.

📒 Files selected for processing (7)

• .changeset/fix-cluster-reply-defect-isolation.md
• packages/effect/src/unstable/cluster/HttpRunner.ts
• packages/effect/src/unstable/cluster/RunnerServer.ts
• packages/effect/src/unstable/cluster/Runners.ts
• packages/effect/test/cluster/Runners.test.ts
• packages/platform-node/test/RpcServer.test.ts
• packages/platform-node/test/cluster/SocketRunner.test.ts

---

📝 Walkthrough

Walkthrough

Cluster runner reply serialization and defect handling now isolate failures to their originating request or stream. Server adapters disable fatal defects, malformed replies become terminal defect replies, and RPC clients distinguish volatile, persisted, and transport failures.

Changes

Cluster reply defect isolation

Layer / File(s)	Summary
Request-scoped server reply handling packages/effect/src/unstable/cluster/RunnerServer.ts, packages/effect/src/unstable/cluster/HttpRunner.ts, .changeset/...	Reply serialization catches MalformedMessage, emits request-scoped defect replies, terminates affected streams, and disables fatal defects for multiplexed runner connections and HTTP adapters.
RPC client defect mapping packages/effect/src/unstable/cluster/Runners.ts, packages/effect/test/cluster/Runners.test.ts	Transport errors map to RunnerUnavailable; peer defects become failure replies for volatile requests and RunnerUnavailable for persisted requests, with coverage for both paths.
Connection and sibling-request isolation validation packages/platform-node/test/RpcServer.test.ts, packages/platform-node/test/cluster/SocketRunner.test.ts	Tests verify that failing calls, malformed replies, and fatal handler defects do not interrupt concurrent ticker streams or sibling requests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: bug

Suggested reviewers: imax153

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[marbemac]

Least sure about this one — the other two changes fix the issue without it, this just stops other handler defect classes from failing every in-flight request on the shared connection. Happy to drop it if connection-fatal defects are intentional on this channel (same with the other 2 flips of disableFatalDefects to true).