avoid multiple serialize in tcp broadcast#84
Merged
Merged
Conversation
gd-0
approved these changes
Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TcpConnector'sSendBehavior::Broadcastran the caller's serialise closureonce per connection.
ConnectionManager::broadcastlooped over everyconnection and called the per-stream
write_or_enqueue_with(registry, &serialise),which in turn called
serialise_frame→serialise(&mut self.send_buf)againstthat stream's own buffer. So broadcasting to N connections meant N
serialisations plus N independently built frame headers (each with its own
Nanos::now()send-ts).Change
Broadcast now serialises the frame once and writes the identical bytes to
every connection.
stream.rs[len][ts]header layout into a sharedwrite_frame_header()so there is a single source of truth for the wire framing;
serialise_framenow delegates to it.
build_frame_vec()for contiguous backlog allocation from borrowedslices.
TcpStream::write_or_enqueue_shared(registry, &header, &payload): thehappy path does
write_vectored([header, payload])straight from the borrowedslices with no per-connection copy; only a blocked or partially-written
socket allocates a backlog copy (unchanged behaviour — backpressure is
inherently per-connection).
FRAME_HEADER_SIZEis nowpub(crate).connector.rsConnectionManagergainedbcast_header+bcast_payloadscratch buffers.broadcastclearsbcast_payload, runs the closure once into it, stamps asingle shared send-ts, then loops connections calling
write_or_enqueue_shared.Empty payloads short-circuit.
Cost
Behavioural note
A broadcast now carries a single shared send-ts across all connections
instead of a per-connection
Nanos::now(). Receivers see that one timestamp viapoll_with— arguably more correct for measuring fan-out latency.Testing
flux-networkbuilds clean;tcp_broadcast_burst,tcp_multi_client_backpressure,tcp_roundtrip, andtcp_dcacheall pass.