Concurrent outbound app-frame sends race on the raw Stream.WriteAsync (no write-side serialization)

[pedrosakuma]

Summary

FixpClientSession.SendApplicationFrameAsync — and every other outbound send path in the class (Negotiate, Establish, heartbeats, etc.) — writes directly to the underlying Stream with no lock/semaphore/queue serializing concurrent callers:

// FixpClientSession.cs:411-420
public async Task SendApplicationFrameAsync(byte[] buffer, int length, CancellationToken ct)
{
    if (_machine.State != FixpClientState.Established)
        throw new InvalidOperationException(...);
    ...
    await _stream.WriteAsync(buffer.AsMemory(0, length), ct).ConfigureAwait(false);
    if (_options.AutoFlushOutboundFrames)
        await _stream.FlushAsync(ct).ConfigureAwait(false);
}

NextOutboundSeqNum() (line 441) is race-free — it uses Interlocked.Increment to hand out sequence numbers atomically. But sequence assignment and sequence wire order are two different guarantees. In EntryPointClient.CancelAsync/SubmitAsync/ReplaceAsync/MassActionAsync/SubmitCrossAsync:

var seq = _session!.NextOutboundSeqNum();          // (1) atomic reservation
var len = OrderEntryEncoder.Encode...(buffer, ..., seq);  // (2) sync encode
await _session.SendApplicationFrameAsync(buffer, len, ct); // (3) async write — NOT serialized

If two callers race between (1)/(2) and (3), the caller that reserved the higher seq number can win the race to actually complete _stream.WriteAsync first. Depending on the transport:

• Plain NetworkStream: concurrent WriteAsync calls from multiple threads are unsupported and can interleave/corrupt bytes on the wire with no exception.
• SslStream: concurrent writes are explicitly documented as unsupported and can throw NotSupportedException/InvalidOperationException, or hang, depending on the frame overlap.

Either way, the peer (venue/matching gateway) can end up seeing a corrupted frame or the wrong wire order for a given MsgSeqNum — which is exactly the kind of gap/NotApplied class of bug this SDK has already hit once (#208/#209).

Why this matters concretely

B3TradingPlatform's trading-host wraps a single EntryPointClient/session per firm and calls into it from B3EntryPointClientGateway.CancelAsync/SubmitAsync with no additional serialization either (confirmed — only _reconnectLock exists there, guarding reconnects, not sends). The frontend's "Cancel All" panic action (frontend/js/app.js, CANCEL_ALL_CONCURRENCY = 8) fires up to 8 concurrent cancel requests against the same firm session — this is the most likely real-world trigger for hitting this race.

Live symptom observed on pedrosakuma/b3deploy's prod stack (2026-07-07 ~16:09 UTC): a trader used "Cancel All"; the blotter showed Canceled for one order (client-side ack received), but the depth-of-book snapshot did not change — consistent with another concurrently-cancelled order's frame losing the write race and never reaching matching correctly, silently leaving a resting order the trader believed was cancelled. (Could not get further live confirmation — trading-host/matching only log at Info level, which doesn't cover per-order lifecycle detail, so this report is based on the code-level race identified above plus the observed symptom, not a captured wire-level repro yet.)

Suggested fix

Serialize the "reserve seq → encode → write to stream" critical section end-to-end, e.g. a SemaphoreSlim(1,1) (or single-writer Channel<T> drained by one background loop) wrapping every outbound application-frame send (and ideally every outbound frame, including heartbeats/control frames, since those hit the same unsynchronized _stream.WriteAsync call sites). This guarantees wire order matches seq-number assignment order and that no two callers ever call WriteAsync on _stream concurrently.

Suggested test coverage

An integration/conformance test that fires N concurrent CancelAsync/SubmitAsync calls against a real (or TestPeer) venue connection and asserts:

1. All N frames arrive at the peer with strictly increasing, gap-free MsgSeqNum in the order they were encoded.
2. No decode/corruption errors on the peer side.
3. (Regression guard) Repeat with CANCEL_ALL_CONCURRENCY-equivalent concurrency (8) to mirror the real trigger in B3TradingPlatform.

Related: #208/#209 (a different but adjacent class of outbound-sequencing bug in this same session/write path).

[pedrosakuma]

Update: retested live on `pedrosakuma/b3deploy`'s prod stack after this issue was filed. Created a fresh batch of orders and used the trading UI's "Cancel All" (3 concurrent cancels) — all 3 cancelled cleanly, book updated correctly, no ghost orders left behind.

This weakens the case that the originally-reported symptom (a cancelled order not disappearing from the book) was actually hitting the write race described above. More likely explanation: that order was a leftover/ghost from the FIXP session-roll incident (#208), created before pedrosakuma/B3TradingPlatform#577 (SDK 0.16.1) and #582 (gateway session-roll reconcile fix) were deployed — i.e. a pre-existing inconsistent state, not a fresh repro of this issue.

I'm leaving this issue open because the underlying code-level finding is still valid and unaddressed (_stream.WriteAsync with no write-side serialization across SendApplicationFrameAsync and the other outbound send paths in FixpClientSession.cs) — it's a latent bug that a wider burst of concurrent sends, a slower/lossier network, or SslStream specifically could still trigger. But downgrading confidence: we no longer have a live incident actively reproducing it, only the static-analysis argument. Recommend treating as a hardening/latent-bug ticket rather than an active-incident one.

[pedrosakuma]

Worth flagging explicitly: even though this specific incident downgraded to "not confirmed live," the underlying gap (no write-side serialization on the shared `_stream`) is a latent scalability/robustness risk, not just a cosmetic one — it gets more likely to trigger, not less, as usage scales up (more firms, higher order/cancel throughput, larger "cancel all" batches, busier networks with more write latency variance widening the race window). Recommend prioritizing the fix (single-writer serialization on outbound sends) ahead of any load/throughput increase, rather than waiting for a second live incident to confirm it under production pressure.