Flaky test: RemotingSpec "should resume the outbound reader when passive read handoff fails" on Scala 3

[He-Pin]

Description

The test RemotingSpec — "should resume the outbound reader when passive read handoff fails" intermittently times out on the Scala 3 CI job.

Failure

java.lang.AssertionError: assertion failed: timeout (6 seconds) during expectMsg while waiting for after-handoff-failure
  at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
  at org.apache.pekko.testkit.TestKitBase.expectMsg_internal(TestKit.scala:471)
  at org.apache.pekko.testkit.TestKitBase.expectMsg(TestKit.scala:448)
  at org.apache.pekko.remote.classic.RemotingSpec.f$proxy19$1(RemotingSpec.scala:797)

Location

remote/src/test/scala/org/apache/pekko/remote/classic/RemotingSpec.scala:797

outboundRemoteHandle.association.write(payload("after-handoff-failure"))
receiverProbe.expectMsg("after-handoff-failure")  // <-- times out here

Observed in

• PR stream: deprecate Source.queue overloads that materialize SourceQueueWithComplete #3095 (unrelated changes — Source.queue deprecation): CI run
• The test was added by PR Resume classic remoting reader after failed handoff #3205

Analysis

The test sets up a passive read-only inbound association, sends a broken packet to trigger disassociation of the read-only endpoint, then expects the outbound writer to resume delivering messages. The expectMsg at line 797 times out, suggesting the outbound reader did not resume after the passive read handoff failure.

This is a timing-sensitive test involving multiple transport associations and handoffs. The 6-second timeout may be insufficient under CI load, or the resume logic may have a race condition that manifests more often on Scala 3 (possibly due to different compilation/optimization characteristics affecting thread scheduling).

Possible fixes

1. Increase the expectMsg timeout with a dilated factor
2. Investigate whether the resume logic in the endpoint reader has a race condition
3. Add retry or awaitAssert around the assertion to tolerate transient timing