fix(tests): make a stalled integration test fail fast instead of consuming the job - #934
Open
jbachorik wants to merge 1 commit into
Open
fix(tests): make a stalled integration test fail fast instead of consuming the job#934jbachorik wants to merge 1 commit into
jbachorik wants to merge 1 commit into
Conversation
…uming the job A stalled integration test used to burn the entire 30-minute CI budget and report nothing: no failing assertion, no stack trace, and orphaned JVMs left behind. Localising one meant reading raw job logs and comparing timestamps. Four changes, the first three of which apply whatever the underlying cause. Time out an individual test method after five minutes. The whole suite runs in about twelve, and the slowest single test is far below the ceiling, so this fires only on a genuine stall. SEPARATE_THREAD mode is deliberate: the default interrupts the test thread, and a thread blocked in socket I/O ignores an interrupt, so a timeout raised that way would fire and still report nothing. Bound the target shutdown. TestApp.stop() waited on process.waitFor() with no limit, so a target that ignores "done" -- because the agent still holds it, or a probe never detached -- blocked the test forever. It now waits thirty seconds, then destroys forcibly. This is also the only candidate that explains the seven orphaned JVMs seen at CI cleanup; a single lingering worker thread does not. Run probe submission on daemon threads. Tests hand submission to an executor and tear it down with shutdownNow(), which interrupts -- and a thread blocked in socket I/O does not answer. A non-daemon worker in that state keeps the test JVM alive after the class finishes, so the build stalls with nothing to report. The threads are also named now, so a stuck one is identifiable in a dump. Release the reader before closing the protocol. A command loop blocks in a read with no timeout, because V2 negotiation sets one and then restores the previous value, which is unlimited. Closing the protocol streams first means waiting on that in-flight reader to leave. Shutting the input down first makes the read return, so the closes that follow have nothing to wait on. The last one targets a mechanism inferred from code and the CI log shape rather than from a captured dump; twelve local reproduction attempts on macOS all passed. It is safe regardless -- shutdownInput is guarded and inert when no reader is active -- but it should not be treated as a confirmed fix until a stall is caught with a thread dump. Verified: the full suite passes with no timeout firing, and a deliberately stalling test fails at five minutes with a TimeoutException naming the line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the diagnosability half of #932. Does not claim to fix the flake itself.
Problem
A stalled integration test consumed the entire 30-minute CI budget and reported nothing — no
failing assertion, no stack trace, and seven orphaned JVMs at cleanup. Working out which test had
stalled meant reading the raw job log and comparing timestamps.
Changes
Per-method timeout, five minutes (
junit-platform.properties). The whole suite runs in ~12minutes and the slowest single test is well under the ceiling, so this fires only on a genuine
stall.
SEPARATE_THREADmode is deliberate rather than incidental: the default interrupts the testthread, and a thread blocked in socket I/O ignores an interrupt — so a timeout raised that way
would fire and still report nothing, which is the exact failure being fixed.
Bounded target shutdown.
TestApp.stop()waited onprocess.waitFor()with no limit, so atarget that ignores
doneblocked the test indefinitely. Now 30s, thendestroyForcibly(). Thisis also the only candidate that accounts for seven orphaned JVMs — one lingering worker thread
does not.
Daemon threads for probe submission. Tests submit to an executor and tear it down with
shutdownNow(), which interrupts; a thread blocked in socket I/O does not answer. A non-daemonworker in that state keeps the test JVM alive after the class finishes, so the build stalls with
nothing to report. Threads are now named, so a stuck one is identifiable in a dump.
Release the reader before closing the protocol (
Client.close()). The command loop blocks in aread with no timeout — V2 negotiation sets one and restores the previous value, which is unlimited.
Closing the protocol streams first means waiting on that in-flight reader to exit.
sock.shutdownInput()makes the read return, so the closes that follow have nothing to wait on.Confidence
The first three are correct on their own merits and independent of root cause.
The fourth targets a mechanism inferred from code reading and the CI log shape, not from a
captured dump — 12 local reproduction attempts on macOS all passed, so the stall was never caught
in the act. It is safe regardless (
shutdownInputis guarded and inert when no reader is active),but it should not be considered a confirmed fix for #932 until a stall is caught with a thread
dump. The platform gap is the likely reason local reproduction failed:
NioSocketImpl's wake-up ofa blocked reader differs between Linux and macOS, which is precisely the implicated path.
Verification
TimeoutExceptionnaming the stalled line. That probe was removed; it is not part of this diff.The second check matters — "no timeout fired" on its own is equally consistent with a timeout that
is misconfigured and cannot fire.
🤖 Generated with Claude Code
This change is