Skip to content

fix(tests): make a stalled integration test fail fast instead of consuming the job - #934

Open
jbachorik wants to merge 1 commit into
developfrom
agent/harden-integration-test-teardown
Open

fix(tests): make a stalled integration test fail fast instead of consuming the job#934
jbachorik wants to merge 1 commit into
developfrom
agent/harden-integration-test-teardown

Conversation

@jbachorik

@jbachorik jbachorik commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Addresses the diagnosability half of #932. Does not claim to fix the flake itself.

Problem

A stalled integration test consumed the entire 30-minute CI budget and reported nothing — no
failing assertion, no stack trace, and seven orphaned JVMs at cleanup. Working out which test had
stalled meant reading the raw job log and comparing timestamps.

Changes

Per-method timeout, five minutes (junit-platform.properties). The whole suite runs in ~12
minutes and the slowest single test is well under the ceiling, so this fires only on a genuine
stall.

SEPARATE_THREAD mode is deliberate rather than incidental: the default interrupts the test
thread, and a thread blocked in socket I/O ignores an interrupt — so a timeout raised that way
would fire and still report nothing, which is the exact failure being fixed.

Bounded target shutdown. TestApp.stop() waited on process.waitFor() with no limit, so a
target that ignores done blocked the test indefinitely. Now 30s, then destroyForcibly(). This
is also the only candidate that accounts for seven orphaned JVMs — one lingering worker thread
does not.

Daemon threads for probe submission. Tests submit to an executor and tear it down with
shutdownNow(), which interrupts; a thread blocked in socket I/O does not answer. A non-daemon
worker in that state keeps the test JVM alive after the class finishes, so the build stalls with
nothing to report. Threads are now named, so a stuck one is identifiable in a dump.

Release the reader before closing the protocol (Client.close()). The command loop blocks in a
read with no timeout — V2 negotiation sets one and restores the previous value, which is unlimited.
Closing the protocol streams first means waiting on that in-flight reader to exit.
sock.shutdownInput() makes the read return, so the closes that follow have nothing to wait on.

Confidence

The first three are correct on their own merits and independent of root cause.

The fourth targets a mechanism inferred from code reading and the CI log shape, not from a
captured dump — 12 local reproduction attempts on macOS all passed, so the stall was never caught
in the act. It is safe regardless (shutdownInput is guarded and inert when no reader is active),
but it should not be considered a confirmed fix for #932 until a stall is caught with a thread
dump. The platform gap is the likely reason local reproduction failed: NioSocketImpl's wake-up of
a blocked reader differs between Linux and macOS, which is precisely the implicated path.

Verification

  • Full integration suite: 58 tests pass, no timeout fired, 12m34s on 17.0.19-tem.
  • Timeout proven to actually fire: a temporary test sleeping for ten minutes failed at five with
    TimeoutException naming the stalled line. That probe was removed; it is not part of this diff.

The second check matters — "no timeout fired" on its own is equally consistent with a timeout that
is misconfigured and cannot fire.

🤖 Generated with Claude Code


This change is Reviewable

…uming the job

A stalled integration test used to burn the entire 30-minute CI budget and
report nothing: no failing assertion, no stack trace, and orphaned JVMs left
behind. Localising one meant reading raw job logs and comparing timestamps.

Four changes, the first three of which apply whatever the underlying cause.

Time out an individual test method after five minutes. The whole suite runs in
about twelve, and the slowest single test is far below the ceiling, so this
fires only on a genuine stall. SEPARATE_THREAD mode is deliberate: the default
interrupts the test thread, and a thread blocked in socket I/O ignores an
interrupt, so a timeout raised that way would fire and still report nothing.

Bound the target shutdown. TestApp.stop() waited on process.waitFor() with no
limit, so a target that ignores "done" -- because the agent still holds it, or a
probe never detached -- blocked the test forever. It now waits thirty seconds,
then destroys forcibly. This is also the only candidate that explains the seven
orphaned JVMs seen at CI cleanup; a single lingering worker thread does not.

Run probe submission on daemon threads. Tests hand submission to an executor and
tear it down with shutdownNow(), which interrupts -- and a thread blocked in
socket I/O does not answer. A non-daemon worker in that state keeps the test JVM
alive after the class finishes, so the build stalls with nothing to report. The
threads are also named now, so a stuck one is identifiable in a dump.

Release the reader before closing the protocol. A command loop blocks in a read
with no timeout, because V2 negotiation sets one and then restores the previous
value, which is unlimited. Closing the protocol streams first means waiting on
that in-flight reader to leave. Shutting the input down first makes the read
return, so the closes that follow have nothing to wait on.

The last one targets a mechanism inferred from code and the CI log shape rather
than from a captured dump; twelve local reproduction attempts on macOS all
passed. It is safe regardless -- shutdownInput is guarded and inert when no
reader is active -- but it should not be treated as a confirmed fix until a
stall is caught with a thread dump.

Verified: the full suite passes with no timeout firing, and a deliberately
stalling test fails at five minutes with a TimeoutException naming the line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 11:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants