What happened?
On the macOS amber-integration job, a LoopIntegrationSpec workflow intermittently stops making progress mid-run with no error logged, burns the harness completion deadline, and is re-run by withRetry (failing the suite when both attempts hang). Observed in 3 of 4 attempts across two runs on 2026-07-29; never observed on ubuntu.
Log signature (job log, WARN backstop level): steady Python-worker spawns every ~2s, then total silence for ~the full deadline, then the retry's spawns resume. In the failed run an EndHandler worker still has unprocessed messages race (recovered ~23s before the silence) appears nearby, but the same race also self-heals in passing runs and locally, so it is not confirmed as the cause. RegionExecutionManager region-termination retries never log a second attempt during the silence, pointing at an await with no timeout between iterations (e.g. a termination or scheduling future that never resolves).
Expected: a healthy loop workflow completes in well under a minute; a broken one fails with a diagnosable error instead of silently idling.
How to reproduce?
Re-run build / amber-integration (macos-latest, 17) on any PR until it exceeds ~13 min; the suite hangs most runs (e.g. runs 30479437487 and 30478651425 attempt 2 on 2026-07-29). Does not reproduce locally on an M-series Mac (JDK 17, venv with amber requirements, regenerated proto bindings): 4/4 pass in ~1.5 min with zero hangs, so the window is likely specific to the 3-core CI runner's timing. #7053 trims the suite and deadline so each hang costs ~2 min instead of ~4.
Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
No response
What browsers are you seeing the problem on?
No response
Relevant log output
18:45:19 WARN [WF6-LimitOpDesc-...-main-0] Received EndHandler before all messages are processed. Unprocessed: ControlInvocation(queryStatistics, ...)
18:45:19 ERROR [COORDINATOR] AsyncRPCClient - received error from Worker:WF6-LimitOpDesc-...
18:45:19 WARN [COORDINATOR] RegionExecutionManager - Failed to terminate region 1 on attempt 1 of 150. Retrying in 200 ms.
18:45:42 <last Python worker spawn; total silence for ~171s until the deadline fires and the test is retried>
What happened?
On the macOS
amber-integrationjob, aLoopIntegrationSpecworkflow intermittently stops making progress mid-run with no error logged, burns the harness completion deadline, and is re-run bywithRetry(failing the suite when both attempts hang). Observed in 3 of 4 attempts across two runs on 2026-07-29; never observed on ubuntu.Log signature (job log, WARN backstop level): steady Python-worker spawns every ~2s, then total silence for ~the full deadline, then the retry's spawns resume. In the failed run an
EndHandlerworker still has unprocessed messagesrace (recovered ~23s before the silence) appears nearby, but the same race also self-heals in passing runs and locally, so it is not confirmed as the cause.RegionExecutionManagerregion-termination retries never log a second attempt during the silence, pointing at an await with no timeout between iterations (e.g. a termination or scheduling future that never resolves).Expected: a healthy loop workflow completes in well under a minute; a broken one fails with a diagnosable error instead of silently idling.
How to reproduce?
Re-run
build / amber-integration (macos-latest, 17)on any PR until it exceeds ~13 min; the suite hangs most runs (e.g. runs 30479437487 and 30478651425 attempt 2 on 2026-07-29). Does not reproduce locally on an M-series Mac (JDK 17, venv with amber requirements, regenerated proto bindings): 4/4 pass in ~1.5 min with zero hangs, so the window is likely specific to the 3-core CI runner's timing. #7053 trims the suite and deadline so each hang costs ~2 min instead of ~4.Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
No response
What browsers are you seeing the problem on?
No response
Relevant log output