Skip to content

Harden Agent Host E2E and protocol integration tests#327421

Merged
roblourens merged 2 commits into
mainfrom
roblou/agents/flaky-tests-hardening-e2e-protocol
Jul 25, 2026
Merged

Harden Agent Host E2E and protocol integration tests#327421
roblourens merged 2 commits into
mainfrom
roblou/agents/flaky-tests-hardening-e2e-protocol

Conversation

@roblourens

Copy link
Copy Markdown
Member

Hardens the Agent Host E2E and protocol integration tests following the failures reported in this PR comment.

Changes

  • use platform-aware protocol operation timeouts with extra Windows and coverage headroom
  • register pending JSON-RPC calls before sending to avoid fast-response races
  • centralize bounded graceful server shutdown with forced termination fallback
  • await server shutdown across protocol suites to avoid process overlap
  • track and clean up secondary WebSocket clients on failure paths
  • replace wall-clock sleeps with protocol barriers and turn-specific notification matching
  • add retry-backed temporary resource cleanup for Windows

Validation

  • npm run typecheck-client
  • npm run precommit
  • Agent Host protocol integration tests: 115 passing, 1 pending
  • reported terminal E2E cases for Claude: 2 passing
  • reported terminal E2E cases for Copilot: 2 passing
  • final code review: no findings

(Written by Copilot)

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 24, 2026 23:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Hardens Agent Host E2E + protocol integration tests to reduce flakiness (timeouts/races), improve shutdown/cleanup reliability, and avoid cross-suite process overlap.

Changes:

  • Introduces platform/coverage-aware protocol operation timeouts and removes wall-clock sleeps in favor of protocol barriers.
  • Registers pending JSON-RPC calls before sending to avoid fast-response races; improves client cleanup paths.
  • Centralizes graceful server shutdown with a bounded timeout and SIGKILL fallback; updates protocol suites to await shutdown.
Show a summary per file
File Description
src/vs/platform/agentHost/test/node/serverIntegrationTestHelpers.ts Adds protocol timeouts, fixes JSON-RPC call ordering race, and centralizes bounded server shutdown (stopServer).
src/vs/platform/agentHost/test/node/protocol/turnExecution.integrationTest.ts Replaces sleeps with turn-specific notification matching and uses shared stopServer.
src/vs/platform/agentHost/test/node/protocol/toolApproval.integrationTest.ts Switches suite teardown to await stopServer with extended timeout.
src/vs/platform/agentHost/test/node/protocol/sessionLifecycle.integrationTest.ts Tracks/cleans secondary clients; replaces sleeps with protocol barriers; awaits server shutdown.
src/vs/platform/agentHost/test/node/protocol/sessionFeatures.integrationTest.ts Switches suite teardown to await stopServer with extended timeout.
src/vs/platform/agentHost/test/node/protocol/sessionDiffs.integrationTest.ts Switches suite teardown to await stopServer with extended timeout.
src/vs/platform/agentHost/test/node/protocol/sessionConfig.integrationTest.ts Replaces bespoke stdin-close shutdown with centralized stopServer.
src/vs/platform/agentHost/test/node/protocol/resourceOperations.integrationTest.ts Adds retry-backed temp dir cleanup (Windows-friendly) and secondary client cleanup; awaits server shutdown.
src/vs/platform/agentHost/test/node/protocol/otlpLogs.integrationTest.ts Updates timeouts and suite teardown to await stopServer.
src/vs/platform/agentHost/test/node/protocol/networkDiagnostics.integrationTest.ts Awaits target close and server shutdown with extended timeout.
src/vs/platform/agentHost/test/node/protocol/multiClient.integrationTest.ts Tracks/cleans secondary clients; replaces sleeps with protocol barriers; awaits server shutdown.
src/vs/platform/agentHost/test/node/protocol/handshake.integrationTest.ts Improves raw client cleanup with try/finally; updates timeouts and server teardown.
src/vs/platform/agentHost/test/node/protocol/clientTools.integrationTest.ts Switches suite teardown to await stopServer with extended timeout.
src/vs/platform/agentHost/test/node/protocol/agentHostServer.integrationTest.ts Switches suite teardown to await stopServer with extended timeout.
src/vs/platform/agentHost/test/node/e2e/harness/agentHostE2ETestHarness.ts Removes local shutdown helper and reuses shared stopServer.

Review details

Comments suppressed due to low confidence (1)

src/vs/platform/agentHost/test/node/serverIntegrationTestHelpers.ts:1

  • In the catch block, reject(error) can reject with a non-Error value (unknown), while the rest of this helper (and IPendingCall.reject) assumes Error. To keep error handling consistent and predictable across tests, coerce error into an Error instance before rejecting (e.g., error instanceof Error ? error : new Error(String(error))).
/*---------------------------------------------------------------------------------------------
  • Files reviewed: 15/15 changed files
  • Comments generated: 2
  • Review effort level: Low

Comment thread src/vs/platform/agentHost/test/node/serverIntegrationTestHelpers.ts
Comment thread src/vs/platform/agentHost/test/node/protocol/turnExecution.integrationTest.ts Outdated
(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@roblourens
roblourens marked this pull request as ready for review July 25, 2026 17:14
@roblourens
roblourens enabled auto-merge (squash) July 25, 2026 17:15
@roblourens
roblourens merged commit 3e7bfbf into main Jul 25, 2026
29 checks passed
@roblourens
roblourens deleted the roblou/agents/flaky-tests-hardening-e2e-protocol branch July 25, 2026 19:10
@vs-code-engineering vs-code-engineering Bot added this to the 1.131.0 milestone Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants