Skip to content

feat: Copilot-in-VS-Code opt-in E2E validation suite#484

Open
AlexanderZ-Band wants to merge 8 commits into
mainfrom
feat/intel-github-copilot-in-vs-code-mvp-integration-l0-INT-1110
Open

feat: Copilot-in-VS-Code opt-in E2E validation suite#484
AlexanderZ-Band wants to merge 8 commits into
mainfrom
feat/intel-github-copilot-in-vs-code-mvp-integration-l0-INT-1110

Conversation

@AlexanderZ-Band

@AlexanderZ-Band AlexanderZ-Band commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

Opt-in E2E validation suite for the GitHub Copilot in VS Code surface (L0–L4 bar): drives a real signed-in VS Code window through the code chat CLI (agent mode), connects Copilot to Band via a harness-owned band-mcp SSE server, and asserts everything Band-side with the existing baseline toolkit (ReplyCapture + judge) — the VS Code UI is never scraped.

Inherently semi-manual (a signed-in GUI window; one-time trust/approval clicks on first run, unattended reruns after) — can never run in CI. The README states this up front; a directory-scoped Claude Code skill (/vscode-chat-e2e) walks the prerequisites and first-run dialogs.

Results (live, macOS / VS Code 1.130.0 / band-mcp 1.3.2)

6/6 cells pass + L4 N/A — artifacts committed: artifacts/int-1110-copilot-vscode-scorecard.json + .meta.json (environment evidence sidecar). Evidence + screenshot on INT-1110.

Cell Level Result
participation reply round-trip L0 pass
original functions retained (file + roster) L1 pass
multi-participant echo peer L0 pass
recall across chat sessions L2 pass
no leak between rooms L2 pass
bridge restart: recall + function L3 pass
usage accounting L4 n/a — surface exposes no per-turn usage/billing signal

Why

All recorded Copilot evidence exercises the CLI (copilot --acp). The VS Code extension surface had never been driven; this suite closes that gap with reproducible, recorded evidence.

Design

  • Turn protocol: the surface has no inbound channel (no push, and band-mcp exposes no message-read tool), so every turn is driver-initiated — the prompt relays the triggering room message and pins the room chat_id. Mirrors real usage.
  • Recall path: band memory tools (band_store_memory/band_list_memories, platform-persisted, session-independent) — band-mcp 1.3.2 has no room-history tool (verified live). Records are room-keyed; seeding uses a Band-side store barrier (SurfaceRoom.remember) so a store failure is distinguishable from a retrieval failure.
  • Prompt/dialog model: deliberately no chat.tools.global.autoApprove (machine-wide "YOLO mode"). Instead: stable workspace path + fixed band-mcp port keep .vscode/mcp.json byte-identical so VS Code's remembered MCP trust holds; chat.autoReply + remembered per-tool "Always allow" cover the rest. First run = a few one-time clicks; reruns unattended.
  • Gate: VSCODE_CHAT_TESTS_ENABLED + vscode_chat marker, via the new declarative GATED_MARKERS table in root conftest (also covers e2e/docker_build/sandbox). Pure parts (prompt template, workspace scaffolding, server lifecycle, scorecard, version capture) are unit-tested in tests/framework_conformance/ on every PR.
  • Seam: PromptDriver protocol so a @vscode/test-electron backend can slot in later without test changes.

Pieces

File Purpose
tests/e2e/vscode/rooms.py SurfaceRoom — the cells' one intent object (turns, memory seeding, reply waits)
tests/e2e/vscode/driver.py code chat driver, vscode_window context manager, turn_prompt, preflight, version evidence
tests/e2e/vscode/server.py restartable band-mcp SSE subprocess (holds the provisioned agent identity; rejects occupied port, never orphans a failed start)
tests/e2e/vscode/workspace.py .vscode/mcp.json + chat settings scaffolding (chat.mcp.access, chat.autoReply)
tests/e2e/vscode/scorecard.py suite-local collector reusing the baseline row schema (+ fixed L4 usage N/A row)
tests/e2e/vscode/settings.py pydantic-settings knobs (CODE_COMMAND, BAND_MCP_COMMAND, BAND_MCP_PORT, …)
tests/e2e/vscode/test_copilot_chat.py the six live cells
tests/e2e/vscode/README.md runbook (leads with the semi-manual nature; prerequisites, one-time dialogs, troubleshooting)
tests/e2e/vscode/.claude/skills/vscode-chat-e2e/SKILL.md directory-scoped skill running the whole flow

Verified live during development

  • band-mcp 1.3.2: SSE /sse endpoint, --tools memory opt-in group, BAND_AGENT_KEY/BAND_BASE_URL env, ALLOWED_HOSTS for loopback SSE, no message-read/history tool (tool list captured over stdio MCP client)
  • VS Code 1.130: .vscode/mcp.json schema (servers root, type: sse), chat.mcp.access (current key; chat.mcp.enabled only lives on via a settings-migration shim — checked in the installed bundle), workspace-trust requirement for Copilot AI features (--disable-workspace-trust only defers the dialog), MCP trust remembered per byte-identical config, code chat -m agent window targeting
  • Platform memory: org-scoped by design (MemoryStoreScope has no private scope) — hence room-keyed records + explicit "match the chat_id" retrieval instructions

Test plan

  • uv run pytest tests/framework_conformance/test_vscode_chat_toolkit.py tests/test_env_gates.py (19 tests)
  • pytest --collect-only tests/e2e/vscode → all skipped when gate unset
  • Full unit suite green (4007 passed)
  • Live run per runbook → 6/6 pass, scorecard + meta artifacts committed
  • Two review rounds applied (manual + /code-review): occupied-port rejection, no orphan on failed start, hung-CLI kill on submit timeout, CWD-independent scorecard filter, gate-off runs never shell out, README trust-flag correction

Linear: INT-1110

🤖 Generated with Claude Code

@linear-code

linear-code Bot commented Jul 23, 2026

Copy link
Copy Markdown

INT-1110

@AlexanderZ-Band
AlexanderZ-Band marked this pull request as ready for review July 23, 2026 09:52
@AlexanderZ-Band
AlexanderZ-Band requested a review from a team July 23, 2026 10:20
@AlexanderZ-Band
AlexanderZ-Band enabled auto-merge (squash) July 26, 2026 08:21
@AlexanderZ-Band
AlexanderZ-Band force-pushed the feat/intel-github-copilot-in-vs-code-mvp-integration-l0-INT-1110 branch from f059e12 to b1f3e8f Compare July 26, 2026 08:23
AlexanderZ-Band and others added 7 commits July 26, 2026 01:37
@AlexanderZ-Band @claude
feat(e2e): opt-in Copilot-in-VS-Code validation suite — toolkit + gate
11716d1 
Adds the harness plumbing for validating the GitHub Copilot in VS Code
surface against the L0-L4 bar: a code-chat CLI driver (PromptDriver seam
for a future test-electron backend), restartable band-mcp SSE server
holding the provisioned agent identity, workspace scaffolding
(.vscode/mcp.json + auto-approve settings), and a suite-local scorecard
collector reusing the baseline row schema. Gated behind
VSCODE_CHAT_TESTS_ENABLED (vscode_chat marker) since the surface needs a
GUI window and interactive Copilot sign-in — never runs in CI. Pure
parts are unit-tested in framework_conformance on every PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexander Zaikman <alexander.zaikman@band.ai>
@AlexanderZ-Band @claude
feat(e2e): Copilot-in-VS-Code live cells, suite conftest, and runbook
8bf45de 
Six live cells map the surface to the bar: L0 participation (echo
round-trip through band_send_message), L1 retained functions (native
workspace file + band_get_participants in one turn), L0 multi-participant
(peer-driven turn), L2 persistence (memory tools across a fresh chat
session), L2 isolation (per-room markers, leak asserted absent), L3
restart (band-mcp bridge restart between turns). L4 usage ships as the
scorecard's fixed N/A row — the surface exposes no per-turn usage signal.

Turns are driver-initiated with the room message relayed in the prompt:
the surface has no inbound channel and band-mcp exposes no message-read
tool. Replies are awaited over captured room messages rather than the
baseline delivery-status barrier, which only SDK runtimes ack.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexander Zaikman <alexander.zaikman@band.ai>
@AlexanderZ-Band @claude
feat(e2e): live-green VS Code Copilot suite — evidence, skill, unatte…
826e104 
…nded reruns

Live validation is green: 6/6 cells pass unattended (L0 participation +
multi-participant, L1 native+platform functions, L2 memory recall and
room isolation, L3 bridge restart), with the L4 usage N/A rationale on
the scorecard. Artifacts recorded (scorecard + environment sidecar:
macOS, VS Code 1.130.0, bundled Copilot Chat, band-mcp 1.3.2).

Hardening from the live runs:
- session-loop marker prepended (append=False) so pytest-asyncio's bare
  auto-mode marker cannot strand tests on a function loop away from the
  session-scoped WS/REST fixtures (subscribe timeouts, closed-loop
  teardowns)
- settings field names now equal their env vars (the pydantic-settings
  contract) — VSCODE_CHAT_SCORECARD_JSON/VSCODE_CHAT_TIMEOUT were
  silently unread
- stable workspace (~/band-e2e, visible path — hidden dirs trip
  Copilot's sensitive-file edit guard) + fixed band-mcp port keep
  mcp.json byte-identical, so folder trust, MCP trust, and per-tool
  approvals are one-time; reruns are fully unattended
- chat.autoReply instead of chat.tools.global.autoApprove — the latter
  is machine-wide YOLO mode and triggers a global consent escalation
- memory cells store room-keyed records, verify after store, and add a
  Band-side stored-memory barrier so a failure distinguishes the store
  path from the retrieval path from the prompt
- judge transcripts label peer message vs agent reply (a bare reply
  reads as unjudgeable echo)
- directory-scoped Claude skill (tests/e2e/vscode/.claude/skills) runs
  the whole flow: prereq checks, auto-install, dialog coaching
- README leads with the semi-manual nature of the suite

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexander Zaikman <alexander.zaikman@band.ai>
@AlexanderZ-Band @claude
refactor(e2e): intent-oriented VS Code suite — SurfaceRoom, vscode_wi…
23a50a6 
…ndow, gate table

Review pass over the suite for plumbing:

- SurfaceRoom (rooms.py) is the cells' one intent object: user_turn /
  peer_turn / remember hide the mention wiring, prompt relay, and
  captured-reply wait, so a cell reads as scenario prose. remember()
  encodes the store-seed shape learned from captured failures — fresh
  session (a continued chat sees an earlier store and acks without
  storing) plus a hard imperative (a soft 'store and confirm' gets
  satisficed into a bare confirmation) — with a Band-side stored-record
  barrier separating store failures from retrieval failures.
- vscode_window() async context manager owns the become-drivable flow
  (preflight -> open workspace -> startup grace); exit is a documented
  no-op — the window is human-owned.
- Root conftest gate branches collapse into one GATED_MARKERS table;
  env var and skip reason derive from the settings field name, and
  tests/test_env_gates.py guards the table against both the settings
  fields and pyproject's marker registration.
- Scorecard outcome mapping is a match/case over the report tuple;
  conformance tests assert typed rows and behavioral contracts (no
  exact-dict or join-format assertions).

Live: full suite 6/6 unattended after the refactor; scorecard artifact
refreshed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexander Zaikman <alexander.zaikman@band.ai>
@AlexanderZ-Band @claude
test: drop tautological guards from the VS Code toolkit tests
329b0ca 
Three tests restated definitions instead of protecting behavior — the
marker-path helper's own expression, a static preamble constant, and a
frozen row's fields echoed back. The N/A row's presence in the emitted
scorecard (the part that can actually break) stays covered by the
scorecard-rows test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexander Zaikman <alexander.zaikman@band.ai>
@AlexanderZ-Band @claude
fix(e2e): harden band-mcp startup and use current MCP access setting
fb26e75 
Review findings on the VS Code suite:

- workspace: write chat.mcp.access="all" (current key) instead of the
  deprecated boolean chat.mcp.enabled, which only works via VS Code's
  settings-migration shim.
- server: reject a port already held by another process before spawning —
  a stale listener would answer the bare-TCP readiness probe for a child
  that failed to bind, and the suite would run against the wrong Band
  identity.
- server: terminate the child when startup times out — the session
  fixture never reaches its yield on a start() failure, so its teardown
  would never stop() the orphan squatting the stable port.

Both server fixes carry regression tests that fail on the old code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AlexanderZ-Band @claude
fix(e2e): code-review hardening for the VS Code suite harness
410c320 
- driver: kill a hung `code chat` CLI on submit timeout — left alone it
  could still deliver its prompt minutes later, into a rerun's session.
- scorecard: filter reports by rootdir-relative nodeid prefix instead of
  resolving report file paths against the CWD, which silently dropped
  every row when pytest ran from outside the repo root.
- conftest: register the scorecard plugin (and its code/band-mcp version
  capture subprocesses) only when the suite gate is on — a lingering
  VSCODE_CHAT_SCORECARD_JSON must not stall an unrelated unit run on a
  cold uvx download mid-collection.
- README: workspace-trust bullet claimed --disable-workspace-trust is
  used; the launch deliberately omits it (Copilot requires a trusted
  workspace) — document the real one-time Trust Folder click.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AlexanderZ-Band
AlexanderZ-Band force-pushed the feat/intel-github-copilot-in-vs-code-mvp-integration-l0-INT-1110 branch from b1f3e8f to 410c320 Compare July 26, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AlexanderZ-Band