feat: Copilot-in-VS-Code opt-in E2E validation suite

pyproject.toml

@@ -276,6 +276,7 @@ markers = [
     "flaky_reason: baseline flakiness taxonomy stamp (kind + reason); see tests/e2e/baseline/flaky.py",
     "docker_build: builds/runs a real Docker image via subprocess (requires a local Docker daemon; skipped unless DOCKER_TESTS_ENABLED=true, including in CI, which has Docker but shouldn't pay this cost by default)",
     "sandbox: drives a Docker Sandbox via the sbx CLI (needs sbx + a nested-virtualization-capable host + live Band; skipped unless SANDBOX_TESTS_ENABLED=true — CI has neither the CLI nor nested virt)",
+    "vscode_chat: drives a real signed-in VS Code window via the code chat CLI (GUI + interactive Copilot sign-in; skipped unless VSCODE_CHAT_TESTS_ENABLED=true — can never run in CI)",
 ]
 
 [tool.uv]

tests/conftest.py

@@ -73,6 +73,7 @@ class CollectionGateSettings(BaseSettings):
     e2e_tests_enabled: bool = False  # E2E_TESTS_ENABLED
     docker_tests_enabled: bool = False  # DOCKER_TESTS_ENABLED
     sandbox_tests_enabled: bool = False  # SANDBOX_TESTS_ENABLED
+    vscode_chat_tests_enabled: bool = False  # VSCODE_CHAT_TESTS_ENABLED
 
 
 def pytest_ignore_collect(collection_path: Path) -> bool | None:
@@ -89,35 +90,39 @@ def pytest_ignore_collect(collection_path: Path) -> bool | None:
     return None
 
 
+# Opt-in suite gates: marker -> the CollectionGateSettings field that opens it.
+# The env var IS the field name uppercased (the pydantic-settings contract), so
+# the skip reason is derived — one row here is all a new gated suite needs.
+GATED_MARKERS: dict[str, str] = {
+    "e2e": "e2e_tests_enabled",
+    "docker_build": "docker_tests_enabled",
+    "sandbox": "sandbox_tests_enabled",
+    "vscode_chat": "vscode_chat_tests_enabled",
+}
+
+
 def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
-    """Skip e2e- and docker_build-marked tests unless explicitly enabled.
-
-    tests/e2e/ gates itself through its own conftest; this covers
-    e2e-marked tests living elsewhere (e.g. the codex ACP protocol
-    tests), which spawn real backends and must never ride a normal
-    unit run.
-
-    docker_build-marked tests shell out to a real `docker build`/`docker
-    run` — CI runners do have a Docker daemon (unlike the nested
-    virtualization sbx tests need), so a plain Docker-availability check
-    isn't enough to keep them off CI; they need the same explicit opt-in
-    as e2e tests.
+    """Skip gate-marked suites unless their env gate is explicitly enabled.
+
+    tests/e2e/ gates itself through its own conftest; this covers marked
+    tests living elsewhere (e.g. the codex ACP protocol tests). These
+    suites spawn real backends, real `docker build`s, sbx microVMs, or a
+    live VS Code window — none may ride a normal unit run, and none can
+    rely on mere tool availability (CI runners do have Docker), so each
+    needs its explicit opt-in.
     """
     gates = CollectionGateSettings()
-    skip_e2e = pytest.mark.skip(reason="set E2E_TESTS_ENABLED=true to run e2e tests")
-    skip_docker = pytest.mark.skip(
-        reason="set DOCKER_TESTS_ENABLED=true to run docker_build tests"
-    )
-    skip_sandbox = pytest.mark.skip(
-        reason="set SANDBOX_TESTS_ENABLED=true to run sbx sandbox tests"
-    )
+    closed = {
+        marker: pytest.mark.skip(
+            reason=f"set {field.upper()}=true to run {marker}-marked tests"
+        )
+        for marker, field in GATED_MARKERS.items()
+        if not getattr(gates, field)
+    }
     for item in items:
-        if not gates.e2e_tests_enabled and item.get_closest_marker("e2e"):
-            item.add_marker(skip_e2e)
-        if not gates.docker_tests_enabled and item.get_closest_marker("docker_build"):
-            item.add_marker(skip_docker)
-        if not gates.sandbox_tests_enabled and item.get_closest_marker("sandbox"):
-            item.add_marker(skip_sandbox)
+        for marker, skip in closed.items():
+            if item.get_closest_marker(marker):
+                item.add_marker(skip)
 
 
 @pytest.fixture(autouse=True)

tests/e2e/vscode/.claude/skills/vscode-chat-e2e/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: vscode-chat-e2e
+description: Run the semi-manual Copilot-in-VS-Code E2E validation suite (tests/e2e/vscode) end to end — verify prerequisites, auto-install what can be installed (VS Code, band-mcp), start the live run, and coach the human through the few one-time dialogs only they can click (Copilot sign-in, folder trust, MCP/tool approvals). Use when asked to run, rerun, or validate the VS Code Copilot surface, or to produce its scorecard evidence.
+---
+
+# Copilot-in-VS-Code E2E run
+
+This suite is **inherently semi-manual** (see `tests/e2e/vscode/README.md`): it
+drives a real, visible VS Code window with a human-signed-in Copilot. Your job
+is to make the human's part as small as possible: check every prerequisite,
+install whatever can be installed non-interactively, run the suite, and tell
+the user exactly what to click **only** when there is no other way.
+
+## Phase 1 — prerequisites (install, don't ask, where possible)
+
+Check in this order; fix silently where a non-interactive fix exists.
+
+1. **VS Code + `code` CLI** — `code --version` and `code chat --help`.
+   - Missing entirely: `brew install --cask visual-studio-code` (macOS; the
+     cask links `code` onto PATH itself).
+   - App present but no CLI: link it —
+     `ln -sf "/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code" /opt/homebrew/bin/code`.
+   - `chat` subcommand missing: VS Code too old (< 1.101) —
+     `brew upgrade --cask visual-studio-code`.
+2. **Copilot Chat extension** — ships built into current VS Code; verify with
+   `code --install-extension GitHub.copilot-chat` (a "already installed"
+   response is success).
+3. **band-mcp** — `uvx band-mcp --version` (uvx fetches it on first use; no
+   install step). Pass `BAND_MCP_COMMAND="uvx band-mcp"` to the run.
+4. **Credentials** — `.env.test` must define `BAND_API_KEY_USER` (provisioning
+   + observer) and `ANTHROPIC_API_KEY` (judge). Missing keys cannot be
+   installed: stop and ask the user for them.
+5. **Copilot sign-in** — cannot be automated (interactive GitHub OAuth, no
+   headless path). If unsure whether a login exists, just start Phase 2: a
+   signed-out Copilot shows "Sign in to use Copilot" in the chat panel — then
+   instruct: *open Chat (⌃⌘I), click "Sign in to use Copilot", finish the
+   browser OAuth, then say "signed in"* and rerun.
+
+## Phase 2 — the run
+
+```bash
+VSCODE_CHAT_TESTS_ENABLED=true E2E_TESTS_ENABLED=true \
+BAND_MCP_COMMAND="uvx band-mcp" \
+VSCODE_CHAT_SCORECARD_JSON=artifacts/<issue-or-run-id>-copilot-vscode-scorecard.json \
+uv run pytest tests/e2e/vscode -v -s --no-cov
+```
+
+Run it in the background and monitor per-cell PASSED/FAILED lines. Kill any
+leftover `band-mcp --transport sse` process before starting. Expect ~4 minutes
+for 6 cells.
+
+## Phase 3 — coach the one-time dialogs (first run on a machine only)
+
+The workspace (`~/band-e2e/vscode-chat-workspace`) and band-mcp port (8631)
+are stable, so every choice below is remembered — reruns are unattended.
+When the VS Code window opens, tell the user to click, as dialogs appear:
+
+1. "Do you trust the authors…" → **Trust Folder & Continue**
+2. MCP server `band` start/trust → **Allow/Trust**
+3. Tool confirmations → dropdown next to Allow → **"Always allow…" (workspace)**
+   — once per tool (~4 band tools), never "this session"
+4. "Allow edits to sensitive files" (shouldn't appear on the visible-path
+   workspace; if it does) → dropdown → **Always allow**
+
+Never tell the user to enable `chat.tools.global.autoApprove` ("YOLO mode") —
+machine-wide security downgrade; the harness deliberately avoids it.
+
+## Phase 4 — verdict + evidence
+
+- Green: report `N passed`, point at the scorecard JSON + `.meta.json`
+  sidecar (versions evidence). Commit artifacts only when asked.
+- A cell failing on reply content is usually live-model flakiness — rerun
+  once before investigating; two identical failures = real, read the
+  traceback (full failure section prints at session end).
+- "Quota reached" badge in the window = Copilot premium requests exhausted;
+  turns will stall or degrade. Surface it to the user: paid seat, wait for
+  reset, or switch the chat model picker to an included model.
+- Reap check: provisioned agents/rooms are auto-reaped (`BAND_E2E_AUTOCLEAN`);
+  leftover `band-mcp` processes should be killed if the run was interrupted.

tests/e2e/vscode/README.md

@@ -0,0 +1,163 @@
+# Copilot-in-VS-Code validation suite (opt-in, dev-machine only)
+
+> ## ⚠️ This is inherently a SEMI-MANUAL test suite
+>
+> It drives a **real, visible VS Code window** with a **human-signed-in
+> Copilot**. A person must be at the machine: sign in to Copilot once, and on
+> the **first run** click through a handful of one-time dialogs (folder trust,
+> MCP server allow, per-tool "Always allow"). Those choices are remembered, so
+> **reruns are unattended** — but the GUI window, the interactive GitHub OAuth,
+> and the possibility of a new confirmation dialog never go away. It can never
+> run in CI and is not a hands-off matrix suite; treat every green run as
+> "passed on a supervised dev machine", and record it via the scorecard
+> artifact.
+
+Drives **GitHub Copilot Chat inside a real VS Code window** (agent mode) against
+the live Band platform and validates the L0–L4 common bar. The harness:
+
+1. provisions a fresh Band agent identity,
+2. runs **band-mcp** (SSE, loopback) holding that identity's key,
+3. scaffolds a throwaway workspace whose `.vscode/mcp.json` points Copilot at it,
+4. opens the workspace window once, then submits each turn via `code chat -m agent`,
+5. asserts **Band-side only** (reply capture + LLM judge) — the VS Code UI is
+   never scraped.
+
+**This suite can never run in CI**: the Copilot extension authenticates through
+interactive GitHub OAuth in a browser (no headless/service-account path), and a
+GUI window must stay open. It is gated behind `VSCODE_CHAT_TESTS_ENABLED`
+(marker `vscode_chat`), like the `DOCKER_TESTS_ENABLED` pattern.
+
+## Surface semantics (why the tests look like this)
+
+- **No inbound channel.** Copilot in VS Code cannot be pushed a Band message,
+  and band-mcp exposes **no message-read/history tool** (verified against
+  band-mcp 1.3.2). Every turn is driver-initiated: the prompt relays the room
+  message and pins the room `chat_id`. This mirrors real usage, where the
+  developer relays context into chat and Copilot posts via `band_send_message`.
+- **Recall = platform memory tools.** Cross-session/cross-restart recall flows
+  through `band_store_memory` / `band_list_memories` (the server runs with
+  `--tools memory`), which are platform-persisted and session-independent.
+- **No delivery acks.** band-mcp is REST-only, so the baseline
+  `wait_for_reply` barrier (delivery-status PROCESSED) never fires; the cells
+  wait on captured room messages instead.
+- **L4 usage is N/A**: the surface exposes no per-turn usage/billing signal.
+  The scorecard carries the rationale as a fixed `na` row.
+
+## Prerequisites
+
+- **VS Code ≥ 1.102** with the `code` CLI on PATH (macOS: run
+  "Shell Command: Install 'code' command in PATH" from the Command Palette),
+  or set `CODE_COMMAND`.
+- **GitHub Copilot Chat extension installed and signed in** (interactive; do
+  this once in the normal VS Code UI before running).
+- **band-mcp ≥ 1.3.2** installed (`uv tool install band-mcp`), or set
+  `BAND_MCP_COMMAND` (e.g. `"uvx band-mcp"`).
+- `.env.test` with `BAND_API_KEY_USER` (provisioning/observer) and
+  `ANTHROPIC_API_KEY` (the judge).
+
+## Running
+
+**Easiest**: use the directory-scoped Claude Code skill — `/vscode-chat-e2e`
+(defined in `.claude/skills/vscode-chat-e2e/SKILL.md` next to this suite). It
+checks prerequisites, installs what it can, runs the suite, and tells you
+exactly what to click on a first run.
+
+Manual equivalent:
+
+```bash
+VSCODE_CHAT_TESTS_ENABLED=true E2E_TESTS_ENABLED=true \
+VSCODE_CHAT_SCORECARD_JSON=artifacts/int-1110-copilot-vscode-scorecard.json \
+uv run pytest tests/e2e/vscode -v -s --no-cov
+```
+
+During the run a VS Code window opens on the scaffolded workspace. The harness
+minimizes prompts, and after the first run reruns are unattended:
+
+- **Workspace trust**: VS Code asks once per folder ("Do you trust the
+  authors…") — click **Trust Folder & Continue**. The workspace path is
+  stable by default, so the choice is remembered for every rerun. (The
+  launch deliberately does *not* use `--disable-workspace-trust`: Copilot's
+  AI features require a trusted workspace, so that flag would only defer
+  the same dialog to chat time.)
+- **MCP server trust**: VS Code asks once per server *configuration*. The
+  workspace path and the band-mcp port are stable by default, so `mcp.json`
+  is byte-identical across runs — allow the `band` server on the first run
+  and the remembered trust holds for every rerun. (Changing
+  `VSCODE_CHAT_WORKSPACE` or `BAND_MCP_PORT` re-triggers the prompt.)
+- **Tool approvals**: on each tool's first-ever call, open the dropdown next
+  to Allow and pick **"Always allow…" (workspace)** — one click per tool
+  (~4 band tools), remembered afterwards. The workspace also sets
+  `chat.autoReply` so agent-side questions never stall a turn.
+- Leave the window open and unfocused-but-alive until the run finishes.
+
+> **Security note:** the harness deliberately does **not** enable
+> `chat.tools.global.autoApprove` ("YOLO mode") — that disables tool approval
+> for every workspace on the machine and VS Code escalates it to a global
+> consent dialog. Approvals stay per-tool and workspace-scoped. If you enable
+> YOLO in your own user settings, that is a machine-wide security trade-off
+> you own.
+
+Artifacts (when `VSCODE_CHAT_SCORECARD_JSON` is set):
+
+- `artifacts/int-1110-copilot-vscode-scorecard.json` — pass/fail/skip rows plus
+  the fixed L4 `na` row (baseline scorecard row schema, merge-compatible).
+- `…-scorecard.json.meta.json` — environment evidence: OS, `code --version`,
+  Copilot extension versions, band-mcp version.
+
+## Environment knobs
+
+| Env var | Default | Purpose |
+|---|---|---|
+| `VSCODE_CHAT_TESTS_ENABLED` | `false` | The collection gate |
+| `CODE_COMMAND` | `code` | VS Code CLI binary (may be a full path) |
+| `BAND_MCP_COMMAND` | `band-mcp` | band-mcp launcher (e.g. `uvx band-mcp`) |
+| `BAND_MCP_PORT` | `8631` | band-mcp SSE port (`0` = ephemeral; stable keeps MCP trust remembered) |
+| `VSCODE_CHAT_WORKSPACE` | `~/band-e2e/vscode-chat-workspace` | Workspace dir VS Code opens (stable keeps MCP trust remembered) |
+| `VSCODE_CHAT_SCORECARD_JSON` | empty | Scorecard output path (empty = don't emit) |
+| `VSCODE_CHAT_TIMEOUT` | `300` | Seconds allowed per live turn |
+
+Band endpoints, credentials, autoclean/orphan-sweep policy, and the judge model
+come from the baseline settings (`tests/e2e/baseline/settings.py`).
+
+## Cells
+
+| Test | Level | Proves |
+|---|---|---|
+| `test_participation_reply_round_trip` | L0 | Room message → Copilot → `band_send_message` reply with the echo token |
+| `test_original_functions_retained` | L1 | Native function (workspace file) + platform tool (`band_get_participants`) in one turn |
+| `test_multi_participant_echo_peer` | L0 | A peer agent's message drives a turn; the reply engages the peer |
+| `test_recall_across_chat_sessions` | L2 | Fact stored via memory tools survives to a fresh chat session |
+| `test_no_leak_between_rooms` | L2 | Room B's answer names room B's marker, never room A's |
+| `test_restart_recall_and_function` | L3 | band-mcp restart between turns; recall + native function still work |
+| *(no test)* `usage_accounting` | L4 | `na` scorecard row — no per-turn usage signal exposed |
+
+## Manual variants / known weak spots
+
+- **Fresh chat session** (`new_session=True`) is expressed as a prompt preamble —
+  the `code chat` CLI has no verified per-invocation new-session switch. For a
+  stronger variant, click **New Chat** in the window between the two turns of
+  `test_recall_across_chat_sessions` while it waits.
+- **Full window restart** (quit VS Code between the turns of
+  `test_restart_recall_and_function`, reopen with `code <workspace-dir>`) is a
+  manual variant; the automated cell restarts the platform bridge (band-mcp)
+  instead, which is deterministic.
+
+## Troubleshooting
+
+- **Prompt lands in the wrong window**: `code chat` targets the window whose
+  workspace matches the CWD; keep only the harness's workspace window open, or
+  close other VS Code windows.
+- **421 responses from band-mcp**: the server sets
+  `ALLOWED_HOSTS='["localhost:*","127.0.0.1:*"]'` itself; if you changed the
+  host/port wiring, keep the allowlist in sync.
+- **Turn times out with no reply**: check the window — a pending trust/approval
+  dialog blocks the turn. Approve it; the run continues on the next cell.
+- **`PreflightError`**: the `code` CLI is missing or predates the `chat`
+  subcommand — upgrade VS Code / fix `CODE_COMMAND`.
+
+## Future work
+
+`driver.PromptDriver` is a protocol: a `@vscode/test-electron` backend (pinned
+VS Code download, persistent signed-in profile, programmatic
+`workbench.action.chat.open`) can replace `CodeChatDriver` without touching the
+cells.