Skip to content

test: add MCP stdio E2E test suite for vibration server#293

Open
flamehaven01 wants to merge 2 commits into
IBM:mainfrom
flamehaven01:contrib/mcp-e2e-test-suite
Open

test: add MCP stdio E2E test suite for vibration server#293
flamehaven01 wants to merge 2 commits into
IBM:mainfrom
flamehaven01:contrib/mcp-e2e-test-suite

Conversation

@flamehaven01
Copy link
Copy Markdown

Summary

  • Adds test_mcp_e2e.py: a pytest-native end-to-end test suite for the MCP stdio transport layer of the vibration server
  • Covers 6 scenarios over real subprocess-based stdio transport (no CouchDB, no API keys required)
  • Fills a coverage gap: all existing tests in src/servers/vibration/tests/test_tools.py call call_tool(mcp, ...) in-process and never exercise the MCP wire protocol

Scenarios

ID Name What It Tests
SC-01 Tool Listing Server starts over stdio; all 8 expected tools exposed
SC-02 Static Tool Happy Path list_known_bearings returns bearing DB without CouchDB
SC-03 ISO Severity Classification assess_vibration_severity returns correct ISO 10816 zones (A and D)
SC-04 Pydantic Boundary diagnose_vibration with missing data_idField required, no crash
SC-05 Data Not Found compute_fft_spectrum with unknown data_id → structured error dict
SC-06 Teardown Regression Full lifecycle within anyio.fail_after(40s) — FIND-010 deadlock guard

Key Engineering Decisions

  • sys.executable instead of command="python": resolves to the Python running pytest regardless of venv/conda/pyenv; avoids ModuleNotFoundError in subprocess
  • anyio.fail_after instead of asyncio.wait_for: propagates cancellation through anyio task groups (how stdio_client manages subprocess streams internally); terminates server subprocess on deadlock rather than leaving a zombie process
  • _SENSITIVE_KEYS air-gap: strips LLM credentials (WATSONX_APIKEY, OPENAI_API_KEY, etc.) from subprocess environment before tests run; prevents accidental billable API calls if server logic changes
  • OTEL_SDK_DISABLED=true: disables OTEL in subprocess; prevents concurrent JSONL write collisions under pytest -n auto (xdist parallel runs)
  • Function-scoped vibration_session fixture: each test gets a fresh server subprocess, eliminating cross-test in-memory state leaks

Usage

# Drop at repo root alongside pyproject.toml, then:
pytest test_mcp_e2e.py -v          # all 6 scenarios
pytest test_mcp_e2e.py -v -k sc04  # single scenario
pytest -n auto test_mcp_e2e.py     # parallel (xdist-safe)

Verified On

Field Value
Commit a928284b06411ceb9e31663d936cf8b342b15ca2
OS Windows 11
Python 3.12.10
Result 6 passed in 18.54s

Related

  • Addresses FIND-010 (async teardown deadlock static risk) via SC-06 regression guard
  • Confirms FIND-002 (DSP tools functional) via SC-03 ISO zone classification
  • Confirms FIND-006 (Pydantic strict validation) via SC-04 boundary test

Six end-to-end scenarios over real MCP stdio transport.

Key design decisions:
- sys.executable: env-agnostic spawn (no PATH/venv dependency)
- anyio.fail_after: propagates cancellation through anyio task groups,
  preventing zombie server processes on deadlock (FIND-010 regression guard)
- _SENSITIVE_KEYS: strips LLM credentials from subprocess env before tests
- OTEL_SDK_DISABLED=true: prevents concurrent JSONL write collisions under
  pytest-xdist parallel runs
- vibration_session fixture: function-scoped for per-test process isolation

Verified on commit a928284, Python 3.12.10: 6 passed in 18.54s.

Signed-off-by: Flamehaven Initiative <flamehaven01@gmail.com>
@flamehaven01 flamehaven01 force-pushed the contrib/mcp-e2e-test-suite branch from 0e1c192 to 29cdda4 Compare May 11, 2026 05:07
Companion documentation for test_mcp_e2e.py explaining the five
design decisions (sys.executable, anyio.fail_after, _SENSITIVE_KEYS,
OTEL_SDK_DISABLED, vibration_session fixture) and extension patterns
for adding E2E coverage to other MCP servers.

Signed-off-by: Flamehaven Initiative <flamehaven01@gmail.com>
@DhavalRepo18
Copy link
Copy Markdown
Collaborator

Thanks for the thorough write-up and the clear separation from the in-process tests in src/servers/vibration/tests/test_tools.py — the wire-protocol coverage is a real gap to fill.

One location concern before merging: test_mcp_e2e.py is vibration-specific (it only imports servers.vibration.main) but sits at the repo root with a generic name. That breaks the project's src/servers/<name>/tests/ convention and implies broader coverage than the file actually has.

Could you move it to src/servers/vibration/tests/test_mcp_e2e.py alongside the existing test_tools.py? They're complementary — in-process unit tests vs. stdio wire-protocol E2E — and belong together. A couple of things to update along with the move:

  • The usage examples in docs/mcp-e2e-test-suite.md (pytest test_mcp_e2e.py) and the "Drop at repo root alongside pyproject.toml" line
  • Any repo-root path logic inside the file itself (the doc mentions "repo-root locator hardening")
  • Confirm python -m servers.vibration.main still resolves from the new location — test_tools.py should already have the pattern

Happy to take another look once it's moved.

@DhavalRepo18
Copy link
Copy Markdown
Collaborator

@flamehaven01 we provided feedback, appreciate your input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants