Skip to content

ci(conformance): integrate MCP client conformance test pipeline#637

Open
anubhav756 wants to merge 4 commits intomainfrom
anubhav-conformance
Open

ci(conformance): integrate MCP client conformance test pipeline#637
anubhav756 wants to merge 4 commits intomainfrom
anubhav-conformance

Conversation

@anubhav756
Copy link
Copy Markdown
Contributor

@anubhav756 anubhav756 commented Apr 20, 2026

Description

This PR integrates the official MCP conformance testing framework into the toolbox-core package to evaluate our SDK Tier status.

Changes

  • Adds client.py to handle initialize and other MCP client scenarios via the ToolboxClient.
  • Adds a GitHub Action to run the conformance suite against PRs. It utilizes an expected failures baseline (conformance-baseline.yml) to keep CI green while preventing new regressions.
  • Adds a scheduled workflow that runs the tier-check audit.
    • It automatically opens (and updates) a single GitHub issue labeled periodic-failure to track our operational gaps preventing Tier 1 status.

Note

We have opened an upstream PR to fix the Repository Health table, in the generated GitHub issue, in the official conformance tool: modelcontextprotocol/conformance#239
image

@anubhav756 anubhav756 self-assigned this Apr 20, 2026
@anubhav756 anubhav756 changed the title ci (core): Adds GitHub Action workflow, test harness, and baseline config ci(core): Adds GitHub Action workflow, test harness, and baseline config Apr 20, 2026
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 2 times, most recently from 5e401fb to 861d9ff Compare April 20, 2026 15:51
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 2 times, most recently from 8e57c3e to 58013b5 Compare April 20, 2026 16:30
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from 1002613 to f5783c6 Compare April 20, 2026 22:02
@anubhav756 anubhav756 changed the title ci(core): Adds GitHub Action workflow, test harness, and baseline config ci(conformance): integrate MCP client conformance test pipeline Apr 20, 2026
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from f5783c6 to 10e19c9 Compare April 20, 2026 22:14
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 2 times, most recently from 00b274f to 6641a24 Compare April 20, 2026 23:00
@anubhav756 anubhav756 marked this pull request as ready for review April 20, 2026 23:14
@anubhav756 anubhav756 requested a review from a team as a code owner April 20, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant