Skip to content

Fix test result outcome filtering#1182

Open
danmoseley wants to merge 2 commits intomicrosoft:mainfrom
danmoseley:azure-devops-mcp-test-results-filter
Open

Fix test result outcome filtering#1182
danmoseley wants to merge 2 commits intomicrosoft:mainfrom
danmoseley:azure-devops-mcp-test-results-filter

Conversation

@danmoseley
Copy link
Copy Markdown
Member

@danmoseley danmoseley commented Apr 25, 2026

Note

This PR description was drafted with GitHub Copilot assistance.

Fixes #1129

Avoid sending the invalid Azure DevOps result-details $filter expression for outcomes in testplan_show_test_results_from_build_id.

The current implementation builds an expression like:

Outcome eq 'Failed' or Outcome eq 'Aborted'

Azure DevOps rejects this for the result-details endpoint, this is what I got in Copilot:

Tool: ado-dnceng-testplan_show_test_results_from_build_id
Args: { "project": "internal", "buildid": 2952924, "outcomes": ["Failed"] }
Result: Error fetching test results: Argument filter with value Outcome eq 'Failed' is not correct.

Cause

This is because of a bug in test-plans.ts, where it assumed values should be quoted and joined with "or" when what actually is expected by the API is unquoted and joined with comma:
https://github.com/microsoft/azure-devops-mcp/blob/main/src/tools/test-plans.ts#L431-L432

Proof -- this example working URL filters for "failed or passed" and contains Outcome%20eq%20Failed%2CPassed ie unquoted comma separated.

GitHub issue number

#1129

Associated Risks

Relies on the filter syntax, which is not well (or at all?) documented, however the existing code already relies on this. Plus, this repo is owned by the Azure Devops team, who can presumably be confident of what is OK to rely on.

PR Checklist

  • I have read the contribution guidelines
  • I have read the code of conduct guidelines
  • Title of the pull request is clear and informative.
  • 👌 Code hygiene
  • 🔭 Telemetry added, updated, or N/A
  • 📄 Documentation added, updated, or N/A
  • 🛡️ Automated tests added, or N/A

🧪 How did you test it?

  • npm test -- --runTestsByPath test/src/tools/test-plan.test.ts --runInBand
  • npm run validate-tools

Avoid sending the invalid Azure DevOps result-details filter expression and filter requested outcomes client-side after fetching result details.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danmoseley danmoseley requested a review from a team as a code owner April 25, 2026 08:34
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 25, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@546e37d). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1182   +/-   ##
=======================================
  Coverage        ?   90.25%           
=======================================
  Files           ?       19           
  Lines           ?     2124           
  Branches        ?      499           
=======================================
  Hits            ?     1917           
  Misses          ?      111           
  Partials        ?       96           
Flag Coverage Δ
unittests 90.25% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@danmoseley
Copy link
Copy Markdown
Member Author

Updated this PR to restore server-side filtering with the correct filter format, discovered by inspecting ADO web client network traffic.

The API accepts unquoted values, comma-separated for multiple outcomes:

Outcome eq Failed,Passed

The original code's only mistake was quoting the values (Outcome eq 'Failed'). The client-side filtering approach in the previous commit works but pulls all results before filtering — wasteful for large builds.

You can verify the filter format works directly (no auth needed, public project):

The unit tests remain mocked so can't prove the filter string is accepted server-side, but the above URLs demonstrate it.

@danmoseley danmoseley force-pushed the azure-devops-mcp-test-results-filter branch 2 times, most recently from 32a281d to 4dd2308 Compare April 26, 2026 07:15
The ADO API accepts: Outcome eq Failed,Passed (unquoted, comma-separated).
The original code used quoted strings which the API rejects. Restore
server-side filtering with the correct syntax instead of filtering
client-side after fetching all results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danmoseley danmoseley force-pushed the azure-devops-mcp-test-results-filter branch from 4dd2308 to 01691e2 Compare April 26, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: show_test_results_from_build_id fails with any outcomes filter value

2 participants