Skip to content

FE-1093: Harden orchestrator agent harness testing diagnostics#269

Open
kostandinang wants to merge 3 commits into
mainfrom
ka/fe-1093-harness-diagnostics
Open

FE-1093: Harden orchestrator agent harness testing diagnostics#269
kostandinang wants to merge 3 commits into
mainfrom
ka/fe-1093-harness-diagnostics

Conversation

@kostandinang

@kostandinang kostandinang commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What

Adds profile-owned runner diagnostics for cook test execution.

Why

Runner/toolchain load failures, like Vitest being denied under confinement, should route as infra instead of product test failures.

Verification

  • npx vitest run src/orchestrator/src/project-profile.test.ts src/orchestrator/src/test-runner.test.ts
  • npm run verify reached the full test suite; unrelated graph type-contract tests timed out.

@kostandinang kostandinang changed the title FE-1093: add profile-owned runner diagnostics FE-1093: Harden orchestrator agent harness testing diagnostics Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@kostandinang kostandinang marked this pull request as ready for review June 26, 2026 21:51
@cursor

cursor Bot commented Jun 26, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes how cook routes failed verify runs (infra vs test vs absent); misclassification could skip real reds or send infra to the wrong agent, though new tests target the tricky EPERM/assertion cases.

Overview
Each toolchain profile now carries RunnerDiagnostics (runnerName, runnerPackages, noTestsPatterns) so the cook harness can classify failures using stack-specific facts instead of hardcoded Vitest-only patterns.

classifyTestFailure takes those diagnostics and treats permission/load denials under profile-listed node_modules runner packages as infra, while keeping real assertion reds and unrelated EPERM (e.g. Playwright) as test. “No tests matched” wording is likewise profile-owned for the absent bucket. Spawn errors now also treat EACCES / EPERM as infra via isInfraSpawnError, with boundary checks so embedded error codes and assertion text are not misclassified.

ToolchainTestRunner passes this.toolchain.diagnostics into classification; tests cover runner-package EPERM, custom absent patterns, and registry expectations for package-backed vs built-in runners.

Reviewed by Cursor Bugbot for commit 7511c72. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread src/orchestrator/src/test-runner.ts Outdated
Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4145572. Configure here.

Comment thread src/orchestrator/src/test-runner.ts Outdated
Co-authored-by: Cursor <cursoragent@cursor.com>
@kostandinang kostandinang self-assigned this Jun 29, 2026
@kostandinang kostandinang requested a review from lunelson June 30, 2026 07:51
@kostandinang kostandinang enabled auto-merge June 30, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant