Execute direct MCP steps in test pipeline

[Ker102]

Summary

• prepare test pipeline plan steps into a single ordered execution queue
• generate Python only for execute_code steps while preserving direct MCP tool calls
• add a regression test that prevents the dev route from silently skipping direct-tool plans

Validation

• npx tsx scripts/test/test-test-pipeline-direct-step-execution.ts
• npx tsx scripts/test/test-direct-tool-planning-prompt.ts
• npx tsx scripts/test/test-workflow-direct-tool-recommendations.ts
• npx tsc --noEmit --incremental false
• npm run lint

Summary by CodeRabbit

• Bug Fixes

  • Improved pipeline step execution handling for code generation and MCP actions, with enhanced execution tracking and timing metrics.

• Tests

  • Added validation tests for pipeline step execution implementation.

[coderabbitai]

Warning

Rate limit exceeded

@Ker102 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 20 minutes and 26 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c58cac2e-8176-4992-b0f5-878120f4003a

📥 Commits

Reviewing files that changed from the base of the PR and between e5783ea and 737c9f1.

📒 Files selected for processing (2)

• app/api/test-pipeline/route.ts
• scripts/test/test-test-pipeline-direct-step-execution.ts

📝 Walkthrough

Walkthrough

This PR refactors the test-pipeline route to unify execution of all MCP steps (both code generation and non-code actions) through a single PreparedPipelineStep data structure. Step 3 now prepares all plan steps into an ordered list, generating code as needed; Step 4 executes them sequentially with differentiated monitoring. A validation test confirms the implementation structure.

Changes

Unified pipeline step execution

Layer / File(s)	Summary
Data contract and step preparation app/api/test-pipeline/route.ts	PreparedPipelineStep type standardizes the shape of executable steps (step number, action, params, description). Step 3 refactors to iterate all plan steps, reusing inline code when provided or calling generateCode with per-step timing, pushing execute_code steps with code params and other steps with planner params into a unified preparedSteps list.
Step execution and skip logic app/api/test-pipeline/route.ts	Step 4 iterates over preparedSteps in order, executing each step's MCP action with a single client, emitting monitor payloads that differ for execute_code (with codeChars) vs other actions, recording per-step timing and results. Adds a skip condition when preparedSteps is empty.
Implementation validation test scripts/test/test-test-pipeline-direct-step-execution.ts	Test script validates that the route contains PreparedPipelineStep, preparedSteps, plan step iteration, execute_code action handling, and generateCode calls, while asserting the absence of prior skip/codegen-iteration patterns.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

backend, scripts

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provided is a summary format with key changes and validation commands, but does not follow the template structure with required sections like 'Type of Change', 'Related Issues', 'Motivation and Context', and 'Testing' checkboxes.	Restructure the description to include all template sections: Description, Type of Change (with checkbox), Related Issues, Motivation and Context, How Has This Been Tested (with checkboxes), and Checklist items.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main change: executing direct MCP steps in the test pipeline, which aligns with the core refactoring of the route to handle both execute_code and direct MCP actions.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/test-pipeline-direct-tools

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/api/test-pipeline/route.ts`:
- Around line 295-299: preparedSteps currently uses model-produced step.action
directly; enforce a fail-closed allowlist by validating step.action against the
MCP_TOOLS set before adding it to preparedSteps (in the block that constructs
the step object where planStepNumber, step.action, step.parameters, and
step.rationale/expected_outcome are used). If step.action is not present in
MCP_TOOLS, replace the action with a safe no-op identifier (e.g., "noop" or
"disallowed_action") and annotate params or description with a warning (or
alternatively throw/reject depending on desired behavior) so no unallowlisted
tool will be invoked at runtime; apply the same validation where the other
prepared step construction occurs to cover both occurrences.
- Around line 351-365: The logging call can throw when JSON.stringify encounters
circular refs or returns undefined, causing a successful step to be treated as
an error; fix by precomputing a safe, non-throwing preview string before calling
agentMonitor.log: create a short helper or inline safe serialization for
stepResult.output (e.g., try to JSON.stringify and slice, but catch exceptions
and fallback to a string like "[unserializable]" or String(value), and treat
undefined as "undefined"), then pass that safe preview to agentMonitor.log
(references: agentMonitor.log, stepResult, executable.planStepNumber,
executionResults.push) so the stringify cannot escape into the step try/catch
and falsely mark the step as failed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 35ebb7d6-aec0-461c-8be9-b09344a07508

📥 Commits

Reviewing files that changed from the base of the PR and between 18469a0 and e5783ea.

📒 Files selected for processing (2)

• app/api/test-pipeline/route.ts
• scripts/test/test-test-pipeline-direct-step-execution.ts