test(uipath-mcp-servers): fix e2e-allkinds grading flake + create-task save-location flake

[cosmyo]

Problem

The skill-mcp-servers-e2e-uipath-allkinds task scored 0.933 / FAILURE on an otherwise-successful run. All 6 tools landed on the live tenant and the Jira curated activity was authored correctly — but one criterion failed:

command_executed / tool_name: Read / integration-service/(resources|reference-resolution).md → 0/1

The agent did load integration-service/resources.md (for the Jira cascade) — via cat in a Bash command rather than the Read tool:

# Read the resources.md reference for cascade / Parent-Field-Driven Custom Fields
cat .../uipath-platform/references/integration-service/resources.md

The criterion graded the tool mechanism (which tool opened the file), not whether the reference was consulted. Both tool_name: Read criteria in this task had this flaw.

Root cause (confirmed in coder_eval source)

coder_eval/criteria/command_executed.py:

if criterion.tool_name is not None and cmd.tool_name != criterion.tool_name:
    continue                                   # tool filter only applies when set
...
if cmd.tool_name == "Bash" and cmd.parameters.get("command"):
    cmd_text = cmd.parameters["command"]       # Bash → the command string
else:
    cmd_text = json.dumps(cmd.parameters)      # Read → {"file_path": "..."}

So omitting tool_name matches the pattern against every tool — the Read tool's file_path and a Bash cat/grep/head command string. The filename can stay in the check; it just shouldn't be pinned to the Read tool.

Fix (criteria 13 → 15)

• Keep the reference-load process signals, drop tool_name: Read — now they match whether the agent used Read or cat/grep/head. Verified against the real transcript: is-activity-workflow.md matched via the Read tool, integration-service/resources.md matched via the Bash cat.
• Add two live-tenant OUTCOME checks the 6-tool count can't give:
  • Gmail get-labels authored for the right connector/operation (Metadata.connector.key == uipath-google-gmail, object method == GET)
  • Jira curated tool resolved its -f cascade — InputSchema.properties has fields.summary and fields.description

Both jq programs verified green (exit 0) against the captured live mcp-tools list output from the failing run.

Result: the check is now both robust (no dependency on which tool opens a file) and more accurate (grades actual tool correctness, not just that a file was opened).

Validation

• YAML parses (15 criteria)
• Both file-load patterns match the real transcript (Read + cat); both outcome jq programs return true
• lint-task: OK; CLI-verb check: 0 High/Medium (all Info)
• No skill change — the skill already steered the agent to the reference correctly

Follow-up (not in this PR)

The tool_name: Read reference-load pattern is used in ~10 other task files (uipath-review, uipath-api-workflow) and carries the same latent flake. Candidate for a repo-wide sweep to drop the tool_name pin on file-load criteria.

🤖 Generated with Claude Code

---

Commit 2 — create-task artifact save location (outlook/gmail/slack/jira)

skill-mcp-servers-outlook-create scored 0.000 (0/4) with "File 'metadata.json' does not exist" — yet the agent built the Outlook tool correctly (right connector/resource, baked UTC timezone, dry-run OK) and produced every artifact. It just wrote them to a self-created output/ subdirectory (mkdir -p .../output) while the criteria check root paths.

Root cause: the prompt said Save: metadata.json, ... with no location, so the agent invented output/. Prompt-ambiguity/test issue, not a skill issue.

All four create smoke tasks shared the ambiguity (none forbade a subdir; slack's "to cwd" wouldn't have helped — output/ was under cwd). Fixed all four prompts to pin the save location to the working-directory root and explicitly forbid a subdirectory. Prompts only — no skill or success_criteria changes.

[github-actions]

Claude finished @cosmyo's task in 2m 8s —— View job

---

PR Review: test(uipath-mcp-servers): grade e2e-allkinds is-activity by outcome, not Read-tool

• Read review criteria and project conventions
• Read full diff
• Read full changed file for context
• Check PR conversation for existing feedback
• Verify YAML structure and success criteria
• Post review findings

---

Summary

Replaces two command_executed / tool_name: Read criteria in the e2e-uipath-allkinds task with run_command checks that verify actual tool state on the live tenant. This fixes a flake where the agent opened references via cat/grep instead of the Read tool, causing a false failure on an otherwise-correct run.

Change-by-Change Review

1. tests/tasks/uipath-mcp-servers/e2e-uipath-allkinds/task.yaml

Severity: OK — well-executed change.

What changed:

• Two command_executed criteria (lines 99–112 old) that checked for tool_name: Read with command_pattern matching is-activity-workflow.md and integration-service/(resources|reference-resolution).md are replaced by two run_command criteria that verify behavioral outcomes on the live tenant.
• The Gmail check (line 105–111) verifies the tool has Metadata.connector.key == "uipath-google-gmail" and object.method == "GET".
• The Jira check (line 119–125) verifies the tool's InputSchema.properties has fields.summary and fields.description (proving the -f cascade resolved).
• Both use jq -e (exit 1 on false/null) with expected_exit_code: 0, timeout: 60, weight 1.5 — consistent with the existing run_command pattern at line 149.
• Criterion count stays at 13; weights unchanged at 1.5 each.
• Comment block updated to explain the grading philosophy.

Verification:

• YAML is valid (confirmed via grep-based count: 13 - type: entries).
• No tool_name: Read criteria remain (only in explanatory comments at lines 103, 116).
• The run_command structure matches the sibling pattern in e2e-platform-server/task.yaml (same fields: command, timeout, expected_exit_code).
• jq programs look correct: fromjson on the string-typed Metadata/InputSchema fields, then field access and boolean assertions via -e.
• The $(jq -r .slug report.json) subshell is consistent with the existing check at line 151.

One minor note: The comments at lines 99–104 and 113–118 are longer than strictly needed (6 lines each explaining the old proxies). The repo's token-optimization rules say to strip words that don't carry information — the "replaces a former…" sentences are changelog context that belongs in the PR description (which already has it) rather than the YAML. This is cosmetic; not flagging as an issue since comments inside test YAMLs don't burn agent tokens at runtime.

What's Missing

Nothing identified.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed
E2E Tests	OK	Criteria correctly grade behavior over mechanism
Skill Body	N/A	No skill files changed
References & Assets	N/A	No references changed
Repo Hygiene	OK	Scoped change, no secrets, no unrelated modifications

Issues for Manual Review

• The jq programs were verified against captured output from the failing run (per the PR description). Worth confirming on a fresh nightly run that the Metadata and InputSchema JSON shapes haven't changed since that capture.

Conclusion

Clean change that aligns the test with the repo's "Grade Behavior, Not Self-Reports" principle. The new run_command criteria are structurally consistent with existing patterns, the jq assertions are sound, and criterion count/weights are preserved. Approve.

[github-actions]

Claude finished @cosmyo's task in 3m 6s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 1 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

⚠️ Borderline — PR body does not explicitly claim the modified task was re-run and passed end-to-end, but does state: "Both jq programs were verified against the captured live output from the failing run (each returns true, exit 0)." The author verified the new criteria against real tenant output from the run that exposed the flake (0.933 / FAILURE), demonstrating the modified task would have passed. This is semantically close to a passing-run claim, but consider adding a line like: Ran skill-mcp-servers-e2e-uipath-allkinds and it passed. after the next nightly to make it explicit.

Per-task lint

tests/tasks/uipath-mcp-servers/e2e-uipath-allkinds/task.yaml — verdict: OK

The change replaces two command_executed / tool_name: Read proxies (lines 99–125 old) with two run_command criteria that validate live tenant outcomes — Gmail tool connector/operation metadata and Jira cascade field resolution. This is a textbook improvement per the repo rule "Grade Behavior, Not Self-Reports":

• Self-report anti-pattern: Clean. report.json is an operational fixture for cleanup/slug lookup, not a self-assessment.
• Prompt over-specification: Clean. Prompt supplies ground-truth anchors (entity names, connection IDs, folder keys) required for the live tenant — no procedure leakage.
• Meaningful coverage: Strong. 13 criteria span command execution (7), skill trigger (1), live tenant validation (3), and file existence (1). The new run_command criteria with jq -e assertions on metadata fields are more meaningful than the replaced read-proxies.
• Could pass for the wrong reason: Clean. run_command criteria query the live tenant — can't be faked without actually creating the tools.
• Near-duplicate: Clean. Only e2e task exercising all tool kinds on a single server; neighbors test individual server types or smoke-tier mocks.
• CLI verb reachability: Skipped (sandbox did not permit python3 scripts/check-cli-verbs.py). PR body reports lint-task CLI-verb check as all Info.
• Redundant uip CLI in sandbox: Clean — no sandbox block; inherits from experiment.
• Run-limit fields under agent: Clean — run_limits: at top level (lines 12–14), no agent: block.

Conclusion

✅ All changed tasks pass the rubric. Evidence of passing run is borderline-acceptable (verified against captured live output, not a full re-run). Consider confirming after the next nightly.

---