test(uipath-maestro-case): add aged-invoice Phase-0 doc-derived over-cap task

[abhiram-vad]

Summary

Adds a uipath-maestro-case Phase-0 coder-eval task seeded from the aged-invoice case-management PDD (Smit's Woolworths-style PoV), exercising the doc-derived Listen path and the over-cap continue behavior.

• Stages the full aged-invoice PDD as fixtures/pdd.md. Phase 0 reads it via the Listen doc-detection path (phase-0-interview.md § File / attachment / document detection) and sketches sdd.draft.md from it.
• The PDD is intentionally over every Phase-0 complexity cap (9 primary stages, 4 exception lanes, many integrations, a child case). At that size the skill would normally soft-redirect to uipath-planner; the prompt + simulation persona instruct the agent to stay in uipath-maestro-case and continue past the threshold prompt (Continue-with-warning), not redirect.
• Stops at the design document — no tasks.md, no caseplan.json build.

Fourth domain alongside candidate_interview, loan_origination, and expense_reimbursement; the only PDD-seeded, over-cap case.

Success criteria

Criterion	Weight	Purpose
Design document produced (sdd.draft.md/sdd.md)	1.5	Phase 0 ran
≥8 of the PDD's 9 primary stages captured	1.5	Anti-redirect signal — a soft-redirect emits a plain-text planner suggestion, not a 9-stage maestro-case draft
SLA-escalation + automation-incident exception lanes	1.0	Exception modeling
Triage root-cause + priority/SLA scoring	1.0	Domain anchor
Full task-type mix (api-workflow/agent/action/rpa/timer/connector)	1.0	Coverage
No caseplan.json	1.0	Stayed in Phase 0
Domain-coherence LLM judge (rewards "full case survived, not trimmed/redirected")	3.0	Coherence

Stage counting follows the sibling convention (### Stage N: for primary; exception lanes matched by name). No warning-header criterion: the banner lands in the finalized sdd.md, but this task stops at the draft, so requiring it would false-fail — the ≥8-stage count carries the no-redirect assertion instead.

Verification

• YAML parses; required tags present (uipath-maestro-case + integration + mode:build).
• python3 scripts/check-cli-verbs.py --json <task> → exit 0, no findings.
• sandbox stages fixtures/ via template_dir; no node.env_packages (no CLI pinning).
• /lint-task → OK (0 Critical/High/Medium/Low).
• Grep criteria self-tested: PASS on a coherent multi-stage draft, FAIL when truncated.
• Not yet run end-to-end through coder-eval — needs a sandbox + Claude Code access. Opening for review; a passing run will confirm the agent honors "continue" and emits ≥8 stages under this steer.

🤖 Generated with Claude Code

[github-actions]

Claude finished @abhiram-vad's task in 3m 9s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdicts: 0 Critical, 1 High, 0 Medium, 0 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states: "Not yet run end-to-end through coder-eval — needs a sandbox + Claude Code access." Please edit the PR description to add a passing-run claim once verified, e.g.: Ran skill-case-phase-0-aged-invoice-payment locally and it passed.

Per-task lint

tests/tasks/uipath-maestro-case/phase_0_to_case/aged_invoice_payment/aged_invoice_payment.yaml — verdict: High

The High verdict is solely from the missing passing-run evidence above. The task design itself is clean across all rubric axes:

• Self-report anti-pattern: No issue — sdd.draft.md is the skill's actual design artifact, not a self-grading summary. Criteria check structural content of the artifact, not agent self-reports.
• Prompt over-specification: No issue — prompt states the goal ("run Phase 0 interview, seeding from the PDD"), the behavior constraint ("do NOT redirect"), and the stop condition ("draft only"). Does not prescribe CLI flags, internal skill steps, or procedural details. "Load the uipath-maestro-case skill" is a routing hint, not over-specification.
• Meaningful coverage: No issue — 5 run_command content checks (≥8 stage headings, exception lanes, triage/priority, task-type mix) + 1 negative boundary check (no caseplan.json) + 1 llm_judge at weight 3.0 with a detailed domain-coherence prompt. Good mix of structural greps and holistic judge.
• Could pass for the wrong reason: No issue — while individual grep criteria could be gamed, the LLM judge (weight 3.0, pass_threshold 0.7) validates end-to-end domain coherence against a detailed rubric (stage order, task-type fit, exception-lane triggers, semantic gaps). A dummy file would fail the judge.
• Near-duplicate: No issue — compared against siblings candidate_interview, loan_origination, expense_reimbursement. This task is the only PDD-seeded (doc-derived Listen path) and over-cap (Continue-with-warning) case. Different domain (aged invoice AP), different scale (9 stages / 4 exception lanes), different entry mechanism (fixture file vs. interview). Scaffold reuse is appropriate, not duplication.
• Validate-only flow tests: N/A — not a uipath-maestro-flow task.
• CLI verb reachability: N/A — no command_executed criteria present (all criteria are run_command and llm_judge).
• Redundant or pinned uip CLI in sandbox: No issue — sandbox uses only template_sources with template_dir: fixtures; no node.env_packages.
• Run-limit fields under agent: No issue — max_turns, turn_timeout, task_timeout correctly placed under top-level run_limits:, not under agent:.

Within-PR duplicates

No duplicate clusters detected (only 1 task YAML added).

Conclusion

⚠ 1 task has issues, max severity High. The sole finding is the missing passing-run evidence — the task design is well-structured and follows the established Phase-0 sibling convention. Advisory only — not blocking merge.

---

[github-actions]

Claude finished @abhiram-vad's task in 2m 59s —— View job

---

PR Review: test(uipath-maestro-case) — aged-invoice Phase-0 task

• Gather context (review criteria, project conventions, existing comments)
• Read full diff and changed files
• Run skill-specific checks (tags, task_id, success_criteria, sandbox)
• Verify fixture content and links
• Check for secrets/paths/cross-skill refs
• Post review findings

---

Summary

Adds a fourth Phase-0 integration task for uipath-maestro-case — an aged-invoice PDD-seeded, over-cap scenario. The task stages a 1327-line process design document as fixtures/pdd.md, exercises the doc-derived Listen path, and tests that the agent continues past the complexity-threshold prompt rather than soft-redirecting to uipath-planner. Well-structured PR that closely follows sibling task conventions.

Change-by-Change Review

1. tests/tasks/uipath-maestro-case/phase_0_to_case/aged_invoice_payment/aged_invoice_payment.yaml (+210)

Severity: OK

The task YAML is well-crafted and closely mirrors the sibling tasks (expense_reimbursement, loan_origination, candidate_interview).

• task_id (skill-case-phase-0-aged-invoice-payment): Follows the skill-<domain>-<capability> convention. Consistent with siblings.
• Tags (uipath-maestro-case, integration, lifecycle:generate, mode:build, shape:multi-node, feature:phase-0, feature:stages, feature:case-exit, feature:conditions): All required tags present; all values from the closed vocabulary. Intentionally omits feature:trigger (correct — this is doc-derived, not trigger-based, unlike expense_reimbursement).
• Agent config: Matches siblings — adds AskUserQuestion and TodoWrite to allowed_tools, caps max_thinking_tokens: 10000.
• run_limits: Higher than siblings (task_timeout: 3600 vs 3000, turn_timeout: 2700 vs 2400). Justified by the larger 9-stage + 4-exception design. Comment at lines 40-45 explains the rationale.
• sandbox.template_sources: Correctly stages fixtures/ via template_dir. No node.env_packages (compliant with the no-CLI-pinning rule).
• initial_prompt (lines 57-75): Minimal and goal-oriented. Clearly instructs the agent to load the skill, seed from PDD, and stay in uipath-maestro-case on the over-cap threshold. Stops at draft.
• success_criteria (lines 78-178): 7 criteria covering file existence (1.5), ≥8/9 stage count (1.5), exception lanes (1.0), triage (1.0), task-type mix (1.0), no caseplan.json (1.0), and an LLM judge (3.0). Weights and pass_threshold values follow sibling conventions. The stage-count criterion (≥8 of 9) doubles as the anti-redirect signal — clever and well-documented.
• LLM judge (lines 137-178): Comprehensive prompt covering stage order, task-type fit, exception lane behavior, triage/risk gating, and semantic-gap detection. max_file_chars: 150000 is generous but appropriate for a 9-stage draft. pass_threshold: 0.7 matches siblings.
• Simulation (lines 181-210): Persona, goal, and constraints are thorough. The "always choose to continue" constraint (line 204) is essential for the over-cap test. max_turns: 40 matches siblings.

One observation: the task-type mix criterion (line 122) checks 6 of the 7 types listed in the description (misses case-management/child-case). This is acceptable since the LLM judge prompt explicitly checks for "a payment-tracking child case" (line 162) with weight 3.0. Adding it to the grep chain would make the criterion more fragile without significant benefit.

2. tests/tasks/uipath-maestro-case/phase_0_to_case/aged_invoice_payment/fixtures/pdd.md (+1327)

Severity: OK

A comprehensive, fictional aged-invoice PDD that covers 9 primary stages, 4 exception lanes, 5 agent roles, 7 demo scenarios, ~60 tasks, Data Fabric entities, SLA rules, and acceptance criteria. The document is large by design — the test specifically exercises the over-cap continue path.

• No real company data, credentials, or personal paths (grep confirmed — only contextual mentions of "credentials" in documentation prose).
• Fictional supplier names and invoice data throughout.
• Well-structured with section numbering, tables, and clear stage/task/rule taxonomies.
• The PDD intentionally exceeds every Phase-0 complexity cap (9 stages, 4 exception lanes, many integrations, child case), which is the test's core assertion.

What's Missing

• No passing-run claim — acknowledged in the PR description. The PR body states: "Not yet run end-to-end through coder-eval — needs a sandbox + Claude Code access." A passing run should be added before merge to validate that the criteria work as expected.
• Nothing else identified.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	This is a test task YAML, not a skill SKILL.md
E2E Tests	OK	Integration-tier task; no new skill folder added, so smoke+e2e pair not required
Skill Body	N/A	No skill changes
References & Assets	OK	PDD fixture is well-structured, no broken links
Repo Hygiene	OK	CODEOWNERS covers /tests/tasks/uipath-maestro-case/; no secrets; scoped changes

Issues for Manual Review

• Over-cap continue behavior: The test's primary assertion (agent stays in uipath-maestro-case and produces ≥8 stages rather than soft-redirecting) depends on the prompt steer + simulation persona working together. This can only be validated by an end-to-end run — the criteria themselves are well-designed to detect the difference.
• PDD faithfulness: The LLM judge's domain-coherence scoring depends on the PDD's content aligning with what the skill's Phase 0 interview can reasonably consume and convert. A passing run will confirm this.

Conclusion

Clean, well-structured PR that follows established sibling conventions closely. The task design is thoughtful — the ≥8-stage criterion as an anti-redirect signal is particularly well-reasoned. The only gap is the missing end-to-end passing-run claim, which the author has already flagged. Approve once a passing run is documented.

---