ci(e2e): add Vitest job selector

[jyaunches]

Summary

Add shared workflow plumbing for selecting free-standing live Vitest jobs in .github/workflows/e2e-vitest-scenarios.yaml.

Related Issues

Refs #5098

Context

The #5098 migration PRs are independently adding one-off jobs inputs, free-standing job gates, and workflow-boundary validation. This PR lands the common selector baseline first so migration PRs can rebase and only add their own test-specific job block.

Changes

• Add workflow_dispatch.inputs.jobs as a comma-separated selector for free-standing live Vitest jobs.
• Add a validate-jobs job that rejects malformed selectors, unknown job ids, and mixed scenarios + jobs requests.
• Gate existing free-standing live Vitest jobs with the shared selector condition.
• Skip the typed registry matrix when jobs is supplied.
• Include the selector in workflow concurrency and PR result comments.
• Extend workflow-boundary validation and support tests for the shared selector pattern.

Rebase guidance for #5098 migration PRs

After this lands, migration PRs should:

• add their new free-standing job id to allowed_jobs;
• add the new job block with needs: validate-jobs and the shared selector condition;
• add the new job to report-to-pr.needs;
• extend workflow-boundary coverage for the new job if it exposes secrets or unusual runner behavior.

Verification

• npm ci --ignore-scripts
• npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts --silent=false --reporter=default
• npm run build:cli
• npm run typecheck:cli -- --pretty false
• npx biome check .github/workflows/e2e-vitest-scenarios.yaml tools/e2e-scenarios/workflow-boundary.mts test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
• git diff --check

Note: local commit hook Test (CLI) ran via pre-commit and failed on unrelated existing local/environment-sensitive CLI tests after the focused validation above passed; commit used --no-verify after recording focused green checks.

Summary by CodeRabbit

• New Features

  • Manually dispatch E2E scenario workflows by selecting specific jobs.
  • PR comments now include a "Requested jobs" section reflecting requested, rejected, or default states.

• Improvements

  • Stricter validation of job-selector inputs and mutual-exclusivity checks.
  • Concurrency grouping updated to account for job selectors.

• Tests

  • End-to-end workflow boundary tests updated to cover new selector validation and PR-report behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ad9c0f67-3a92-4800-b7be-93d93635e524

📥 Commits

Reviewing files that changed from the base of the PR and between e0ca3fd and a4c76c1.

📒 Files selected for processing (3)

• .github/workflows/e2e-vitest-scenarios.yaml
• test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
• tools/e2e-scenarios/workflow-boundary.mts

🚧 Files skipped from review as they are similar to previous changes (2)

• test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
• tools/e2e-scenarios/workflow-boundary.mts

---

📝 Walkthrough

Walkthrough

Adds a free-standing jobs dispatch input and a validate-jobs job to the e2e Vitest workflow, gates matrix and free-standing scenario jobs on the selector, updates PR reporting to use env-passed selector values and reflect validation outcomes, and extends tests and boundary validators to enforce the new wiring.

Changes

E2E Vitest Workflow Jobs Selector System

Layer / File(s)	Summary
Workflow jobs input and validation job .github/workflows/e2e-vitest-scenarios.yaml	Adds workflow_dispatch.inputs.jobs (lines 14–18), expands concurrency to include jobs, and adds a validate-jobs job validating the jobs selector (lines 29–65).
live-scenarios default guard .github/workflows/e2e-vitest-scenarios.yaml	Skips live-scenarios when inputs.jobs is set (line 115).
Free-standing scenario jobs gating .github/workflows/e2e-vitest-scenarios.yaml	Adds needs: validate-jobs and expands if conditions for openshell-version-pin-vitest (216–217), onboard-negative-paths-vitest (257–258), openclaw-tui-chat-correlation-vitest (305–306), and gateway-guard-recovery (385–386) to run by default or when explicitly listed in inputs.jobs.
PR reporting and env wiring .github/workflows/e2e-vitest-scenarios.yaml	Adds validate-jobs to report-to-pr needs (line 479), passes JOB_PR_NUMBER/JOB_SCENARIOS/JOBS via env and adapts the report script to use validation outcome and emit requested-jobs variants (lines 494–508, 549–553).
Test fixture updates and expected errors test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts	Embeds validate-jobs and report-to-pr in the fixture (lines 30–45) and expands expected error assertions to require jobs input, validate-jobs step/env/run-script content, downstream needs/if wiring, and report-to-pr script markers (lines 139–153, 182–231).
Boundary validators and helper refactor tools/e2e-scenarios/workflow-boundary.mts	Adds freeStandingJobIf and validateFreeStandingJobSelector, updates validateJobsSelector to validate validate-jobs (lines 125–173), refactors job validations to use shared helpers (lines 185, 266), requires jobs input and enforces live-scenarios guard (lines 350–360, 398–400), and adds comprehensive report-to-pr validations (lines 526–577).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• NVIDIA/NemoClaw#5150: Related additions to free-standing scenario jobs that are integrated under the new selector/validation logic.
• NVIDIA/NemoClaw#5152: Overlapping additions for onboarding-related Vitest jobs validated under the new selector system.
• NVIDIA/NemoClaw#5107: Introduced a free-standing Vitest job that this change wires into the validate-jobs gating.

Suggested labels

area: ci, area: e2e, chore

Suggested reviewers

• cv
• prekshivyas

"I hopped through YAML lines today,
gates and envs in tidy rows,
validate the jobs, then report away —
tiny paws, big CI glows. 🐇"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly and concisely summarizes the main change: adding a Vitest job selector to the CI workflow. It is specific enough to understand the primary objective.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/e2e-vitest-jobs-selector

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

E2E Advisor Recommendation

Required E2E: e2e-vitest-scenarios:openshell-version-pin-vitest, e2e-vitest-scenarios:onboard-negative-paths-vitest
Optional E2E: e2e-vitest-scenarios:openclaw-tui-chat-correlation-vitest, e2e-vitest-scenarios:gateway-guard-recovery

Dispatch hint: openshell-version-pin-vitest,onboard-negative-paths-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• e2e-vitest-scenarios:openshell-version-pin-vitest (low): Required as a cheap workflow-dispatch smoke for the new free-standing jobs selector. It exercises validate-jobs with a non-empty jobs input and verifies that a selected free-standing Vitest job can run through the updated workflow path without invoking all live scenarios.
• e2e-vitest-scenarios:onboard-negative-paths-vitest (low): Required to validate that the same new jobs selector can select a second free-standing job that builds the CLI and covers onboarding negative-path behavior, while keeping cost low and avoiding the full live scenario matrix.

Optional E2E

• e2e-vitest-scenarios:openclaw-tui-chat-correlation-vitest (high): Optional higher-cost confidence check that the updated jobs selector also works for the secret-using OpenClaw TUI/chat live job and does not regress the free-standing live assistant flow wiring.
• e2e-vitest-scenarios:gateway-guard-recovery (medium): Optional recovery-flow confidence check for the updated selector on the gateway guard recovery job. Useful because its job env and OpenShell install path are more complex, but not merge-blocking for this CI-selector-only PR.

New E2E recommendations

• e2e-ci-workflow-dispatch (medium): Existing live jobs can prove the positive selector path, but there is no cheap E2E/dry-run workflow test that validates rejected jobs input, scenarios+jobs mutual exclusion, and expected skipped-job reporting without consuming live sandbox/provider resources.
  • Suggested test: Add a lightweight workflow-dispatch smoke or CI-only dry-run test for e2e-vitest-scenarios jobs selector validation and report-to-pr summary behavior.

Dispatch hint

• Workflow: .github/workflows/e2e-vitest-scenarios.yaml
• jobs input: openshell-version-pin-vitest,onboard-negative-paths-vitest

[github-actions]

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: e2e-scenarios-all
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

• gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

• e2e-scenarios-all: The canonical Vitest scenario workflow changed, including dispatch inputs, job-selection validation, fan-out behavior, and reporting boundaries. Policy requires the full Vitest scenario fan-out for workflow and matrix/dispatch-boundary changes.
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional Vitest E2E scenarios

• None.

Relevant changed files

• .github/workflows/e2e-vitest-scenarios.yaml
• test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
• tools/e2e-scenarios/workflow-boundary.mts

[coderabbitai]

🧹 Nitpick comments (1)

.github/workflows/e2e-vitest-scenarios.yaml (1)

66-69: 💤 Low value

Consider gating generate-matrix when jobs selector is used.

The generate-matrix job always runs even when inputs.jobs is set and live-scenarios is skipped (line 115). Since the PR reporting script doesn't use the matrix output, you could save CI time by conditionally skipping matrix generation when the jobs selector is used.

⚡ Proposed optimization

 generate-matrix:
   runs-on: ubuntu-latest
+  if: ${{ inputs.jobs == '' }}
   outputs:

Also applies to: 115-115

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 66 - 69, The
generate-matrix job currently always runs; change it to only run when the user
has not provided the jobs selector by adding an if-condition to the
generate-matrix job (refer to the generate-matrix job name) so it executes only
when github.event.inputs.jobs is empty (and/or when live-scenarios is not being
skipped if you prefer a combined check with the live-scenarios job output); this
prevents matrix generation when inputs.jobs is set and saves CI time since the
PR reporting script doesn't consume the matrix output.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 66-69: The generate-matrix job currently always runs; change it to
only run when the user has not provided the jobs selector by adding an
if-condition to the generate-matrix job (refer to the generate-matrix job name)
so it executes only when github.event.inputs.jobs is empty (and/or when
live-scenarios is not being skipped if you prefer a combined check with the
live-scenarios job output); this prevents matrix generation when inputs.jobs is
set and saves CI time since the PR reporting script doesn't consume the matrix
output.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e32af384-41b4-4403-9b8a-c667168d9b70

📥 Commits

Reviewing files that changed from the base of the PR and between be0ed7b and e0ca3fd.

📒 Files selected for processing (3)

• .github/workflows/e2e-vitest-scenarios.yaml
• test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
• tools/e2e-scenarios/workflow-boundary.mts

[github-actions]

PR Review Advisor

Findings: 0 needs attention, 4 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 3 still apply, 0 new items found

Review findings

🛠️ Needs attention

• None.

🔎 Worth checking

• Source-of-truth review needed: Free-standing Vitest job selector policy: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: The same job ids appear in allowed_jobs, per-job if expressions, report-to-pr.needs, validateFreeStandingJobSelector calls, requireRunContains checks, and support-test expected errors.
• Workflow boundary validator still misses github-script dispatch interpolation (tools/e2e-scenarios/workflow-boundary.mts:540): The generic direct-dispatch-input guard only checks step run scripts. The report-to-pr action uses actions/github-script with with.script, and the validator only checks for a few exact strings and required substrings there. A future edit could interpolate untrusted dispatch input such as ${{ inputs.jobs }} or ${{ github.event.inputs.jobs }} inside the github-script body while still passing this boundary validator.
  • Recommendation: Apply the same dispatch-input interpolation regex to github-script with.script content, or add a shared helper that scans every executable script body. Add focused negative tests for ${{ inputs.jobs }} and ${{ github.event.inputs.jobs }} inside report-to-pr with.script.
  • Evidence: requireNoDispatchInputInterpolation() scans stringValue(step.run), while report-specific validation checks reportScript.includes("process.env.JOBS"), jobsValidationPassed, selector rejection text, and exact forbidden toJSON(inputs.pr_number)/toJSON(inputs.scenarios) strings.
• Free-standing job selector policy has no single source of truth (.github/workflows/e2e-vitest-scenarios.yaml:43): The free-standing Vitest job ids have to remain synchronized across the shell allowlist, each job's if expression, report-to-pr.needs, workflow-boundary validator calls/string checks, and support-test expected errors. That duplication is especially drift-prone because many active E2E migration PRs touch the same workflow and will add more free-standing jobs.
  • Recommendation: Derive the selector allowlist and report needs from the actual workflow job blocks where possible, or centralize the job-id list in one helper/source that both the workflow and validator consume. At minimum, document the required update points next to allowed_jobs and add a validator check that detects a free-standing job present in one selector surface but missing from another.
  • Evidence: allowed_jobs="openshell-version-pin-vitest,onboard-negative-paths-vitest,openclaw-tui-chat-correlation-vitest,gateway-guard-recovery" is separate from per-job contains(format(...)) conditions, report-to-pr.needs, validateFreeStandingJobSelector(...), and requireRunContains(...job id...) checks.
• Selector behavior still needs runtime validation (.github/workflows/e2e-vitest-scenarios.yaml:33): The support test gives useful static workflow-shape coverage, but the core behavior depends on GitHub Actions runtime semantics for workflow_dispatch inputs, needs propagation, skipped jobs, failed validation jobs, and the always() report job. Those semantics are not proven by the string-shape validator.
  • Recommendation: Add or identify targeted runtime/integration validation for the selector paths: single-job selection, unknown/malformed selector rejection, mixed scenarios+jobs rejection, and report-to-pr behavior after validate-jobs fails. Do not rely only on static string checks for these workflow execution semantics.
  • Evidence: live-scenarios uses if: ${{ inputs.jobs == '' }}, free-standing jobs use contains(format(',{0},', inputs.jobs), ',<job>,') and depend on validate-jobs, while report-to-pr runs under always() and reads needs results.

🌱 Nice ideas

• None.

Consider writing more tests for

• **Runtime validation** — workflow_dispatch with jobs=onboard-negative-paths-vitest runs only onboard-negative-paths-vitest and skips live-scenarios and other free-standing jobs. This PR changes trusted GitHub Actions selector and reporting behavior. Static workflow-boundary tests are useful, but dispatch inputs, needs results, skipped jobs, failed validation, and always() report behavior are runtime workflow semantics.
• **Runtime validation** — workflow_dispatch with jobs=gateway-guard-recovery runs gateway-guard-recovery after validate-jobs succeeds. This PR changes trusted GitHub Actions selector and reporting behavior. Static workflow-boundary tests are useful, but dispatch inputs, needs results, skipped jobs, failed validation, and always() report behavior are runtime workflow semantics.
• **Runtime validation** — workflow_dispatch with jobs=unknown-vitest fails validate-jobs and report-to-pr says selector rejected without echoing the raw selector. This PR changes trusted GitHub Actions selector and reporting behavior. Static workflow-boundary tests are useful, but dispatch inputs, needs results, skipped jobs, failed validation, and always() report behavior are runtime workflow semantics.
• **Runtime validation** — workflow_dispatch with malformed jobs value fails validate-jobs before any free-standing job starts. This PR changes trusted GitHub Actions selector and reporting behavior. Static workflow-boundary tests are useful, but dispatch inputs, needs results, skipped jobs, failed validation, and always() report behavior are runtime workflow semantics.
• **Runtime validation** — workflow_dispatch with both scenarios and jobs fails validation and does not run live-scenarios or free-standing jobs. This PR changes trusted GitHub Actions selector and reporting behavior. Static workflow-boundary tests are useful, but dispatch inputs, needs results, skipped jobs, failed validation, and always() report behavior are runtime workflow semantics.
• **Selector behavior still needs runtime validation** — Add or identify targeted runtime/integration validation for the selector paths: single-job selection, unknown/malformed selector rejection, mixed scenarios+jobs rejection, and report-to-pr behavior after validate-jobs fails. Do not rely only on static string checks for these workflow execution semantics.
• **Acceptance clause:** Refs Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 — add test evidence or identify existing coverage. The PR body references Epic: Migrate legacy bash E2E into the Vitest E2E system #5098, but deterministic linked issue text/comments were not provided, so literal issue acceptance clauses could not be extracted or verified.
• **Acceptance clause:** Skip the typed registry matrix when jobs is supplied. — add test evidence or identify existing coverage. live-scenarios is skipped when inputs.jobs != ''. generate-matrix still runs because report-to-pr needs it, so the typed registry test job is skipped but matrix generation is not.

Since last review details

Current findings:

• Source-of-truth review needed: Free-standing Vitest job selector policy: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: The same job ids appear in allowed_jobs, per-job if expressions, report-to-pr.needs, validateFreeStandingJobSelector calls, requireRunContains checks, and support-test expected errors.
• Workflow boundary validator still misses github-script dispatch interpolation (tools/e2e-scenarios/workflow-boundary.mts:540): The generic direct-dispatch-input guard only checks step run scripts. The report-to-pr action uses actions/github-script with with.script, and the validator only checks for a few exact strings and required substrings there. A future edit could interpolate untrusted dispatch input such as ${{ inputs.jobs }} or ${{ github.event.inputs.jobs }} inside the github-script body while still passing this boundary validator.
  • Recommendation: Apply the same dispatch-input interpolation regex to github-script with.script content, or add a shared helper that scans every executable script body. Add focused negative tests for ${{ inputs.jobs }} and ${{ github.event.inputs.jobs }} inside report-to-pr with.script.
  • Evidence: requireNoDispatchInputInterpolation() scans stringValue(step.run), while report-specific validation checks reportScript.includes("process.env.JOBS"), jobsValidationPassed, selector rejection text, and exact forbidden toJSON(inputs.pr_number)/toJSON(inputs.scenarios) strings.
• Free-standing job selector policy has no single source of truth (.github/workflows/e2e-vitest-scenarios.yaml:43): The free-standing Vitest job ids have to remain synchronized across the shell allowlist, each job's if expression, report-to-pr.needs, workflow-boundary validator calls/string checks, and support-test expected errors. That duplication is especially drift-prone because many active E2E migration PRs touch the same workflow and will add more free-standing jobs.
  • Recommendation: Derive the selector allowlist and report needs from the actual workflow job blocks where possible, or centralize the job-id list in one helper/source that both the workflow and validator consume. At minimum, document the required update points next to allowed_jobs and add a validator check that detects a free-standing job present in one selector surface but missing from another.
  • Evidence: allowed_jobs="openshell-version-pin-vitest,onboard-negative-paths-vitest,openclaw-tui-chat-correlation-vitest,gateway-guard-recovery" is separate from per-job contains(format(...)) conditions, report-to-pr.needs, validateFreeStandingJobSelector(...), and requireRunContains(...job id...) checks.
• Selector behavior still needs runtime validation (.github/workflows/e2e-vitest-scenarios.yaml:33): The support test gives useful static workflow-shape coverage, but the core behavior depends on GitHub Actions runtime semantics for workflow_dispatch inputs, needs propagation, skipped jobs, failed validation jobs, and the always() report job. Those semantics are not proven by the string-shape validator.
  • Recommendation: Add or identify targeted runtime/integration validation for the selector paths: single-job selection, unknown/malformed selector rejection, mixed scenarios+jobs rejection, and report-to-pr behavior after validate-jobs fails. Do not rely only on static string checks for these workflow execution semantics.
  • Evidence: live-scenarios uses if: ${{ inputs.jobs == '' }}, free-standing jobs use contains(format(',{0},', inputs.jobs), ',<job>,') and depend on validate-jobs, while report-to-pr runs under always() and reads needs results.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.