test(e2e): migrate smoke and onboarding scenarios by cv · Pull Request #5054 · NVIDIA/NemoClaw

cv · 2026-06-09T18:07:54Z

Summary

Migrate the smoke/onboarding family from placeholder status into registry-backed live Vitest coverage.

This PR expands the live Vitest scenario runner so onboarding fixtures can execute the supported smoke/onboarding variants directly from typed scenario metadata. It also keeps the legacy typed shell runner aligned for the migrated negative onboarding paths until that runner is retired.

Related Issue

Refs #4941
Refs #4990
Refs #4348
Depends on #5046, #5052, and #5053.
Stacked on branch codex/e2e-fanout-02-onboarding-variant-fixtures.

Changes

Mark the smoke/onboarding family as live-supported where fixtures now own the setup:
- cloud OpenClaw smoke
- no-Docker preflight negative
- resume after interrupt
- repair existing config
- double same-provider onboarding
- custom policies
- invalid NVIDIA key negative
- gateway port conflict negative
Keep provider-switch onboarding unsupported until inference/provider-switch fixtures own that behavior.
Stop requiring a real NVIDIA_API_KEY for the invalid-key negative scenario; both the Vitest fixture path and shell dispatcher path inject not-a-nvidia-key.
Assert and record expected-failure contracts in scenario-result.json for negative live scenarios.
Hold the configured gateway port in the legacy shell dispatcher for cloud-openclaw-gateway-port-conflict, with validation and cleanup coverage.
Exclude test/e2e-scenario/** from the regular cli Vitest project so live scenario tests remain opt-in through e2e-scenarios-live.
Update matrix, support, compiler, dispatcher, and project-gating tests for the migrated scenario family.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

Note: the full touched-file commit hook remains blocked in this local environment by unrelated SSH signing failures inside test/release-latest-tag.test.ts temporary git repositories. The relevant touched-file hooks were run with --skip test-cli, and the full framework/typecheck/CI checks above passed.

Signed-off-by: Carlos Villela cvillela@nvidia.com

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

copy-pr-bot · 2026-06-09T18:07:58Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-06-09T18:08:01Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 682b603a-4c16-425f-9322-8f16e38c3146

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/e2e-fanout-03-smoke-onboarding-scenarios

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-09T18:08:41Z

E2E Advisor Recommendation

Required E2E: .github/workflows/e2e-vitest-scenarios.yaml#live-scenarios, .github/workflows/e2e-scenarios.yaml#run-scenario
Optional E2E: .github/workflows/e2e-vitest-scenarios.yaml#live-scenarios, .github/workflows/e2e-scenarios-all.yaml#run-scenario

Dispatch hint: scenarios=ubuntu-repo-cloud-openclaw,ubuntu-no-docker-preflight-negative,ubuntu-repo-cloud-openclaw-resume,ubuntu-repo-cloud-openclaw-repair,ubuntu-repo-cloud-openclaw-double-same-provider,ubuntu-repo-cloud-openclaw-custom-policies,ubuntu-invalid-nvidia-key-negative,ubuntu-gateway-port-conflict-negative

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/codex/e2e-fanout-02-onboarding-variant-fixtures
Head: HEAD
Confidence: high

Required E2E

.github/workflows/e2e-vitest-scenarios.yaml#live-scenarios (high; one ubuntu-latest live E2E matrix job per selected scenario, installs/builds CLI and mutates real NemoClaw/OpenShell state): Run the live Vitest-supported onboarding matrix because this PR changes the live support whitelist, matrix generation, fixture dispatcher, expected-failure contract, and onboarding fixture behavior. Include the baseline plus newly supported onboarding variants: ubuntu-repo-cloud-openclaw, ubuntu-no-docker-preflight-negative, ubuntu-repo-cloud-openclaw-resume, ubuntu-repo-cloud-openclaw-repair, ubuntu-repo-cloud-openclaw-double-same-provider, ubuntu-repo-cloud-openclaw-custom-policies, ubuntu-invalid-nvidia-key-negative, and ubuntu-gateway-port-conflict-negative.
.github/workflows/e2e-scenarios.yaml#run-scenario (high; one ubuntu-latest scenario-runner job executing real install/onboard negative flows): Run the typed shell scenario runner for the negative onboarding paths because compiler.ts and nemoclaw_scenarios/onboard/dispatch.sh changed the real shell dispatch path, secretEnv propagation, invalid-key injection, gateway-port holder, and docker-missing executable profile routing.

Optional E2E

.github/workflows/e2e-vitest-scenarios.yaml#live-scenarios (high): Run ubuntu-repo-docker-post-reboot-recovery as an optional confidence check because live registry and matrix code were touched and this scenario remains part of the supported live set, but the PR does not directly change lifecycle fixture dispatch.
.github/workflows/e2e-scenarios-all.yaml#run-scenario (very high; all typed scenario runner jobs across supported runners): Optional full fan-out after merge or as a maintainer-triggered soak if the team wants broad confidence that typed registry matrix changes did not disturb unrelated platform routes.

New E2E recommendations

Workflow route and typed registry parity (medium): The scenario runner workflow keeps a static ROUTES table while the typed registry and live matrix are generated dynamically. This PR changes supported scenario sets, so a future gap could leave a valid scenario unrunnable by workflow_dispatch.
- Suggested test: Add a workflow-contract test that validates .github/workflows/e2e-scenarios.yaml route entries against buildScenarioMatrix()/registered scenario ids.
Unsupported onboarding variants (medium): Provider-switch and token-rotation OpenClaw onboarding scenarios remain skipped by live Vitest support even though they are security- and credential-sensitive user flows adjacent to the changed onboarding profile resolver.
- Suggested test: Add live fixture coverage for ubuntu-repo-cloud-openclaw-double-provider-switch and ubuntu-repo-cloud-openclaw-token-rotation once their inference/credential fixtures are wired.

Dispatch hint

Workflow: .github/workflows/e2e-vitest-scenarios.yaml
jobs input: scenarios=ubuntu-repo-cloud-openclaw,ubuntu-no-docker-preflight-negative,ubuntu-repo-cloud-openclaw-resume,ubuntu-repo-cloud-openclaw-repair,ubuntu-repo-cloud-openclaw-double-same-provider,ubuntu-repo-cloud-openclaw-custom-policies,ubuntu-invalid-nvidia-key-negative,ubuntu-gateway-port-conflict-negative

github-actions · 2026-06-09T18:08:42Z

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/codex/e2e-fanout-02-onboarding-variant-fixtures
Head: HEAD
Confidence: high

Required scenario E2E

e2e-scenarios-all: Scenario framework/runner logic and typed scenario catalog/runtime support changed, including onboarding phase execution, onboarding dispatch helpers, compiler routing, live scenario support, and baseline scenario metadata. Policy requires the all-scenarios fan-out for scenario runtime/runner or catalog-affecting changes.
- Dispatch: gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Optional scenario E2E

None.

Relevant changed files

test/e2e-scenario/framework-tests/e2e-live-project-config.test.ts
test/e2e-scenario/framework-tests/e2e-live-registry-discovery.test.ts
test/e2e-scenario/framework-tests/e2e-onboard-dispatch.test.ts
test/e2e-scenario/framework-tests/e2e-phase-onboarding.test.ts
test/e2e-scenario/framework-tests/e2e-phase-orchestrators.test.ts
test/e2e-scenario/framework-tests/e2e-scenario-matrix.test.ts
test/e2e-scenario/framework/phases/onboarding.ts
test/e2e-scenario/live/registry-scenarios.test.ts
test/e2e-scenario/nemoclaw_scenarios/onboard/dispatch.sh
test/e2e-scenario/scenarios/compiler.ts
test/e2e-scenario/scenarios/onboarding-profiles.ts
test/e2e-scenario/scenarios/runtime-support.ts
test/e2e-scenario/scenarios/scenarios/baseline.ts
vitest.config.ts

github-actions · 2026-06-09T18:12:00Z

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 0 still apply, 0 new items found

Consider writing more tests for

**Runtime validation** — Validate ubuntu-invalid-nvidia-key-negative runs without parent NVIDIA_API_KEY, injects not-a-nvidia-key, writes scenario-result.json expectedFailure { phase: "onboarding", errorClass: "invalid-nvidia-api-key" }, and records gateway-absent and sandbox-absent probes.. Framework and shell tests cover routing, secret-env behavior, expected-failure classification, and port-holder mechanics. Because this PR expands opt-in live scenarios that mutate real NemoClaw/OpenShell gateway, sandbox, and Docker state, targeted runtime validation would improve confidence in artifact contents and forbidden side effects.
**Runtime validation** — Validate ubuntu-no-docker-preflight-negative writes scenario-result.json expectedFailure { phase: "preflight", errorClass: "docker-missing" } and records gateway-absent and sandbox-absent probes.. Framework and shell tests cover routing, secret-env behavior, expected-failure classification, and port-holder mechanics. Because this PR expands opt-in live scenarios that mutate real NemoClaw/OpenShell gateway, sandbox, and Docker state, targeted runtime validation would improve confidence in artifact contents and forbidden side effects.
**Runtime validation** — Validate ubuntu-gateway-port-conflict-negative holds the selected gateway port during onboarding, releases it afterward, writes scenario-result.json expectedFailure { phase: "onboarding", errorClass: "gateway-port-conflict" }, and proves gateway and sandbox are absent.. Framework and shell tests cover routing, secret-env behavior, expected-failure classification, and port-holder mechanics. Because this PR expands opt-in live scenarios that mutate real NemoClaw/OpenShell gateway, sandbox, and Docker state, targeted runtime validation would improve confidence in artifact contents and forbidden side effects.
**Runtime validation** — Add a framework contract test that every default buildLiveScenarioMatrix() entry resolves through resolveExecutableOnboardingProfile() to a profile accepted by both OnboardingPhaseFixture.from() and the shell dispatcher profile set.. Framework and shell tests cover routing, secret-env behavior, expected-failure classification, and port-holder mechanics. Because this PR expands opt-in live scenarios that mutate real NemoClaw/OpenShell gateway, sandbox, and Docker state, targeted runtime validation would improve confidence in artifact contents and forbidden side effects.
**Runtime validation** — Validate ubuntu-repo-cloud-openclaw-resume, ubuntu-repo-cloud-openclaw-repair, ubuntu-repo-cloud-openclaw-double-same-provider, and ubuntu-repo-cloud-openclaw-custom-policies write the expected expectedStateId and probe IDs to scenario-result.json.. Framework and shell tests cover routing, secret-env behavior, expected-failure classification, and port-holder mechanics. Because this PR expands opt-in live scenarios that mutate real NemoClaw/OpenShell gateway, sandbox, and Docker state, targeted runtime validation would improve confidence in artifact contents and forbidden side effects.
**Acceptance clause:** Refs Adopt Vitest fixtures as the E2E scenario execution model #4941 — add test evidence or identify existing coverage. The PR body references this issue, but deterministic linkedIssues was empty and issue body/comment clauses were not available for literal extraction.
**Acceptance clause:** Refs Adopt phase fixtures + registry-driven test discovery for Vitest E2E scenarios #4990 — add test evidence or identify existing coverage. The PR body references this issue, but deterministic linkedIssues was empty and issue body/comment clauses were not available for literal extraction.
**Acceptance clause:** Refs Phase 2: Onboarding and Installer Audit Coverage (E2E audit-coverage) #4348 — add test evidence or identify existing coverage. The PR body references this issue, but deterministic linkedIssues was empty and issue body/comment clauses were not available for literal extraction.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

…ariant-fixtures' into codex/e2e-fanout-03-smoke-onboarding-scenarios

copy-pr-bot · 2026-06-09T20:09:44Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cv · 2026-06-10T08:40:10Z

Closing as superseded by #5106 and the post-#5098 one-E2E migration plan.

This branch belongs to the pre-cutover fanout stack. Any useful helper/scenario work should come back as a fresh, focused draft PR from current main: Vitest as the only E2E harness, GitHub Actions as the matrix, no revived runner path, no long-lived legacy-inventory.json roadmap expansion, and replacement/deletion evidence carried in the PR body plus linked issue.

chore(e2e): scaffold fan-out draft 03

a2abfad

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv self-assigned this Jun 9, 2026

cv added area: e2e End-to-end tests, nightly failures, or validation infrastructure area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow chore Build, CI, dependency, or tooling maintenance labels Jun 9, 2026

This was referenced Jun 9, 2026

test(e2e): wire onboarding variant fixtures #5053

Closed

Phase 2: Onboarding and Installer Audit Coverage (E2E audit-coverage) #4348

Closed

cv added 2 commits June 9, 2026 12:47

Merge remote-tracking branch 'origin/codex/e2e-fanout-02-onboarding-v…

fa51146

…ariant-fixtures' into codex/e2e-fanout-03-smoke-onboarding-scenarios

test(e2e): migrate smoke onboarding live scenarios

0538b31

cv added 7 commits June 9, 2026 13:17

test(e2e): route no-docker live onboarding

0a75c8c

test(e2e): share onboarding profile resolution

232a656

test(e2e): gate live support on executable onboarding profiles

a3dd970

test(e2e): accept live negative onboarding cleanup output

1fe2e9b

test(e2e): hold gateway port in shell bridge

d675951

test(e2e): harden gateway conflict shell bridge

99f73a4

test(e2e): align invalid key shell dispatch

2c149ca

cv mentioned this pull request Jun 9, 2026

test(e2e): retire covered smoke/onboarding shell entrypoints #5055

Closed

15 tasks

cv closed this Jun 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): migrate smoke and onboarding scenarios#5054

test(e2e): migrate smoke and onboarding scenarios#5054
cv wants to merge 10 commits into
codex/e2e-fanout-02-onboarding-variant-fixturesfrom
codex/e2e-fanout-03-smoke-onboarding-scenarios

cv commented Jun 9, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

Review skipped

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 9, 2026

Uh oh!

cv commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cv commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Uh oh!

copy-pr-bot Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

Uh oh!

copy-pr-bot Bot commented Jun 9, 2026

Uh oh!

cv commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cv commented Jun 9, 2026 •

edited

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading