test(e2e): migrate onboard inference smoke to Vitest by cv · Pull Request #5149 · NVIDIA/NemoClaw

cv · 2026-06-10T14:25:38Z

Summary

Migrates the onboard inference smoke regression from the legacy shell script into a Vitest live scenario. The scenario builds the CLI, runs a host-side setupInference() probe with the child process forced out of Vitest mode, and verifies runtime-broken configured routes are rejected by the real chat/completions smoke path.

Related Issue

Refs #4941
Refs #3253

Changes

Add test/e2e-scenario/live/onboard-inference-smoke.test.ts with the compatible-endpoint HTTP 503 regression probe.
Replace test/e2e/test-onboard-inference-smoke.sh in regression-e2e.yaml with direct Vitest execution and artifact upload.
Extend test/regression-e2e-workflow.test.ts to lock the direct Vitest workflow contract.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

copy-pr-bot · 2026-06-10T14:25:42Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-06-10T14:25:47Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c3e04bb8-8060-4935-9701-ed03e5d0b362

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/e2e-migrate-onboard-inference-smoke

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-10T14:27:55Z

E2E Advisor Recommendation

Required E2E: onboard-inference-smoke-e2e
Optional E2E: model-router-provider-routed-inference-e2e

Dispatch hint: onboard-inference-smoke-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/codex/e2e-simplify-migration-tracking
Head: HEAD
Confidence: high

Required E2E

onboard-inference-smoke-e2e (low): Directly modified regression job and test implementation. Run it to validate the new Vitest-based workflow invocation, live scenario gate, build/probe behavior, and artifact upload path still exercise the [Brev][Onboard] Onboard reports SUCCESS without exercising inference path — config bugs reach end users instead of being caught #3253 setupInference HTTP 503 failure guard.

Optional E2E

model-router-provider-routed-inference-e2e (medium): Adjacent inference-routing confidence check: validates provider-routed inference through Model Router, but product routing code was not changed so this is not merge-blocking for this test/workflow migration.

New E2E recommendations

None.

Dispatch hint

Workflow: regression-e2e.yaml
jobs input: onboard-inference-smoke-e2e

github-actions · 2026-06-10T14:27:56Z

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/codex/e2e-simplify-migration-tracking
Head: HEAD
Confidence: high

Required scenario E2E

e2e-scenarios-all: PR adds a new free-standing Vitest live scenario test under test/e2e-scenario/live/. It is not a trusted-main live-supported typed registry scenario ID that can be targeted safely, so run the scenario fan-out via the canonical Vitest scenario workflow.
- Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional scenario E2E

None.

Relevant changed files

test/e2e-scenario/live/onboard-inference-smoke.test.ts

github-actions · 2026-06-10T14:30:54Z

PR Review Advisor

Findings: 0 needs attention, 1 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 1 still applies, 0 new items found

Review findings

🛠️ Needs attention

None.

🔎 Worth checking

Add job-local read-only permissions for the migrated E2E lane (.github/workflows/regression-e2e.yaml:223): The migrated `onboard-inference-smoke-e2e` job runs repository code through `npm run build:cli` and Vitest, but it does not declare job-local permissions. Because the workflow has top-level `checks: write` and `pull-requests: write`, this lane inherits write scopes even though the changed job only needs to read the repository and upload artifacts. This keeps a wider trusted-code boundary than necessary.
- Recommendation: Add `permissions: { contents: read }` under `onboard-inference-smoke-e2e` unless this job specifically needs check or pull-request write access.
- Evidence: Workflow top-level permissions include `contents: read`, `checks: write`, and `pull-requests: write`; the changed job starts at `.github/workflows/regression-e2e.yaml:223` and has no job-local `permissions`, while neighboring `strict-tool-call-probe-e2e` declares `permissions: contents: read`.

🌱 Nice ideas

None.

Consider writing more tests for

**Runtime validation** — Dispatch `onboard-inference-smoke-e2e` and confirm the workflow invokes `test/e2e-scenario/live/onboard-inference-smoke.test.ts` rather than `test/e2e/test-onboard-inference-smoke.sh`.. Deterministic tests cover migration parity and workflow contract, but the changed surface includes a GitHub Actions lane with dependency install, Vitest project selection, child-process probe execution, fixture artifact upload, and workflow permissions.
**Runtime validation** — Confirm the Vitest probe fails if `setupInference()` resolves after the mocked `/chat/completions` HTTP 503.. Deterministic tests cover migration parity and workflow contract, but the changed surface includes a GitHub Actions lane with dependency install, Vitest project selection, child-process probe execution, fixture artifact upload, and workflow permissions.
**Runtime validation** — Confirm uploaded `onboard-inference-smoke-artifacts` include probe stdout/stderr/result files without exposing real secret values.. Deterministic tests cover migration parity and workflow contract, but the changed surface includes a GitHub Actions lane with dependency install, Vitest project selection, child-process probe execution, fixture artifact upload, and workflow permissions.
**Runtime validation** — Add a deterministic assertion that the child probe's `__CALLS__` includes a command containing `/chat/completions`, so an earlier unrelated rejection cannot satisfy the [Brev][Onboard] Onboard reports SUCCESS without exercising inference path — config bugs reach end users instead of being caught #3253 guard.. Deterministic tests cover migration parity and workflow contract, but the changed surface includes a GitHub Actions lane with dependency install, Vitest project selection, child-process probe execution, fixture artifact upload, and workflow permissions.

Since last review details

Current findings:

Add job-local read-only permissions for the migrated E2E lane (.github/workflows/regression-e2e.yaml:223): The migrated `onboard-inference-smoke-e2e` job runs repository code through `npm run build:cli` and Vitest, but it does not declare job-local permissions. Because the workflow has top-level `checks: write` and `pull-requests: write`, this lane inherits write scopes even though the changed job only needs to read the repository and upload artifacts. This keeps a wider trusted-code boundary than necessary.
- Recommendation: Add `permissions: { contents: read }` under `onboard-inference-smoke-e2e` unless this job specifically needs check or pull-request write access.
- Evidence: Workflow top-level permissions include `contents: read`, `checks: write`, and `pull-requests: write`; the changed job starts at `.github/workflows/regression-e2e.yaml:223` and has no job-local `permissions`, while neighboring `strict-tool-call-probe-e2e` declares `permissions: contents: read`.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

wscurran · 2026-06-10T14:56:07Z

✨
Related open issues:

#4941 Adopt Vitest fixtures as the E2E scenario execution model
#3253 [Brev][Onboard] Onboard reports SUCCESS without exercising inference path — config bugs reach end users instead of being caught

copy-pr-bot · 2026-06-10T16:22:46Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

jyaunches · 2026-06-11T00:18:52Z

Closing as superseded by merged PR #5155, which retired test-onboard-inference-smoke.sh under the simplified #5098 migration rules.

cv added 9 commits June 10, 2026 01:40

test(e2e): simplify legacy migration tracking

b30dba2

test(e2e): retire orphaned assertion inventory

e246d20

docs(e2e): clarify Vitest fixture layer terminology

0fbf595

docs(e2e): align scenario advisor wording with Vitest

68e3157

test(e2e): rename migration policy checks

2d8ac4e

test(e2e): rename Vitest support project

d473ba6

docs(e2e): avoid runner wording in release notes

56c7071

test(e2e): rename Vitest support tests directory

0d1c4cb

test(e2e): migrate onboard inference smoke to Vitest

03a0079

cv self-assigned this Jun 10, 2026

cv mentioned this pull request Jun 10, 2026

test(e2e): add Vitest live coverage for onboard resume #5147

Merged

4 tasks

This was referenced Jun 10, 2026

Epic: Migrate legacy bash E2E into the Vitest E2E system #5098

Open

test(e2e): add OpenClaw TUI chat-correlation coverage slice #5150

Merged

test(e2e): guard legacy shell deletion evidence

f9422bc

cv added 2 commits June 10, 2026 08:00

test(e2e): validate legacy deletion replacement paths

debe958

chore(e2e): address foundation review hygiene

b1d33d7

cv mentioned this pull request Jun 10, 2026

test(e2e): simplify legacy migration tracking #5126

Merged

12 tasks

cv added 3 commits June 10, 2026 08:35

test(e2e): block retired ledger reintroduction

6062f53

test(release): disable signing in tag fixtures

c2113a8

test(e2e): rename scenario fixtures surface

3c1ffcc

cv added 3 commits June 10, 2026 09:19

test(e2e): migrate onboard inference smoke to Vitest

172f212

test(e2e): refresh fixture imports

aa83644

merge(e2e): absorb remote draft branch

0bd6057

cv mentioned this pull request Jun 10, 2026

test(e2e): retire onboard inference smoke script #5155

Merged

3 tasks

Base automatically changed from codex/e2e-simplify-migration-tracking to main June 10, 2026 20:53

jyaunches closed this Jun 11, 2026

Conversation

cv commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Uh oh!

copy-pr-bot Bot commented Jun 10, 2026

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

wscurran commented Jun 10, 2026

Uh oh!

copy-pr-bot Bot commented Jun 10, 2026

Uh oh!

jyaunches commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cv commented Jun 10, 2026 •

edited

Loading

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading