Skip to content

test(e2e): migrate test-issue-4434-tui-unreachable-inference.sh to vitest#5233

Merged
jyaunches merged 7 commits into
NVIDIA:mainfrom
jyaunches:e2e-migrate/test-issue-4434-tui-unreachable-inference
Jun 12, 2026
Merged

test(e2e): migrate test-issue-4434-tui-unreachable-inference.sh to vitest#5233
jyaunches merged 7 commits into
NVIDIA:mainfrom
jyaunches:e2e-migrate/test-issue-4434-tui-unreachable-inference

Conversation

@jyaunches

@jyaunches jyaunches commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrate test/e2e/test-issue-4434-tui-unreachable-inference.sh with focused equivalent Vitest live coverage.

Related Issues

Refs #5098
Refs #4434

Contract mapping

  • Legacy assertion: when the NVIDIA endpoint is blocked from the sandbox, openclaw tui must show a visible inference error and stop the active connected spinner.
    • Replacement: test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
    • Boundary preserved: real cloud OpenClaw sandbox, host sudo iptables DOCKER-USER DROP rules, openshell sandbox exec --tty, openclaw tui, terminal capture parsing.
  • Legacy assertion: endpoint is reachable before the firewall block and unreachable after the block from inside the sandbox.
    • Replacement: pre/post sandbox curl probes in the Vitest test.
    • Boundary preserved: real sandbox network path to https://integrate.api.nvidia.com/v1/models.

Simplicity check

  • Test shape: simple live Vitest test
  • Original runner/lane: nightly-e2e.yaml issue-4434-tui-unreachable-inference-e2e on ubuntu-latest with Docker/OpenShell, expect, iptables, passwordless sudo, and NVIDIA_API_KEY
  • Replacement runner: same runner class via .github/workflows/e2e-vitest-scenarios.yaml issue-4434-tui-unreachable-inference-vitest on ubuntu-latest
  • New shared helpers: none; parsing and shell snippets remain local to the test
  • New framework/registry/ledger: none
  • Workflow changes: add selective Vitest job; legacy shell deletion/workflow retirement deferred to Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 Phase 11
  • Selective dispatch: gh workflow run e2e-vitest-scenarios.yaml --repo NVIDIA/NemoClaw --ref e2e-migrate/test-issue-4434-tui-unreachable-inference -f jobs=issue-4434-tui-unreachable-inference-vitest -f pr_number=5233

Verification

  • npm run build:cli
  • npx biome check test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
  • npx tsc --noEmit --pretty false --allowImportingTsExtensions --moduleResolution bundler --module ESNext --target ES2022 --types vitest,node test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
  • NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts --silent=false --reporter=default (local gated import/skip)
  • npm run typecheck:cli -- --pretty false
  • git diff --check

Live proof note

Full live execution requires Linux Docker, iptables, passwordless sudo, expect, and NVIDIA_API_KEY; this macOS host lacks Docker/iptables/passwordless sudo, so CI or a suitable Linux runner must run the privileged proof.

Summary by CodeRabbit

  • Tests

    • Added an opt-in live end-to-end scenario that runs in a real sandbox, captures TUI output, validates an inference-unreachable behavior, performs host prerequisite checks, and ensures firewalling and cleanup are handled.
  • Chores

    • CI updated to include the new free-standing test job, aggregate its results into PR reports, and upload related artifacts for investigation.

Latest selective run

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ea383109-da8b-481f-8dbd-886d40fc241f

📥 Commits

Reviewing files that changed from the base of the PR and between bacced0 and 6037981.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (2)
  • tools/e2e-scenarios/workflow-boundary.mts
  • .github/workflows/e2e-vitest-scenarios.yaml

📝 Walkthrough

Walkthrough

Adds an opt-in live Vitest E2E scenario reproducing issue #4434 by blocking NVIDIA endpoints with host iptables, capturing TUI output via an expect script, asserting error/connection semantics, and wiring the new test into the Vitest workflow and its boundary validator and PR reporting.

Changes

Issue #4434 TUI Unreachable Inference Live Test

Layer / File(s) Summary
CI job wiring and job definition
.github/workflows/e2e-vitest-scenarios.yaml
Adds issue-4434-tui-unreachable-inference-vitest to validation allowlists and matrix generation; defines the free-standing job with optional Docker Hub login retries and anonymous fallback, host deps, CLI build, OpenShell install and verification, targeted Vitest run for the live test file, and artifact upload; includes the job in report-to-pr needs.
Workflow boundary validator updates
tools/e2e-scenarios/workflow-boundary.mts
Maps the new scenario id to its Vitest job, validates the job selector wiring (validateFreeStandingJobSelector) and ensures report-to-pr waits for the new job.
Test constants, configs, and helpers
test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
Adds SPDX/header, gating opt-in, sandbox/name constants and validation, blocked IP list and timeouts, regexes for visible errors/connected-spinner/status, and utility helpers for quoting, reading bundled OpenClaw version, ANSI stripping, and TUI analysis.
Firewall cleanup and expect script generation
test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
Generates a shell script to remove inserted DOCKER-USER DROP rules and an expect script that runs openclaw tui, matches expected visible error patterns, and encodes timeout/eof exit behavior.
Live test flow: sandbox provisioning, blocking, capture, and assertions
test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts
Implements the main opt-in test: env/platform validation, scenario metadata, host prerequisites, sandbox provisioning, bundled openclaw --version validation, pre-block probes, insertion/removal of iptables rules with tracked cleanup, confirmation of endpoint unreachable, running expect with capture and redaction, parsing TUI output, writing artifacts, and asserting final error semantics.

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant WorkflowJob as issue-4434-tui-unreachable-inference-vitest
  participant Runner
  participant OpenShell
  participant Sandbox
  participant ExpectRunner

  GitHubActions->>WorkflowJob: select via inputs.jobs / free_standing_scenarios
  WorkflowJob->>Runner: checkout, setup, optional docker login (retries/anonymous)
  Runner->>OpenShell: install openshell, verify openshell --version
  Runner->>Sandbox: provision sandbox, run openshell sandbox exec --tty
  Sandbox->>ExpectRunner: run expect script (openclaw tui) and capture logs
  ExpectRunner->>Runner: upload artifacts (expect.log, scenario.json, captures)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5243: Related Vitest job-selector and boundary-check plumbing touched by similar workflow changes.
  • NVIDIA/NemoClaw#5234: Related wiring for free-standing live Vitest scenarios that the scenario-advisor inspects.
  • NVIDIA/NemoClaw#5226: Adds a different free-standing Vitest job and updates workflow/boundary validation similarly.

Suggested labels

area: e2e, area: ci, area: inference

Suggested reviewers

  • cv
  • prekshivyas

Poem

🐰 I hop where packets fall and tunnels go awry,
I scrawl an Expect script beneath the sandbox sky,
Firewall drops hush the cloud; the TUI tells its tale,
Logs and artifacts collected, redactions without fail,
A rabbit's little test to chase a stubborn trail.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: migrating a legacy shell-based test (test-issue-4434-tui-unreachable-inference.sh) to Vitest, which is the primary objective documented in the PR objectives.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts Fixed
@jyaunches

Copy link
Copy Markdown
Contributor Author

Temporarily closing to make room for the maintainer PR-limit exemption fix; will reopen after that lands.

@jyaunches jyaunches closed this Jun 11, 2026
@jyaunches jyaunches reopened this Jun 11, 2026
@jyaunches jyaunches changed the title test(e2e): add issue 4434 Vitest guard test(e2e): P2 independent 3 migrate test-issue-4434-tui-unreachable-inference.sh to vitest Jun 11, 2026
@jyaunches jyaunches changed the title test(e2e): P2 independent 3 migrate test-issue-4434-tui-unreachable-inference.sh to vitest test(e2e): add issue 4434 Vitest guard Jun 11, 2026
@jyaunches jyaunches changed the title test(e2e): add issue 4434 Vitest guard test(e2e): add issue 4434 Vitest guard [IND-3] Jun 11, 2026
@jyaunches jyaunches force-pushed the e2e-migrate/test-issue-4434-tui-unreachable-inference branch from d4740b0 to 21c5e2c Compare June 11, 2026 15:25
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27357868104
Workflow ref: e2e-migrate/test-issue-4434-tui-unreachable-inference
Requested scenarios: issue-4434-tui-unreachable-inference
Summary: 3 passed, 2 failed, 2 skipped

Job Result
gateway-guard-recovery ❌ failure
generate-matrix ✅ success
issue-4434-tui-unreachable-inference-vitest ❌ failure
live-scenarios ⏭️ skipped
onboard-negative-paths-vitest ✅ success
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ✅ success

Failed jobs: gateway-guard-recovery, issue-4434-tui-unreachable-inference-vitest. Check run artifacts for logs.

@cv cv added the v0.0.64 Release target label Jun 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/workflows/e2e-vitest-scenarios.yaml (1)

398-407: ⚡ Quick win

Move or drop the duplicated recovery-scenario header.

This banner describes gateway-guard-recovery, but it now sits above openclaw-tui-chat-correlation-vitest and then repeats again at Line 482 where it actually applies. Keeping it here makes the section boundaries misleading.

♻️ Proposed cleanup
-  # ── Free-standing recovery scenarios (`#2701`) ─────────────────────────
-  # Recovery / disruption scenarios don't fit the steady-state expected-state
-  # registry that drives `live-scenarios` above. They run as free-standing
-  # Vitest test files using the same `e2e-scenarios-live` project, framework
-  # fixtures, and live-project gate — just outside the matrix.
-  #
-  # First failing-test-first guard for `#2701` (gateway recovery does not
-  # restore the /tmp guard chain after pod recreate). Will fail on `main`
-  # until the `#2701` fix lands; flips green afterwards.
-
   # Focused coverage slice for the `#2603/`#3145 OpenClaw websocket
   # protocol/history contract. The retained legacy bash lane remains the
   # source for full closeout until a later PR proves replacement and deletes it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 398 - 407, Remove
the duplicated "Free-standing recovery scenarios" / gateway recovery banner that
is currently sitting above the openclaw-tui-chat-correlation-vitest entry and
either move it so it immediately precedes the actual gateway-guard-recovery
scenario block (the banner that currently appears again later) or drop the
earlier copy entirely; locate the duplicate by searching for the banner text and
the scenario names "openclaw-tui-chat-correlation-vitest" and
"gateway-guard-recovery" and ensure only the single correct header remains
immediately before the gateway-guard-recovery scenario.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts`:
- Around line 30-33: TUI_TIMEOUT_SEC may be NaN or non-positive; validate and
clamp it right after parsing by converting to an integer, falling back to the
default (180) if parseInt returns NaN or value <= 0, and optionally enforce an
upper bound (e.g., max 3600); update uses of TUI_TIMEOUT_SEC (e.g., where
timeoutMs or the expect that checks env timeout) to rely on this sanitized value
so timeoutMs = TUI_TIMEOUT_SEC * 1000 is always a valid positive number.

---

Nitpick comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 398-407: Remove the duplicated "Free-standing recovery scenarios"
/ gateway recovery banner that is currently sitting above the
openclaw-tui-chat-correlation-vitest entry and either move it so it immediately
precedes the actual gateway-guard-recovery scenario block (the banner that
currently appears again later) or drop the earlier copy entirely; locate the
duplicate by searching for the banner text and the scenario names
"openclaw-tui-chat-correlation-vitest" and "gateway-guard-recovery" and ensure
only the single correct header remains immediately before the
gateway-guard-recovery scenario.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bf4d18a6-44a1-4c81-93e6-767ba23cc0be

📥 Commits

Reviewing files that changed from the base of the PR and between c160879 and 6f0a0ce.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts

Comment thread test/e2e-scenario/live/issue-4434-tui-unreachable-inference.test.ts Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Selective E2E Results — ❌ Some jobs failed

Run: 27377278793
Target ref: e2e-migrate/test-issue-4434-tui-unreachable-inference
Workflow ref: main
Requested jobs: issue-4434-tui-unreachable-inference-e2e
Summary: 0 passed, 1 failed, 0 cancelled, 0 skipped

Job Result
issue-4434-tui-unreachable-inference-e2e ❌ failure

Failed jobs: issue-4434-tui-unreachable-inference-e2e. Check run artifacts for logs.

…e-4434-tui-unreachable-inference

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
@jyaunches jyaunches changed the title test(e2e): add issue 4434 Vitest guard [IND-3] test(e2e): migrate test-issue-4434-tui-unreachable-inference.sh to vitest Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27381862992
Workflow ref: e2e-migrate/test-issue-4434-tui-unreachable-inference
Requested scenarios: (default — all supported)
Requested jobs: issue-4434-tui-unreachable-inference-vitest
Summary: 3 passed, 0 failed, 11 skipped

Job Result
credential-migration-vitest ⏭️ skipped
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
issue-4434-tui-unreachable-inference-vitest ✅ success
launchable-smoke-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ⏭️ skipped
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

…e-4434-tui-unreachable-inference

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
@jyaunches jyaunches enabled auto-merge (squash) June 12, 2026 00:19
@jyaunches jyaunches merged commit 0ebd79f into NVIDIA:main Jun 12, 2026
34 checks passed
@wscurran wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance labels Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance v0.0.64 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants