fix(agent): regenerate proxy-env.sh guard chain during recovery (#2701)

[hunglp6d]

Fixes #5262

✨ [AI-generated issue]

Summary

When /tmp/nemoclaw-proxy-env.sh is missing at recovery time (e.g. after a pod recreate on DGX Spark), the recovery script now scans for preload .js files that still exist on disk and regenerates a minimal proxy-env.sh with the corresponding NODE_OPTIONS --require entries. Previously, the recovery script logged a WARNING and launched the gateway naked, which triggers the @homebridge/ciao crash loop on aarch64 / DGX Spark (#2701).

Related Issue

Changes

• src/lib/agent/runtime.ts (buildOpenClawRecoveryScript + buildRecoveryScript): Replace the warn-and-proceed branch when _PE_MISSING=1 with a regenerate-and-proceed branch that:
  1. Iterates over known preload script paths (/tmp/nemoclaw-*.js)
  2. Builds a NODE_OPTIONS string from whichever files are present
  3. Writes the regenerated /tmp/nemoclaw-proxy-env.sh with mode 444
  4. Sources it and re-checks the guard chain
  5. Falls through to the existing WARNING only if no preload scripts exist at all

Validation

Custom-e2e validation was not run — the available token lacks the workflow scope required to push .github/workflows/custom-e2e.yaml. Manual validation: run issue-2478-crash-loop-recovery-e2e on this branch.

• Original failing run: 27386272836 on a3ae21b6eed8399099fd390bd45ad43e78218258
• Targeted job: issue-2478-crash-loop-recovery-e2e / run (#80933852556)

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• npx prek run --all-files passes
• npm test passes
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes

AI Disclosure

• AI-assisted — tool: Claude Code (nemoclaw-diagnosis skill)

---

Signed-off-by: Hung Le hple@nvidia.com

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 064ebcf3-b9c5-4742-942b-2055eac341d4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/nightly-e2e-recovery-guard-chain-regen-a3ae21b

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

E2E Advisor Recommendation

Required E2E: gateway-guard-recovery
Optional E2E: issue-2478-crash-loop-recovery-e2e, sandbox-survival-e2e, hermes-e2e-vitest

Dispatch hint: gateway-guard-recovery

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• gateway-guard-recovery (medium): Direct live regression coverage for [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701: wipes /tmp guard-chain files, kills the gateway tree, triggers production connect recovery, and asserts the guard chain is restored before the gateway remains stable.

Optional E2E

• issue-2478-crash-loop-recovery-e2e (medium): Legacy bash E2E coverage for the related [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478 crash-loop recovery path; useful as a broader install/onboard/runtime recovery cross-check but largely overlaps the direct [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701 Vitest scenario.
• sandbox-survival-e2e (medium): Adjacent confidence for gateway restart/recovery after sandbox lifecycle disruption, but it is less targeted than gateway-guard-recovery.
• hermes-e2e-vitest (medium): Optional because buildRecoveryScript covers Hermes/non-OpenClaw agents and the diff changes shared recovery behavior; run if reviewers want confidence that Hermes agent runtime still launches after recovery changes.

New E2E recommendations

• None.

Dispatch hint

• Workflow: .github/workflows/e2e-vitest-scenarios.yaml
• jobs input: gateway-guard-recovery

[github-actions]

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: gateway-guard-recovery
Optional Vitest E2E scenarios: ubuntu-repo-docker-post-reboot-recovery

Dispatch required Vitest E2E scenarios:

• gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=gateway-guard-recovery

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

• gateway-guard-recovery: The PR changes gateway recovery guard-chain regeneration in src/lib/agent/runtime.ts, matching the wired free-standing Vitest job that exercises gateway recovery restoring /tmp/nemoclaw-proxy-env.sh and required preload guards after pod-recreate wipe ([DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701).
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=gateway-guard-recovery

Optional Vitest E2E scenarios

• ubuntu-repo-docker-post-reboot-recovery: Adjacent live-supported typed scenario that exercises OpenClaw lifecycle recovery through the registry-driven Vitest path; useful as broader coverage beyond the targeted guard-chain recovery job.
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-docker-post-reboot-recovery

Relevant changed files

• src/lib/agent/runtime.ts

[github-actions]

PR Review Advisor

Findings: 4 needs attention, 2 worth checking, 0 nice ideas
Top item: Regenerate guard chain from trusted packaged preloads, not unverified /tmp leftovers

Review findings

🛠️ Needs attention

• Recovery still launches naked when the linked pod-recreate scenario wipes all guard files (src/lib/agent/runtime.ts:208): The new branch only regenerates /tmp/nemoclaw-proxy-env.sh from preload scripts that still exist under /tmp. The linked [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701 repro explicitly says pod recreate creates a fresh container where /tmp/nemoclaw-proxy-env.sh and the guard JavaScript files are gone. In that case _REGEN_OPTS stays empty, the script logs a warning, and recovery continues to launch the gateway without the safety-net/ciao preloads, preserving the crash loop this PR is meant to fix.
  • Recommendation: Re-emit the guard chain from the durable packaged preload sources, such as /usr/local/lib/nemoclaw/preloads/, before launch. If trusted preloads cannot be installed, fail recovery instead of launching an unguarded gateway.
  • Evidence: The added loop only includes files matching fixed /tmp paths when `[ -f "$_f" ]`; if no files exist it logs `no preload scripts found in /tmp — launching without library guards ([DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478)`. Issue [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701 states fresh container `/tmp` is empty and both `/tmp/nemoclaw-proxy-env.sh` and `/tmp/nemoclaw-ciao-network-guard.js` are missing.
• Recovery trusts predictable /tmp preload files as gateway code without provenance checks (src/lib/agent/runtime.ts:208): The regenerated NODE_OPTIONS chain is built from predictable files under /tmp using only `[ -f ]`, which follows symlinks and does not verify root ownership, mode, non-writability, or trusted content. Because these files are loaded with `--require` by the relaunched gateway, a sandbox-controlled or replaced /tmp file can cross the trusted-code boundary.
  • Recommendation: Do not rebuild the gateway preload chain from unverified /tmp artifacts. Prefer copying from immutable packaged preload sources using the same trust-boundary installer as startup. If any /tmp files are reused, verify regular-file, non-symlink, owner, mode, and non-writability before adding them to NODE_OPTIONS.
  • Evidence: The new code appends `--require $_f` for each `[ -f "$_f" ]` path including `/tmp/nemoclaw-sandbox-safety-net.js` and `/tmp/nemoclaw-ciao-network-guard.js`. Nearby `scripts/lib/sandbox-init.sh` documents /tmp sourced files as a cross-user trust boundary, while the new recovery branch performs no comparable validation.
• proxy-env regeneration writes a sourced trust-boundary file with raw redirection (src/lib/agent/runtime.ts:208): The recovery path writes `/tmp/nemoclaw-proxy-env.sh` using `> /tmp/nemoclaw-proxy-env.sh && chmod 444`, which follows symlinks and is not atomic. That diverges from the repository's existing trust-boundary helpers and from the no-follow care already used in this file for gateway logs.
  • Recommendation: Write the regenerated proxy-env file through a safe temp-file plus rename/no-follow helper with the expected ownership and permissions, or invoke the shared sandbox sourced-file installer. Reject symlinked or otherwise unsafe destinations.
  • Evidence: The new branch uses `printf ... > /tmp/nemoclaw-proxy-env.sh && chmod 444 /tmp/nemoclaw-proxy-env.sh`. `scripts/lib/sandbox-init.sh` says files sourced across user boundaries must be root-owned 444 in root mode and are written via `emit_sandbox_sourced_file` to avoid rm/recreate and symlink races.
• Implementation does not follow the linked issue's source-of-truth recovery design (src/lib/agent/runtime.ts:208): Issue [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701's remaining scope calls for installing from `/usr/local/lib/nemoclaw/preloads/` into `/tmp` with correct ownership/mode and emitting proxy-env dynamically. This PR instead reconstructs proxy-env from whichever /tmp files happen to remain, so it handles only a narrower invalid state and leaves the real source boundary unresolved.
  • Recommendation: Move the recovery source of truth to the packaged preload modules or a shared install-preloads helper, and document why any fallback remains necessary. Add a removal condition if a temporary fallback is retained.
  • Evidence: Issue [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701 says `refactor(runtime): extract entrypoint preload modules #3109 ... baked into the image at /usr/local/lib/nemoclaw/preloads/` is the prerequisite for the proper fix and lists `Shared install-preloads.sh` as remaining scope. The diff does not reference `/usr/local/lib/nemoclaw/preloads/` or add an installer.

🔎 Worth checking

• Source-of-truth review needed: Gateway guard-chain recovery fallback in `buildOpenClawRecoveryScript` and `buildRecoveryScript`: The advisor marked localized patch analysis as missing.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: The diff regenerates from `/tmp/nemoclaw-*.js` when present and falls back to launching without guards when none are found; issue [DGX Spark] Host reboot bricks sandbox until 5-min rebuild --yes: connect recovery path warns about missing /tmp guards but launches gateway naked → @homebridge/ciao crash loop #2701 says fresh-container /tmp is empty and proposes reinstalling from `/usr/local/lib/nemoclaw/preloads/`.
• New recovery and trust-boundary behavior lacks targeted unit coverage (src/lib/agent/runtime.ts:208): No tests are added for the new regeneration path, especially negative cases around missing all guard files, symlinked proxy-env, symlinked or writable preload candidates, partial chains, and preserving trusted permissions. Existing tests cover [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478 warning/refusal shape, but not this new fallback.
  • Recommendation: Add deterministic unit tests for the recovery script shape and negative paths before relying on E2E coverage. Include security cases for symlink/tamper handling and correctness cases for all-files-wiped recovery.
  • Evidence: The PR changes only `src/lib/agent/runtime.ts`. Existing `test/e2e-scenario/live/gateway-guard-recovery.test.ts` models wiping proxy-env and all guard files, while this diff adds no corresponding unit coverage for the new branch.

🌱 Nice ideas

• None.

Consider writing more tests for

• **Runtime validation** — buildOpenClawRecoveryScript restores guard chain from packaged preload sources when proxy-env and all /tmp guard files are absent. The changed code is deterministic script generation and should have unit tests, but because it affects sandbox recovery and trusted-code boundaries, runtime validation of the all-/tmp-wiped recovery path is also important.
• **Runtime validation** — buildOpenClawRecoveryScript refuses or repairs rather than launching naked when no trusted preload scripts can be installed. The changed code is deterministic script generation and should have unit tests, but because it affects sandbox recovery and trusted-code boundaries, runtime validation of the all-/tmp-wiped recovery path is also important.
• **Runtime validation** — buildOpenClawRecoveryScript rejects symlinked /tmp/nemoclaw-proxy-env.sh during regeneration. The changed code is deterministic script generation and should have unit tests, but because it affects sandbox recovery and trusted-code boundaries, runtime validation of the all-/tmp-wiped recovery path is also important.
• **Runtime validation** — buildOpenClawRecoveryScript ignores symlinked or sandbox-writable /tmp/nemoclaw-*.js preload candidates. The changed code is deterministic script generation and should have unit tests, but because it affects sandbox recovery and trusted-code boundaries, runtime validation of the all-/tmp-wiped recovery path is also important.
• **Runtime validation** — buildRecoveryScript fails before launch when regenerated NODE_OPTIONS lacks the ciao preload. The changed code is deterministic script generation and should have unit tests, but because it affects sandbox recovery and trusted-code boundaries, runtime validation of the all-/tmp-wiped recovery path is also important.
• **New recovery and trust-boundary behavior lacks targeted unit coverage** — Add deterministic unit tests for the recovery script shape and negative paths before relying on E2E coverage. Include security cases for symlink/tamper handling and correctness cases for all-files-wiped recovery.
• **Acceptance clause:** Issue [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478: `@homebridge/ciao` calls `os.networkInterfaces()` during init; inside the OpenShell sandbox the syscall fails with EPERM and the unhandled rejection takes the gateway down. — add test evidence or identify existing coverage. The PR continues to require the `nemoclaw-ciao-network-guard` marker after sourcing/regeneration, but only restores that guard if `/tmp/nemoclaw-ciao-network-guard.js` already exists.
• **Acceptance clause:** Issue [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478: Workaround that worked: override `os.networkInterfaces` before ciao loads via a NODE_OPTIONS preload. — add test evidence or identify existing coverage. The diff regenerates a minimal `export NODE_OPTIONS="--require ..."` file from existing /tmp preload scripts, but it does not reinstall the preload from a durable source when /tmp is empty.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

[cv]

Closing this in favor of #5321. The replacement keeps the missing proxy-env.sh recovery path, but restores the guard chain from trusted packaged preloads, writes the recovered proxy env via staged file, validates exact --require entries, and fails closed if trusted staging is unavailable. Thanks for chasing the #2701 path.

[hunglp6d]

@cv Thanks, this is AI-generated PR, will close the associated issue.