fix(installer): Detect early cloud-init completion

[ryanheffernan]

Description

Fix false Air simulation setup failures caused by two issues in the cloud-init watcher:

1. The remote setup can finish before the local watcher observes the completion banner in its live tail -f stream. When cloud-init status becomes done, the watcher previously treated the missing streamed banner as failure even when /var/log/nvcm-setup.log showed that setup completed successfully.
2. Stopping the SSH log-tail process can exceed the five-second grace period. The resulting subprocess.TimeoutExpired exception then masks the setup result and reports the overall run as failed.

This change:

• checks both /var/log/cloud-init-output.log and /var/log/nvcm-setup.log for an existing completion marker before starting the live tail;
• rechecks the remote logs after cloud-init reports done before declaring setup failure;
• moves the completion banner to the end of the setup script, after cluster status and next-step output, so it represents full script completion;
• centralizes log-tail cleanup, escalating from terminate() to kill() when the process does not exit within five seconds;
• prevents cleanup timeouts from overwriting an already-determined success or failure result.

Validation

• Kind integration was not run because this only changes the Air installer watcher and its child-process cleanup; no cluster manifests or runtime components changed.

Checklist

• I am familiar with the contributing guidelines in CONTRIBUTING.md.
• Commits are signed off for DCO compliance.
• New or existing tests cover these changes.
• No documentation update is required for this false-failure correction.
• No generated artifacts are affected.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The installer reorders cloud-init output to show cluster status and a deploy command before the completion banner. It also checks remote logs for the setup marker before and during cloud-init waiting, with shared tail-process cleanup and tests for these paths.

Changes

Cloud-init setup completion

Layer / File(s)	Summary
Setup script ordering installer/src/nv_config_manager_installer/air_sim/cloud_init.py, installer/tests/air_sim/test_cloud_init.py	The generated cloud-init script prints cluster status and a next-step deploy command before the completion banner, and the test verifies that ordering.
Marker detection and process cleanup installer/src/nv_config_manager_installer/air_sim/sim_manager.py, installer/tests/air_sim/test_sim_manager.py	The simulator probes configured remote logs for the setup marker and consistently terminates or kills the cloud-init tail process; tests cover command construction, timeouts, and fallback cleanup.
Cloud-init wait flow installer/src/nv_config_manager_installer/air_sim/sim_manager.py	wait_for_cloud_init() checks for an existing marker after SSH becomes reachable, rechecks after cloud-init reports done, and uses shared cleanup across exit paths.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AirSimulationManager
  participant RemoteServer
  participant CloudInitTailProcess

  AirSimulationManager->>RemoteServer: Check setup marker in configured logs
  RemoteServer-->>AirSimulationManager: Marker found or not found
  alt marker found
    AirSimulationManager-->>AirSimulationManager: Return success
  else marker not found
    AirSimulationManager->>CloudInitTailProcess: Tail cloud-init output
    RemoteServer-->>AirSimulationManager: Report cloud-init done
    AirSimulationManager->>RemoteServer: Recheck setup marker
    RemoteServer-->>AirSimulationManager: Marker found or not found
    AirSimulationManager->>CloudInitTailProcess: Terminate or kill tail process
    AirSimulationManager-->>AirSimulationManager: Return success or failure
  end

Loading

Suggested reviewers: dmathur0, jojung1, polarweasel, spidercensus, susanhooks

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: handling early cloud-init completion in the installer.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-air-sim-setup-watcher

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@installer/src/nv_config_manager_installer/air_sim/sim_manager.py`:
- Around line 771-785: The _remote_setup_marker_exists method is building the
grep call as separate ssh arguments, which causes the remote shell to
re-tokenize the setup banner and miss existing markers. Update the ssh
invocation in _remote_setup_marker_exists so sudo grep is passed as one remote
command string, preserving the exact _SETUP_COMPLETE_MARKER and _SETUP_LOG_PATHS
handling. Keep the fix localized to the _remote_setup_marker_exists path so the
shortcut checks work before falling back to the tail/status flow.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a42922c4-508e-4926-8f49-0d88b8735017

📥 Commits

Reviewing files that changed from the base of the PR and between 3ec74a6 and 0cbb43d.

📒 Files selected for processing (1)

• installer/src/nv_config_manager_installer/air_sim/sim_manager.py

[ryanheffernan]

/ok to test 77c0f4b