machine-a-tron: simulated NVOS switches never reach `ready` in sim (blocks hardware-free rack testing)

[mehrdadmoradi-crusoe]

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement / gap report — directly in support of #3290 (hardware-free NICo scale testing).

Summary

On a clean, hardware-free deployment (kind + machine-a-tron + bmc-mock), a simulated NVIDIA NVOS switch (hw_type = "nvidia_switch_nd5200_ld") is discovered and created, but then stalls permanently in Initializing { WaitForOsMachineInterface } and never reaches ready. Because a rack cannot leave Discovering until all its switches are ready, this also blocks any attempt to drive a rack lifecycle in simulation — which is a prerequisite for the hardware-free scale testing described in #3290.

The node fleet (hosts + DPUs) converges to ready cleanly in the same run; only switches are affected.

Environment

• NICo built from main (verified against main at time of filing).
• kind on a single-node local cluster, bootstrap-prereqs.sh + devspace deploy, no physical hardware.
• Stock machine-a-tron config plus a switches stanza (see repro).

Reproduction

1. Deploy NICo locally (bootstrap-prereqs.sh → devspace deploy -n nico-system) and seed factory credentials.
2. Add a switch section to machine-a-tron config (the default main config has none):

   [machines.switches]
   hw_type = "nvidia_switch_nd5200_ld"
   host_count = 2
   dpu_per_host_count = 0

3. Let discovery run, then:

   nico-admin-cli switch show
   

Observed: both switches sit at
{"state":"initializing","initializing_state":"WaitForOsMachineInterface"} indefinitely.
nico-admin-cli expected-switch show shows empty NVOS Username/Password.
The node fleet in the same deployment reaches ready normally.

Root cause (switch path)

machine-a-tron emulates only the BMC/Redfish side of a switch; it never provides anything for the NVOS (OS/data-plane) side:

• add_expected_switch registers every switch with nvos_username: None, nvos_password: None, nvos_ip_address: None — crates/machine-a-tron/src/api_client.rs:555-575 (fields at ~570-573).
• A switch is treated as BMC-only, so machine-a-tron never simulates the NVOS port booting / doing DHCP (no machine/data-plane DHCP is ever emitted for the nvos_mac).

That leaves two mutually-reinforcing dead ends:

1. No NVOS machine_interface row is ever created, so the switch controller waits forever:

   • WaitForOsMachineInterface requires a machine_interface row whose MAC is one of the switch's nvos_mac_addresses — crates/switch-controller/src/initializing.rs (dispatch ~:43, lookup find_by_mac_address ~:105, associated_count gate); stays in wait(...) while the count is 0.
   • The only ways such a row gets created both require the NVOS side to exist:
     • site-explorer static pre-allocation is gated on nvos_ip_address being Some — crates/site-explorer/src/lib.rs:1977 → try_preallocate_one(... InterfaceType::Data ...) (~:1980); and
     • the DHCP path needs the NVOS port to actually DHCP, which the mock never does.

2. The switch_nvos/<bmc_mac>/admin Vault secret is never seeded, so later reads 404:

   • RotateOsPassword only writes the secret when nvos_username/nvos_password are present; with both None it logs "…no NVOS credentials…skipping" and writes nothing — crates/switch-controller/src/configuring.rs:143-161.
   • (The 404 read itself is handled gracefully in carbide_secrets; it is a symptom, not an unhandled bug.)

Downstream impact on racks (code analysis — not yet observed in sim)

Because switches never reach ready, a rack cannot advance past discovery:

• Discovering → Maintenance requires all switches to be controller_state = "ready" — crates/rack-controller/src/discovering.rs:37,72-76.

So a full rack lifecycle cannot currently be exercised in pure simulation. Separately (and only reachable once switches are fixed), the NVLink/NMX-C step is a hard dependency on a real external service and has no runtime-usable simulator:

• NMX-C is driven via NVIDIA RMS (librms, external gRPC crate — Cargo.toml:30), and the RMS client is only constructed when rms.api_url is set — crates/api-core/src/setup.rs:354. With no RMS configured, ConfigureNmxCluster errors the rack out.
• The only MockRmsApi lives in crates/api-test-helper/src/mock_rms.rs and is used solely from test modules — it is not selectable by the production binary.

We have not observed the NMX-C wall live (the switch stall blocks reaching it); the above is from reading main. Flagging it here because it is the next blocker for #3290's rack goal, not to assert it as tested.

Proposed directions (need maintainer/hardware ground truth)

The fix is in the simulator, and the right shape depends on how real NVOS switches actually onboard — which we don't have hardware to confirm. Two candidate directions:

1. Static NVOS registration — have machine-a-tron assign an NVOS management IP and dummy NVOS credentials on add_expected_switch (single nvos_mac), so site-explorer pre-allocates the NVOS interface and RotateOsPassword seeds the Vault secret. Smaller change; requires an IP-assignment scheme machine-a-tron doesn't have today.
2. Simulate the NVOS port — teach machine-a-tron to emulate the NVOS side booting and doing DHCP (the "realistic" path). Larger change to the switch state machine (switches are currently BMC-only).

Which of these matches how a real nd5200/NVOS switch is expected to come up (DHCP vs static operator assignment for the NVOS management interface, and what seeds switch_nvos/<mac>/admin)?

Offer

Happy to implement whichever direction you prefer and validate it end-to-end in the kind + machine-a-tron sim (switch reaching ready, then a rack advancing to Maintenance). Holding on a PR until the intended real-hardware behavior is confirmed, so the mock reflects reality rather than just passing the state machine.