Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions docs/development/new_hardware_support.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ target/debug/bmc-explorer-cli \
<bmc-ip>
```

`--boot-mac` is the MAC address of the interface from which the host is expected to boot. For the common managed-DPU configuration, use the host-facing `pf0` MAC of the primary DPU; for an integrated-NIC configuration, use the NIC selected as the primary boot interface. Use the same MAC for exploration, machine setup, setup status, and boot-order tests. Omit it only when the platform does not require a selected boot interface. See [Boot Interfaces and DPU Modes](../provisioning/boot-interfaces-and-dpu-modes.md) for how NICo selects and persists this value.
`--boot-mac` is the MAC address of the interface from which the host is expected to boot. For the common managed-DPU configuration, use the host-facing `pf0` MAC of the primary DPU; for an integrated-NIC configuration, use the NIC selected as the primary boot interface. Use the same MAC for exploration, machine setup, setup status, and boot-order tests. Omit it only when the platform does not require a selected boot interface. See [Boot Interfaces and DPU Policies](../provisioning/boot-interfaces-and-dpu-modes.md) for how NICo selects and persists this value.

`--bmc-port` is the (optional) port on which the BMC listens. HTTPS port 443 is the default; use `--bmc-port <port>` for a BMC listening elsewhere.

Expand Down Expand Up @@ -386,4 +386,3 @@ To add one:
1. Wire the variant through `crates/bmc-mock/src/machine_info.rs`: DPU count and type, vendor and product identity, Redfish version, manager, system, chassis, discovery, firmware inventory, and OEM behavior should match the live BMC responses relevant to the test.

1. Add focused tests for the behavior the hardware contribution changes, such as vendor detection, inventory, NIC-mode detection, or provisioning.

10 changes: 9 additions & 1 deletion docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ In current deployments, the DPU is a [NVIDIA BlueField-2 or BlueField-3](https:/

The NVIDIA DPU family used by NICo for tenant isolation and site management. A BlueField card has its own ARM complex, BMC, NIC firmware, and OS image. NICo provisions and manages the BlueField side of each ManagedHost before making the Host available to tenants.

### Host DPU Policy

The operator's desired treatment of DPU hardware on a host. `HostDpuPolicy` has three values: `Manage`, which at the site or resolved level lets NICo provision and attach the DPUs; `UseAsNic`, which asks NICo to run physically present DPU hardware as plain NICs; and `Ignore`, which tells NICo not to configure or attach DPU hardware. For backward compatibility, a per-host `Manage` declaration inherits the site policy rather than overriding it. This policy is intent, not a hardware observation.

### BlueField Operating Mode

The mode a BlueField card reports that it is currently running in. `BlueFieldOperatingMode` is either `Dpu` or `Nic`. NICo compares this observed state with the host DPU policy when deciding whether a card needs reconfiguration. Existing protobuf and Redfish boundaries retain the legacy `NicMode` name for compatibility; model code uses `BlueFieldOperatingMode` so observed state is not confused with desired policy.

## REST API Services and Binaries

### API Server
Expand Down Expand Up @@ -148,7 +156,7 @@ A VPC virtualization type for tenant instances on zero-DPU hosts. A Flat VPC is

### Zero-DPU Host

A managed host that NICo operates without a NICo-managed DPU — either a host with no DPU hardware (`no_dpu`) or a host whose BlueField DPU is run as a plain NIC (`nic_mode`). NICo does not build an overlay for a zero-DPU host; its tenant instances attach directly to HostInband underlay segments and belong to Flat VPCs. DPU mode is set site-wide or per host in the API server configuration.
A managed host that NICo operates without a NICo-managed DPU — either a host whose DPU policy is `ignore`, or a host whose BlueField DPU is run as a plain NIC through `use_as_nic`. NICo does not build an overlay for a zero-DPU host; its tenant instances attach directly to HostInband underlay segments and belong to Flat VPCs. DPU policy is set site-wide in the API server configuration or per host on its Expected Machine.

### HBN in NICo

Expand Down
2 changes: 1 addition & 1 deletion docs/index.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ navigation:
path: provisioning/ingesting-hosts.md
- page: Ingesting Hosts (REST API)
path: provisioning/ingesting-hosts-rest-api.md
- page: Boot Interfaces and DPU Modes
- page: Boot Interfaces and DPU Policies
path: provisioning/boot-interfaces-and-dpu-modes.md
- page: Machine Validation
path: provisioning/machine-validation.md
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ nico-admin-cli-expected-machine-add - Add expected machine
\[**--host_nics**\] \[**--rack_id**\]
\[**--default_pause_ingestion_and_poweron**\] \[**--dpf-enabled**\]
\[**--extended**\] \[**--bmc-ip-address**\]
\[**--bmc-retain-credentials**\] \[**--dpu-mode**\]
\[**--bmc-retain-credentials**\] \[**--dpu-policy**\]
\[**--disable-lockdown**\] \[**--sort-by**\] \[**-h**\|**--help**\]

## DESCRIPTION
Expand Down Expand Up @@ -114,23 +114,21 @@ factory-default credentials in Vault as-is\

- false

**--dpu-mode** *\<DPU_MODE\>*
Per-host DPU operating mode. \`dpu-mode\` (default): DPUs are managed by
NICo; \`nic-mode\`: DPU hardware present but treated as a plain NIC;
\`no-dpu\`: no DPU hardware at all. Unset defers to the site-wide
\`\[site_explorer\] dpu_mode\` setting (which itself falls back to
\`dpu-mode\` when not set).\
**--dpu-policy** *\<DPU_POLICY\>*\
Per-host DPU policy. \`manage\` (default): inherit the site policy,
which defaults to managing DPUs; \`use-as-nic\`: configure DPU hardware
as plain NICs; \`ignore\`: do not configure or attach DPU hardware.
Unset defers to the site-wide \`\[site_explorer\] dpu_policy\` setting.
The legacy \`--dpu-mode\` flag and values remain accepted.\

\
*Possible values:*

- unspecified
- manage

- dpu-mode
- use-as-nic

- nic-mode

- no-dpu
- ignore

**--disable-lockdown** *\<DISABLE_LOCKDOWN\>*
If true, do not lock down the server as part of lifecycle management
Expand Down Expand Up @@ -163,7 +161,7 @@ Print help (see a summary with -h)
nico-admin-cli expected-machine add --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mypassword --chassis-serial-number sample_serial-1
nico-admin-cli expected-machine add --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mypassword --chassis-serial-number sample_serial-1 --meta-name MyMachine --label DATACENTER:XYZ --sku-id DGX-H100-640GB
nico-admin-cli expected-machine add --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mypassword --chassis-serial-number sample_serial-1 --bmc-ip-address 192.0.2.20
nico-admin-cli expected-machine add --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mypassword --chassis-serial-number sample_serial-1 --dpu-mode nic-mode
nico-admin-cli expected-machine add --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mypassword --chassis-serial-number sample_serial-1 --dpu-policy use-as-nic
```

---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ update, preserves unprovided fields).
\[**--meta-description**\] \[**--label**\] \[**--sku-id**\]
\[**--rack-id**\] \[**--default_pause_ingestion_and_poweron**\]
\[**--dpf-enabled**\] \[**--bmc-ip-address**\] \[**--extended**\]
\[**--bmc-retain-credentials**\] \[**--dpu-mode**\]
\[**--bmc-retain-credentials**\] \[**--dpu-policy**\]
\[**--disable-lockdown**\] \[**--sort-by**\] \[**-h**\|**--help**\]

## DESCRIPTION
Expand Down Expand Up @@ -120,22 +120,21 @@ factory-default credentials in Vault as-is\

- false

**--dpu-mode** *\<DPU_MODE\>*
Per-host DPU operating mode. \`dpu-mode\` (default): DPUs are managed by
NICo; \`nic-mode\`: DPU hardware present but treated as a plain NIC;
\`no-dpu\`: no DPU hardware at all. Unset preserves the existing
per-host value.\
**--dpu-policy** *\<DPU_POLICY\>*\
Per-host DPU policy. \`manage\`: inherit the site policy, which defaults
to managing DPUs; \`use-as-nic\`: configure DPU hardware as plain NICs;
\`ignore\`: do not configure or attach DPU hardware. Unset preserves the
existing per-host value. The legacy \`--dpu-mode\` flag and values remain
accepted.\

\
*Possible values:*

- unspecified
- manage

- dpu-mode
- use-as-nic

- nic-mode

- no-dpu
- ignore

**--disable-lockdown** *\<DISABLE_LOCKDOWN\>*
If true, do not lock down the server as part of lifecycle management
Expand Down Expand Up @@ -168,7 +167,7 @@ Print help (see a summary with -h)
nico-admin-cli expected-machine patch --bmc-mac-address 00:11:22:33:44:55 --sku-id DGX-H100-640GB
nico-admin-cli expected-machine patch --id 12345678-1234-5678-90ab-cdef01234567 --sku-id DGX-H100-640GB
nico-admin-cli expected-machine patch --bmc-mac-address 00:11:22:33:44:55 --bmc-username admin --bmc-password mynewpassword
nico-admin-cli expected-machine patch --bmc-mac-address 00:11:22:33:44:55 --dpu-mode no-dpu
nico-admin-cli expected-machine patch --bmc-mac-address 00:11:22:33:44:55 --dpu-policy ignore
```

---
Expand Down
52 changes: 33 additions & 19 deletions docs/manuals/vpc/flat_vpcs_zero_dpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ abbreviated as `…/nico/...` thereafter.

| Task | Role | Interface |
|---|---|---|
| Put hosts in NIC / no-DPU mode (site-wide `dpu_mode`, per-host `ExpectedMachine.dpu_mode`) | Operator | **TOML** — Day 0 / rare; API restart |
| Set hosts to `use_as_nic` or `ignore` through the site-wide `dpu_policy` | Operator | **TOML** — Day 0 / rare; API restart |
| Set one host's `ExpectedMachine.dpu_policy` | Operator | **`nico-admin-cli`** (`em add` / `em patch`) — no restart required |
| Declare `HostInband` underlay segments | Operator | **TOML** (`[networks.<name>]`, `type = "hostinband"`) — Day 0 |
| Create an additional `HostInband` segment after Day 0 | Operator | **TOML** (`[networks]`) + API restart, or **`nico-admin-cli`** (gRPC `CreateNetworkSegment`) — see [Configuring HostInband Segments](#2-configuring-hostinband-network-segments) |
| Inspect / delete a `HostInband` segment | Operator | **`nico-admin-cli`** (`network-segment show` / `delete`) |
Expand Down Expand Up @@ -99,34 +100,47 @@ are running without a NICo-managed DPU, the underlay segments those hosts sit on
are declared as `HostInband` segments, and (optionally) instance types exist so
tenants can request the right machines.

### 1. Put hosts in NIC or no-DPU mode
### 1. Set the host DPU policy

Whether a host is a "zero-DPU" host is decided by its **DPU mode**, which has
three values:
Whether NICo treats a host as a "zero-DPU" host is decided by its **DPU policy**,
which has three values:

| `dpu_mode` value | Meaning |
| `dpu_policy` value | Meaning |
|---|---|
| `dpu_mode` | *(default)* The DPU is managed by NICo: BFB/firmware upgrades, HBN deployment, DPU agent, and the tenant overlay all apply. |
| `nic_mode` | A DPU is physically present but is operated as a plain NIC. NICo skips DPU provisioning and overlay management for the host. |
| `no_dpu` | The host has no DPU hardware; its NIC sits directly on the underlay. |
| `manage` | *(site/effective default)* The DPU is managed by NICo: BFB/firmware upgrades, HBN deployment, DPU agent, and the tenant overlay all apply. A per-host `manage` declaration inherits the site policy rather than overriding it. |
| `use_as_nic` | A DPU is physically present but should be operated as a plain NIC. NICo skips DPU provisioning and overlay management for the host. |
| `ignore` | NICo does not configure or attach DPU hardware. Use this for a host without DPUs or when installed DPUs should be intentionally ignored. |

Both `nic_mode` and `no_dpu` make the host a **zero-DPU host** for the purposes
Both `use_as_nic` and `ignore` make the host a **zero-DPU host** for the purposes
of Flat VPCs: NICo does not manage an overlay for it, and its only valid tenant
attachments are `HostInband` segments.

Set the mode in either of two places, with the per-host value taking precedence:
Set the policy in either of two places. Per-host `use_as_nic` and `ignore`
override the site-wide setting; per-host `manage` or an omitted per-host value
instead defers to the site-wide setting for backward compatibility:

- **Site-wide**, in the API server configuration:

```toml
[site_explorer]
dpu_mode = "nic_mode" # or "no_dpu"; omit entirely for the default "dpu_mode"
dpu_policy = "use_as_nic" # or "ignore"; omit entirely for the default "manage"
```

- **Per host**, on the host's `ExpectedMachine` entry, via the `dpu_mode` field.
An explicit per-host `nic_mode` or `no_dpu` always wins over the site-wide
setting; a per-host default (or no entry) defers to the site-wide value, which
in turn defaults to managed `dpu_mode`.
- **Per host**, on the host's `ExpectedMachine` entry, via the `dpu_policy`
field. Per-host `use_as_nic` and `ignore` override the site-wide setting;
for backward compatibility, per-host `manage` or an omitted policy defers to
the site-wide value, which defaults to `manage`.

Existing configuration and admin JSON input remain readable: `dpu_mode` is
accepted as a field alias, and legacy values `dpu_mode`, `nic_mode`, and
`no_dpu` map to `manage`, `use_as_nic`, and `ignore`, respectively. The admin
CLI likewise accepts `--dpu-mode dpu-mode|nic-mode|no-dpu` as legacy aliases;
use `--dpu-policy manage|use-as-nic|ignore` for new automation.

The Forge protobuf boundary deliberately keeps `ExpectedMachine.dpu_mode`
typed as `DpuMode` for generated-client compatibility. NICo maps `DPU_MODE`,
`NIC_MODE`, and `NO_DPU` immediately to `HostDpuPolicy::Manage`, `UseAsNic`,
and `Ignore`; the model and configuration surfaces use the policy names.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Two related Day-0 settings matter for zero-DPU sites:

Expand Down Expand Up @@ -231,7 +245,7 @@ or `nicocli` (its wrapper), which expose the full surface:
For a zero-DPU site, create instance type(s) describing the zero-DPU machines'
capabilities and associate those machines, so tenants can request instances of
that type. The instance type itself does **not** carry a "no-DPU" flag — what
makes a host zero-DPU is its `dpu_mode` (above). The instance type simply selects
makes a host zero-DPU is its `dpu_policy` (above). The instance type simply selects
which machines are allocatable; whether the selected host is zero-DPU then
governs the network model at allocation time.

Expand Down Expand Up @@ -399,9 +413,9 @@ operator SDN integration can tie its switch-side configuration to the VPC.

**Operator — site is Flat-ready:**

1. Hosts resolve to a zero-DPU mode. Confirm the intended hosts report
`nic_mode` or `no_dpu` (per-host `ExpectedMachine.dpu_mode`, or the site-wide
`[site_explorer] dpu_mode`).
1. Hosts resolve to a zero-DPU policy. Confirm the intended hosts use
`use_as_nic` or `ignore` (per-host `ExpectedMachine.dpu_policy`, or the
site-wide `[site_explorer] dpu_policy`).
2. `HostInband` segments exist. `nico-admin-cli network-segment show` lists the
declared segments with the expected prefix, gateway, and a VLAN/VNI assigned.
3. Each zero-DPU host's data-NIC MACs are registered, and the host shows
Expand Down
2 changes: 1 addition & 1 deletion docs/observability/core_metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ This file contains a list of metrics exported by NVIDIA Infra Controller (NICo).
<tr><td>carbide_site_explorer_create_machines_latency_milliseconds</td><td>histogram</td><td>The time it took to perform create_machines inside site-explorer</td></tr>
<tr><td>carbide_site_explorer_created_machines_count</td><td>gauge</td><td>Number of machine pairs created by Site Explorer after identification</td></tr>
<tr><td>carbide_site_explorer_created_power_shelves_count</td><td>gauge</td><td>Number of power shelves created by Site Explorer after identification</td></tr>
<tr><td>carbide_site_explorer_dpu_migration_signals_count</td><td>gauge</td><td>Number of DPU NIC-mode migration signals by signal type -- mode-mismatch found, set_nic_mode issued, reset requested, and zero-DPU registered for a NicMode host.</td></tr>
<tr><td>carbide_site_explorer_dpu_migration_signals_count</td><td>gauge</td><td>Number of DPU NIC-mode migration signals by signal type -- mode-mismatch found, set_nic_mode issued, reset requested, and zero-DPU registered for a host whose DPU policy is use_as_nic.</td></tr>
<tr><td>carbide_site_explorer_enabled</td><td>gauge</td><td>Whether site-explorer is enabled (1) or paused (0)</td></tr>
<tr><td>carbide_site_explorer_iteration_latency_milliseconds</td><td>histogram</td><td>The time it took to perform one site explorer iteration</td></tr>
<tr><td>carbide_site_explorer_last_run_status</td><td>gauge</td><td>The status of the latest Site Explorer run</td></tr>
Expand Down
Loading
Loading