diff --git a/docs/index.yml b/docs/index.yml index 4e35e54d97..772ddc6106 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -112,6 +112,8 @@ navigation: path: provisioning/ingesting-hosts.md - page: Ingesting Hosts (REST API) path: provisioning/ingesting-hosts-rest-api.md + - page: Site Setup API Parity + path: provisioning/site-setup-api-parity.md - page: Boot Interfaces and DPU Modes path: provisioning/boot-interfaces-and-dpu-modes.md - page: Machine Validation diff --git a/docs/playbooks/machine_reboot.md b/docs/playbooks/machine_reboot.md index a50168c71a..ee612db545 100644 --- a/docs/playbooks/machine_reboot.md +++ b/docs/playbooks/machine_reboot.md @@ -15,31 +15,27 @@ instance lifecycle.** The following steps can be used to reboot a machine: -### 1. Obtain access to `nico-admin-cli` +### 1. Obtain access to `nicocli` -See [nico-admin-cli access on a NICo deployment](nico_admin_cli.md). +Configure `nicocli` for the target REST API. The caller's organization must have an Infrastructure Provider that owns the Site containing the Machine, and the caller must have the `PROVIDER_ADMIN` role. -### 2. Execute the `nico-admin-cli machine reboot` command +### 2. Execute the Machine power control operation -`nico-admin-cli machine reboot` can be used to restart a machine. -It always will require the machine's BMC IP and port to be specified. - -BMC credentials can either be explicitly passed, or the `--machine-id` parameter -can be used to let the NICo site controller read the last known credentials -for the machine. - -Rebooting a machine will also always reset its boot order. The machine -will PXE boot, and thereby will be able to retrieve new boot instructions from -the NICo site controller. - -**Example:** +Use `GracefulRestart` when the operating system can shut down cleanly. Use `ForceRestart` only when a graceful restart is not possible. ```bash -/opt/nico/nico-admin-cli -a https://127.0.0.1:1079 machine reboot --address 123.123.123.123 --port 9999 --machine-id="60cef902-9779-4666-8362-c9bb4b37184f" +nicocli machine power-control-machine machine-power-control-machine \ + --action GracefulRestart \ + ``` -or using username and password: +If the Machine has an attached Instance, acknowledge the workload disruption explicitly: ```bash -/opt/nico/nico-admin-cli -a https://127.0.0.1:1079 machine reboot --address 123.123.123.123 --port 9999 --username myhost --password mypassword +nicocli machine power-control-machine machine-power-control-machine \ + --action GracefulRestart \ + --acknowledge-attached-instance true \ + ``` + +A successful request returns HTTP 202. Retrieve the Machine afterward with `nicocli machine get ` and confirm that it returns to the expected lifecycle state. diff --git a/docs/provisioning/ingesting-hosts.md b/docs/provisioning/ingesting-hosts.md index 0a7b0b69f1..ec1891b953 100644 --- a/docs/provisioning/ingesting-hosts.md +++ b/docs/provisioning/ingesting-hosts.md @@ -2,13 +2,15 @@ Once you have NVIDIA Infra Controller (NICo) up and running, you can begin ingesting machines. +The preferred operator workflow uses the REST API and `nicocli`. Follow [Ingesting Hosts (REST API)](ingesting-hosts-rest-api.md) for credential setup, Expected Machine registration, ingestion verification, and table maintenance. The direct Core workflow below remains available for operations that do not yet have REST parity; see [Site Setup API Parity](site-setup-api-parity.md) for the current status and tracked gaps. + ## Prerequisites Ensure you have the following prerequisites met before ingesting machines: -1. You have the `nico-admin-cli` command available: You can compile it from sources or you can use the pre-compiled binary. Another choice is to use a containerized version. You can also download it from the cluster; see next section for details. +1. You have `nicocli` installed and configured for the target REST API. See the [Quick Start Guide](../getting-started/quick-start.md). -2. You can access the NICo site using the `nico-admin-cli`. +2. For the remaining REST parity gaps, you have `nico-admin-cli` and direct access to the NICo site. See the next section for details. 3. The NICo API service is running at IP address `NICo_API_EXTERNAL`. It is recommended that you add this IP address to your trusted list. @@ -111,95 +113,61 @@ The required credentials include the following: https://api-. --nico-root-ca-path --client-cert-path --client-key-path ``` -### Update Host and DPU BMC Password - -Run this command to update the desired Host and DPU BMC password: - -```bash -nico-admin-cli -a credential add-bmc --kind=site-wide-root --password='x' -``` - -### Update Host UEFI Password +### Store Host and DPU BMC Password -Run this command to generate the desired host UEFI password: +Run this command to store the desired Host and DPU BMC password: ```bash -nico-admin-cli -a host generate-host-uefi-password +nicocli bmc-credential create \ + --site-id \ + --kind SiteWideRoot \ + --password '' ``` +### Store Host and DPU UEFI Passwords -Run this command to update host uefi password: +Run this command to store the desired host UEFI password: ```bash -nico-admin-cli -a credential add-uefi --kind=host --password='' +nicocli uefi-credential create \ + --site-id \ + --kind Host \ + --password '' ``` -Run this command to update DPU uefi password: +Run this command to store the desired DPU UEFI password: ```bash -nico-admin-cli -a credential add-uefi --kind=dpu --password='x' +nicocli uefi-credential create \ + --site-id \ + --kind DPU \ + --password '' ``` ## Add Expected Machines Table NICo needs to know the factory default credentials for each BMC, which is expressed as a JSON table of "Expected Machines". The serial number is used to verify the BMC MAC matches the actual serial number of the chassis. -Prepare an `expected_machines.json` file as follows: +Register a single Expected Machine with `nicocli`: -```json -{ - "expected_machines": [ - { - "bmc_mac_address": "C4:5A:B1:C8:38:0D", - "bmc_username": "root", - "bmc_password": "default-password1", - "chassis_serial_number": "SERIAL-1" - }, - { - "bmc_mac_address": "C4:5A:FF:FF:FF:FF", - "bmc_username": "root", - "bmc_password": "default-password2", - "chassis_serial_number": "SERIAL-2" - } - ] -} +```bash +nicocli expected-machine create \ + --site-id \ + --bmc-mac-address \ + --chassis-serial-number \ + --default-bmc-username \ + --default-bmc-password ``` -Only servers listed in this table will be ingested, so you must include all servers in this file. - -### Optional Per-Host Fields - -Each entry supports additional optional fields: - -- **`host_lifecycle_profile`** (object): Per-host profile for settings that affect - state-machine progression. Future per-host knobs should be added here. - - **`disable_lockdown`** (bool, default `false`): When `true`, the state machine - does not lockdown the host during lifecycle management. This is useful for automation - workflows that need lockdown persistently disabled. - - ```json - { - "bmc_mac_address": "C4:5A:B1:C8:38:0D", - "bmc_username": "root", - "bmc_password": "default-password1", - "chassis_serial_number": "SERIAL-1", - "host_lifecycle_profile": { - "disable_lockdown": true - } - } - ``` - -- **`dpf_enabled`** (bool): Enable/disable DPF for this host. -- **`dpu_mode`** (`"dpu_mode"` | `"nic_mode"` | `"no_dpu"`): Per-host DPU operating mode. -- **`bmc_retain_credentials`** (bool): Skip BMC password rotation. -- **`default_pause_ingestion_and_poweron`** (bool): Pause ingestion and power-on for this host. -- **`bmc_ip_address`** (string): Static BMC IP (pre-allocates a machine interface). - -When the file is ready, upload it to the site with the following command: +For more than one machine, prepare the JSON array documented in [Ingesting Hosts (REST API)](ingesting-hosts-rest-api.md#batch-recommended-for-full-rack-onboarding), then run: ```bash -nico-admin-cli -a em replace-all --filename expected_machines.json +nicocli expected-machine batch-create --data-file expected-machines.json ``` +Only registered Expected Machines will be ingested. + +For optional REST fields and batch JSON examples, use [Ingesting Hosts (REST API)](ingesting-hosts-rest-api.md#registering-expected-machines). + ## Approve all Machines for Ingestion NICo uses Measured Boot using the on-host Trusted Platform Module (TPM) v2.0 to enforce cryptographic identity of the host hardware and firmware. @@ -234,8 +202,8 @@ When a machine is not being created or is stuck in a pre-`Ready` state, `nico-ap You can check the current detailed state of any managed host using: ```bash -nico-admin-cli -a managed-host show --all -nico-admin-cli -a managed-host show +nicocli machine list --output table +nicocli machine get ``` For a full guide on diagnosing stuck objects, including how to use the NICo Grafana dashboard and how to read state handler error logs, see [Stuck Objects Runbook](../playbooks/stuck_objects/stuck_objects.md). @@ -299,26 +267,36 @@ The expected machines table in the nico-api database holds the following fields ### Individual operations -Use `nico-admin-cli` to operate on individual entries: +Use `nicocli` to operate on individual entries: ```bash -nico-admin-cli -a em update ... -nico-admin-cli -a em add ... -nico-admin-cli -a em delete ... +nicocli expected-machine update \ + --description '' \ + +nicocli expected-machine create \ + --site-id \ + --bmc-mac-address \ + --chassis-serial-number \ + --default-bmc-username \ + --default-bmc-password +nicocli expected-machine delete ``` ### Bulk operations -Replace all entries from a JSON file: +Create or update entries from a JSON file: ```bash -nico-admin-cli -a em replace-all --filename expected_machines.json +nicocli expected-machine batch-create --data-file expected-machines.json +nicocli expected-machine batch-update --data-file expected-machine-updates.json ``` -Erase all entries: +See [Ingesting Hosts (REST API)](ingesting-hosts-rest-api.md#batch-update) for the batch update JSON shape. + +Delete an entry by ID: ```bash -nico-admin-cli -a em erase +nicocli expected-machine delete ``` ### Export @@ -326,5 +304,5 @@ nico-admin-cli -a em erase Export the current table as JSON: ```bash -nico-admin-cli -a -f json em show -``` \ No newline at end of file +nicocli expected-machine list --all --output json +``` diff --git a/docs/provisioning/site-setup-api-parity.md b/docs/provisioning/site-setup-api-parity.md new file mode 100644 index 0000000000..496e63a7bd --- /dev/null +++ b/docs/provisioning/site-setup-api-parity.md @@ -0,0 +1,26 @@ +# Site Setup API Parity + +Use the REST API and `nicocli` for site setup whenever an operation is listed as covered below. Direct `nico-admin-cli` access remains necessary only for the gaps that have not yet reached the REST API. + +| Site setup task | Current status | Preferred command or tracked gap | +|---|---|---| +| Store the site-wide BMC root credential | Covered | `nicocli bmc-credential create --site-id --kind SiteWideRoot --password ` | +| Store the site-default host UEFI credential | Covered | `nicocli uefi-credential create --site-id --kind Host --password ` | +| Store the site-default DPU UEFI credential | Covered | `nicocli uefi-credential create --site-id --kind DPU --password ` | +| Register an Expected Machine | Covered | `nicocli expected-machine create --site-id --bmc-mac-address --chassis-serial-number --default-bmc-username --default-bmc-password ` | +| Register Expected Machines in a batch | Covered | `nicocli expected-machine batch-create --data-file expected-machines.json` | +| Approve, list, or remove measured-boot machine trust rules | Gap | Tracked by [#2801](https://github.com/NVIDIA/infra-controller/issues/2801). Use `nico-admin-cli attestation measured-boot site trusted-machine approve`, `list`, or `remove` until that issue is complete. | +| Approve, list, or remove measured-boot profile trust rules | Gap | Tracked by [#2801](https://github.com/NVIDIA/infra-controller/issues/2801). Use `nico-admin-cli attestation measured-boot site trusted-profile approve`, `list`, or `remove` until that issue is complete. | +| Clear a Site Explorer endpoint error | Gap | Tracked by [#2802](https://github.com/NVIDIA/infra-controller/issues/2802). Use `nico-admin-cli site-explorer clear-error ` until that issue is complete. | +| Queue a Site Explorer endpoint for re-exploration | Gap | Tracked by [#2802](https://github.com/NVIDIA/infra-controller/issues/2802). Use `nico-admin-cli site-explorer re-explore ` until that issue is complete. Bulk selection and execution also remain part of this gap. | + +## Remaining parity plan + +The REST/nicocli parity work is tracked under [#2852](https://github.com/NVIDIA/infra-controller/issues/2852): + +- [#2799](https://github.com/NVIDIA/infra-controller/issues/2799) covers site-wide BMC and UEFI credentials and is complete. +- [#2801](https://github.com/NVIDIA/infra-controller/issues/2801) adds measured-boot trust approval operations. +- [#2802](https://github.com/NVIDIA/infra-controller/issues/2802) adds Site Explorer clear-error and re-explore operations. +- [#2803](https://github.com/NVIDIA/infra-controller/issues/2803) covers additional admin-cli inventory. Its implementation merged in [#3241](https://github.com/NVIDIA/infra-controller/pull/3241); the issue remains open for administrative closure. + +After #2801 and #2802 are complete, update this table to replace the remaining direct admin-cli commands with their generated `nicocli` commands.