Is this a new feature, an enhancement, or a change to existing functionality?
Enhancement
How would you describe the priority of this feature request
Medium
Please provide a clear description of problem this feature solves
For Supermicro ARS-121L-DNR systems, there are two nodes in one chassis. For updating firmware, the procedure requires a reboot of the "DPU Chassis" as seen by the bfvcheck output:
# bfvcheck
Beginning version check...
-RECOMMENDED VERSIONS-
ATF: v2.2(release):4.13.2-4-g4437295a8
UEFI: 4.13.2-12-g943a91640d
FW: 32.47.2682
-INSTALLED VERSIONS-
ATF: v2.2(release):4.13.2-4-g4437295a8
UEFI: 4.13.2-12-g943a91640d
FW: 32.47.1026
WARNING: FW VERSION DOES NOT MATCH RECOMMENDED!
WARNING: The firmware has been updated to 32.47.2682, but the chassis
must be power cycled for changes to take effect.
The way we normally do this is by draining the power of the host. This cuts power to the DPU and it does power cycle. However, we have observed that does not happen on a Suparmicro ARS-121L-DNR and the DPU does not perform a power drain. As a result, the machine remains stuck in Reprovisioning/VerifyFirmareVersions ⏱️ until one manually intervenes.
Feature Description
When updating the firmware of a DPU in a Supermicro ARS-121L-DNR the state machine shall first power down the host, then perform a chassis powercycle of the Card1 entry via redfish to initiate the application of the new firmware.
Describe your ideal solution
This process has been tested on two machines where this issue got resolved, both were Supermicro ARS-121L-DNR servers with Bluefield-3 DPUs at the firmware levels from the problem description section. The following was performed after the current statemachine installed the firmware but got stock with verifying firmware:
- Power down the host server via the NICo web interface and wait until the server is powered off.
- Issue the following Redfish call to the BMC of the DPU:
curl -k -s -u "$DPU_BMC_USER:$DPU_BMC_PASSWORD" \
-H 'Content-Type: application/json' \
-X POST \
-d '{"ResetType":"PowerCycle"}' \
https://$DPU_BMC_IP/redfish/v1/Chassis/Card1/Actions/Chassis.ResetThe DPU then did a reboot and the new firmware was applied. If you were to to issue the Redfish command from (2) while the host was powered on, the call would return the following "success" message:
{
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_1_1.Message",
"Message": "The request completed successfully.",
"MessageArgs": [],
"MessageId": "Base.1.18.1.Success",
"MessageSeverity": "OK",
"Resolution": "None."
}
]
}But the DPU would not do a reboot.
Describe any alternatives you have considered
Since this is an enhancement request on the reboot procedure that is built-in to NICo, the above is the manual workaround when the machine gets stuck. An alternative is that the operations team implements this procedure as a workaround when the state is stuck. I.e. trigger an alert if the condition is met and run an automated runbook that performs the above actions.
Additional context
The proposed workflow update was proven successful 100% of the time in the described situation. Variations were tested with "lockdown enabled" and "lockdown disabled" which have no impact to the outcome of the procedure.
Code of Conduct
Is this a new feature, an enhancement, or a change to existing functionality?
Enhancement
How would you describe the priority of this feature request
Medium
Please provide a clear description of problem this feature solves
For Supermicro ARS-121L-DNR systems, there are two nodes in one chassis. For updating firmware, the procedure requires a reboot of the "DPU Chassis" as seen by the bfvcheck output:
The way we normally do this is by draining the power of the host. This cuts power to the DPU and it does power cycle. However, we have observed that does not happen on a Suparmicro ARS-121L-DNR and the DPU does not perform a power drain. As a result, the machine remains stuck in
Reprovisioning/VerifyFirmareVersions ⏱️until one manually intervenes.Feature Description
When updating the firmware of a DPU in a Supermicro ARS-121L-DNR the state machine shall first power down the host, then perform a chassis powercycle of the Card1 entry via redfish to initiate the application of the new firmware.
Describe your ideal solution
This process has been tested on two machines where this issue got resolved, both were Supermicro ARS-121L-DNR servers with Bluefield-3 DPUs at the firmware levels from the problem description section. The following was performed after the current statemachine installed the firmware but got stock with verifying firmware:
The DPU then did a reboot and the new firmware was applied. If you were to to issue the Redfish command from (2) while the host was powered on, the call would return the following "success" message:
{ "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The request completed successfully.", "MessageArgs": [], "MessageId": "Base.1.18.1.Success", "MessageSeverity": "OK", "Resolution": "None." } ] }But the DPU would not do a reboot.
Describe any alternatives you have considered
Since this is an enhancement request on the reboot procedure that is built-in to NICo, the above is the manual workaround when the machine gets stuck. An alternative is that the operations team implements this procedure as a workaround when the state is stuck. I.e. trigger an alert if the condition is met and run an automated runbook that performs the above actions.
Additional context
The proposed workflow update was proven successful 100% of the time in the described situation. Variations were tested with "lockdown enabled" and "lockdown disabled" which have no impact to the outcome of the procedure.
Code of Conduct