feat: Add Fabric Manager partition support for NVLink-enabled multi-GPU VMs#158
feat: Add Fabric Manager partition support for NVLink-enabled multi-GPU VMs#158dnugmanov wants to merge 1 commit into
Conversation
9557994 to
13496fa
Compare
…PU VMs Implements support for NVIDIA Fabric Manager partition-aware GPU allocation, enabling NVLink connectivity for multi-GPU VMs in KubeVirt on DGX/HGX H100 systems using the Shared NVSwitch Virtualization Model. Closes NVIDIA#133 Changes: - Add pkg/fabric_manager/ with FM SDK CGO bindings - Implement GetPreferredAllocation for partition-aware allocation - Add automatic partition activation/deactivation in Allocate - Update Dockerfile with FM SDK installation - Add --fm-enabled and --fm-address CLI flags
13496fa to
620a537
Compare
| name: nvidia-kubevirt-gpu-dp-ds | ||
| spec: | ||
| priorityClassName: system-node-critical | ||
| hostNetwork: true # Required for FM API access on localhost:6666 |
There was a problem hiding this comment.
There should be a way to do this without going over the Node's network. A shared socket would be better.
| - name: vfio | ||
| hostPath: | ||
| path: /dev/vfio | ||
| - name: sys |
There was a problem hiding this comment.
Can you explain what we need access to /sys for?
| return dpi | ||
| } | ||
|
|
||
| // SetPartitionManager sets the Fabric Manager partition manager for NVLink support. |
There was a problem hiding this comment.
Why does the device plugin need to configure the Fabric Manager? Can't this happen another way, like through cloud-init or a side-car?
|
Hi — I ran into this PR while working on fabric-partition activation for the SR-IOV vGPU case (opened as #193). There are now three open PRs adding Fabric Manager partition activation to this plugin — yours (#158), #166, and mine (#193) — and none has had maintainer review. #158 and #166 target the shared-NVSwitch / full-passthrough multi-GPU model ( To make this easier to land, would you be OK with me consolidating the overlapping work into a single PR (#193) that covers both the passthrough and the vGPU cases — with you credited as a co-author ( |
|
@lexfrei, these PRs have had review and have been discussed in several channels. Ultimately, the PRs did not make it across the finish line as other avenues are being looked at, like supporting FM in DRA. If there's a need to resurrect this effort than we can discuss how to best do that. |
3 participants
Related Issue
Closes #133 - How to adapt the Shared NVSwitch Virtualization Model of FM to activate nvlink in multi-gpu VMs
Description
This PR implements support for NVIDIA Fabric Manager (FM) partition-aware GPU allocation, enabling NVLink connectivity for multi-GPU VMs in KubeVirt on DGX/HGX H100 systems using the Shared NVSwitch Virtualization Model.
Background
In virtualized environments with DGX/HGX H100 systems, NVIDIA provides the Shared NVSwitch Virtualization Model to enable NVLink connections for multi-GPU VMs. This requires that GPUs assigned to a VM must belong to the same FM partition to establish NVLink fabric connectivity.
What This PR Does
libnvfm) for partition discovery and activationGetPreferredAllocationto recommend GPUs from the same partitionAllocate()and deactivates on pod deletionEnvironment Assumptions
shared_nvswitchmode on the hostvfio-pcidriver for passthroughhostNetwork: trueto access FM on localhostWe would appreciate feedback from maintainers on this implementation approach and any suggestions for improvement.