feat: activate NVLink fabric partitions for vGPU VFs on NVSwitch systems#193
Open
lexfrei wants to merge 8 commits into
Open
feat: activate NVLink fabric partitions for vGPU VFs on NVSwitch systems#193lexfrei wants to merge 8 commits into
lexfrei wants to merge 8 commits into
Conversation
lexfrei
marked this pull request as ready for review
July 4, 2026 08:32
This was referenced Jul 4, 2026
How to adapt the Shared NVSwitch Virtualization Model of FM to activate nvlink in multi-gpu VMs
#133
Open
Collaborator
|
Can you replace the fabric bindings with https://github.com/NVIDIA/go-nvfm ? |
Author
|
Done — switched the fabric bindings from the inline cgo ABI to go-nvfm. The plugin now drives libnvfm through go-nvfm, which dlopen's it at runtime and ships the SDK headers, so the hand-maintained ABI transcription and the link-time dependency are both gone. Behaviour is unchanged, and it builds/vets/tests clean under linux + cgo. |
This was referenced Jul 12, 2026
…work Ada/Hopper+ GPUs (vGPU 17+) no longer expose vGPU instances through mdev; instead, SR-IOV Virtual Functions stay bound to the "nvidia" host driver and are configured directly on the PCI function via sysfs. The existing vGPU discovery only walks /sys/bus/mdev/devices, so it finds nothing on these GPUs and every vGPU VF ends up indistinguishable from a whole-card passthrough device, since both share the same PCI vendor:device id. Add a second discovery path that walks /sys/bus/pci/devices looking for Nvidia functions bound to the "nvidia" driver, skips Physical Functions (identified by sriov_totalvfs), and treats a Virtual Function as a configured vGPU when its current_vgpu_type is non-zero. The profile name is resolved by matching that type id against the creatable_vgpu_types catalog, merged per physical card (scoped by each VF's physfn parent) since a configured VF's own list is frequently empty or reduced to the active entry, and since vGPU type ids are only unique within a single physical card. Because these VFs are ordinary PCI/IOMMU-group devices, discovered functions are added to the same iommu/BDF maps used by GPU passthrough and dispatched through the existing GenericDevicePlugin, grouped by vGPU profile name instead of PCI device id. No changes were needed to the allocate path or device contract; the mdev-backed vGPU discovery is untouched. Document the new hardware path in the README alongside the existing mdev instructions, since it applies to a different GPU generation and neither replaces the other. Signed-off-by: Aleksei Sviridkin <f@lex.la>
A fully consumed card reduces the creatable_vgpu_types list of every one of its functions down to the header, so the per-card catalog cannot resolve the types of already-configured VFs and the plugin advertised nothing exactly when all capacity was allocated. Fall back to a host-wide catalog merged across all cards; numeric ids that different cards map to different names stay ambiguous and are still skipped rather than guessed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
On a host where every card is fully consumed, no creatable_vgpu_types catalog lists anything at all — sysfs alone cannot resolve configured type ids to profile names no matter how catalogs are merged. Resolve through NVML as the last resort: the supported-vGPU list of a card does not shrink as capacity is allocated. Scoped per physical card like the sysfs catalogs, cached per scan, and requires the host driver library to be loadable in the container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ation On NVSwitch systems running SR-IOV vGPU with Fabric Manager in FABRIC_MODE=2, a whole-card guest cannot initialise CUDA until its VF's NVLink fabric partition is activated through the Fabric Manager SDK (cuInit fails with error 802 otherwise). Add a self-contained pkg/fabric wrapping libnvfm (the nv_fm_agent API): - cgo binding (linux && cgo) linking libnvfm.so.1, with a portable stub for other platforms and CGO_ENABLED=0 so the whole module still builds and unit-tests on developer machines. The FM ABI is declared in the binding; NVIDIA's proprietary headers are not vendored. - Connect / GetSupportedPartitions / ActivateWithVFs / Deactivate / Close with sizeof-based struct-version handling and a full fmReturn_t -> error mapping. The connection address type (Unix socket vs host:port TCP) is selected from the address form. - NVML helpers to resolve a physical GPU's FM physicalId (module id) and MIG mode, and a VF -> single-GPU partition resolver keyed strictly on physicalId, never on PCI enumeration order. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Activate a whole-card vGPU VF's NVLink fabric partition during GenericDevicePlugin.Allocate, before the device is handed to the virt-launcher pod, so the guest's CUDA init succeeds with per-VM NVLink isolation from the first init. For each allocated VF the plugin resolves its single-GPU fabric partition (VF -> parent PF -> FM physicalId -> partition) and activates it for that one VF; fmActivateFabricPartitionWithVFs takes one VF per GPU, so a single-GPU partition is always activated with exactly one VF. MIG-mode GPUs have NVLink disabled and are not in the fabric, so their VFs need no activation and are skipped (nvmlDeviceGetMigMode). The feature is a no-op on non-fabric systems and classic passthrough GPUs. Configuration via environment: FABRIC_PARTITION_ACTIVATION (auto/off), FABRIC_MANAGER_ADDRESS (default the FM TCP command interface), FABRIC_FAIL_MODE (closed: an FM error fails the allocation; open: log and allow), and FABRIC_RECONCILE_INTERVAL. A pod-resources reconciler deactivates a partition once its VF is gone, since the device plugin API has no release callback. Unit-tested with a fake Fabric Manager client. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fabric partition activation binding links libnvfm.so.1 (NEEDED in the binary), so both the builder and the runtime image need it. Fetch a pinned libnvfm from the CUDA repository package at build time (only the versioned shared object; the FM ABI is declared in-tree, headers are not required), link against it, and copy it into the distroless runtime image on the loader search path. The version is a build ARG so it can track the target host's Fabric Manager daemon. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Document the fabric partition activation design: the FM SDK binding, the VF -> single-GPU partition resolver, single-VF activation (one VF per GPU), MIG-skip, the pod-resources reconciler for deactivation, the configuration, the runtime requirements (hostNetwork for the FM TCP command interface, pod-resources mount, FABRIC_MODE=2), and libnvfm image bundling. Includes a draft upstream PR description. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hand-written inline cgo Fabric Manager SDK binding with the official github.com/NVIDIA/go-nvfm library. The previous binding declared the nv_fm_agent struct layouts and entry points by hand in the cgo preamble (NVIDIA's headers cannot be redistributed) and linked libnvfm.so.1 at build time. go-nvfm ships the FM SDK headers and dlopen's libnvfm at runtime, so both the hand-maintained ABI transcription and the link-time dependency are gone. Behaviour is unchanged: same Client interface and call sequence, same single-VF partition activation, same physicalId-based resolution and MIG-mode skip, same 5s connect timeout, same TCP/Unix address handling, and the same return-code to sentinel-error mapping the allocation logic branches on. The runtime image still bundles the versioned libnvfm.so.1; go-nvfm is pointed at that SONAME because the unversioned -dev symlink is not shipped in the distroless image. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
feat/fabric-partition-activation
branch
from
July 20, 2026 00:43
495619e to
a8bf9ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #133.
On NVSwitch systems (HGX H100/H200) running SR-IOV vGPU with Fabric Manager in
FABRIC_MODE=2, a whole-card (NVLink-enabled) guest cannot initialise CUDA —cuInitfails with error802("system not yet initialized") — until its VF's NVLink fabric partition is activated through the Fabric Manager SDK. The device plugin hands the VF to thevirt-launcherpod but nothing activates the fabric partition, so those guests fail CUDA on these systems.This change activates the VF's single-GPU fabric partition during
Allocate, before the guest starts, giving working CUDA from the first init with per-VM NVLink isolation:pkg/fabricwrapslibnvfm(thenv_fm_agentAPI) via cgo, with a portable stub so non-linux /CGO_ENABLED=0builds are unaffected. The FM ABI is declared in the binding (headers not vendored).Allocateresolves the VF → parent PF → FMphysicalId(NVML module id) → single-GPU partition, matched strictly onphysicalId(never PCI order), then activates the partition for exactly that VF (fmActivateFabricPartitionWithVFstakes one VF per GPU).nvmlDeviceGetMigMode). Only whole-card VFs are activated.FABRIC_PARTITION_ACTIVATION(defaultauto), a no-op on non-NVSwitch systems and classic passthrough GPUs.FABRIC_FAIL_MODE(defaultclosed) chooses whether an FM error fails the allocation or is logged and allowed. A pod-resources reconciler deactivates a partition once its VF is gone.libnvfm.so.1; the pod needshostNetwork(to reach the FM command API on127.0.0.1:6666) and the pod-resources socket mounted, and FM inFABRIC_MODE=2.See
docs/fabric-partition-activation.mdfor the full design.Scope
This PR activates single-GPU fabric partitions — one partition per allocated VF, resolved from the VF's parent physical GPU. Multi-GPU partition activation (giving one VM several whole-card vGPUs with NVLink P2P between them — which requires activating a predefined multi-GPU partition with one VF per member GPU, plus
GetPreferredAllocationsupport so the allocated VFs land on GPUs that form a defined partition) is deliberately out of scope here and implemented in the follow-up #195. MIG-backed vGPUs need no activation at all: MIG disables NVLink, so the GPU is not on the fabric and CUDA initializes standalone.