feat(device_plugin): discover vGPU VFs via vendor-specific VFIO framework

[lexfrei]

feat(device_plugin): discover vGPU VFs via vendor-specific VFIO framework

Problem

This plugin discovers vGPU instances by walking /sys/bus/mdev/devices. On Ada Lovelace and newer GPUs (vGPU release 17+, e.g. an H200 SXM system), the host driver no longer exposes vGPU through mdev at all — vGPU profiles are assigned per SR-IOV Virtual Function through a vendor-specific VFIO sysfs interface instead. On these GPUs the mdev walk finds nothing, so no vGPU resources are ever advertised, and every VF (regardless of the profile configured on it) shares the same PCI vendor:device id as the whole card, making per-profile scheduling impossible through the existing GPU-passthrough path either.

KubeVirt's own built-in PCI device plugin already recognizes this pattern (kubevirt/kubevirt#16890, merged in v1.9.0-beta.0): a PCI function bound to the nvidia driver with a non-zero current_vgpu_type is treated as a permitted vGPU VF. This plugin exists specifically to implement the same per-profile external-device-plugin pattern KubeVirt expects (externalResourceProvider: true), so it should recognize these VFs the same way.

Related upstream context: NVIDIA/gpu-operator#2594 tracks the separate gap of creating vGPU profiles on these VFs (writing current_vgpu_type) via the GPU Operator's vgpu-device-manager. This PR does not touch creation — it only discovers and advertises VFs that already have a profile configured (created manually, or by whatever eventually addresses that issue).

What changed

• Added pkg/device_plugin/vfio_vgpu.go, a second vGPU discovery path alongside the existing mdev-based one (createVgpuIDMap, untouched).
• Walks /sys/bus/pci/devices for Nvidia (10de) functions bound to the nvidia driver, skips Physical Functions (identified by the sriov_totalvfs attribute, consistent with how KubeVirt itself distinguishes PFs), and treats a Virtual Function as a configured vGPU when current_vgpu_type != 0.
• The vGPU profile name is resolved by matching the configured type id against creatable_vgpu_types ("<id> : <name>" lines), sanitized the same way the mdev path already sanitizes profile names (whitespace replaced with _) so both frameworks produce identically formatted resource names for a profile of the same name.
• Discovered VFs are grouped by profile name into a new vfioVGpuMap, wired into InitiateDevicePlugin/createDevicePlugins in pkg/device_plugin/device_plugin.go alongside the existing deviceMap (GPU passthrough) and vGpuMap (mdev vGPU) loops.

Design notes

Profile name resolution. On a configured VF, creatable_vgpu_types is reduced (down to just the header once the card's capacity is fully allocated), and on at least one H200 SXM system the Physical Function exposes no nvidia/ sysfs directory at all — so the unconfigured sibling VFs are the only sysfs catalog source. Resolution is a three-step chain, all scoped per physical card (VFs keyed by their physfn parent) where possible:

1. the card's own creatable_vgpu_types catalog;
2. a host-wide catalog merged across all cards, guarded against ambiguity (a numeric type id that different cards map to different names is never guessed);
3. NVML (nvmlDeviceGetSupportedVgpus + nvmlVgpuTypeGetName on the parent PF) as the last resort — unlike the creatable list, the supported list does not shrink as capacity is allocated, so it keeps working on a fully tiled-out host where every sysfs catalog on every card is reduced simultaneously. That state is not hypothetical: it is the steady state of a fully allocated node, exactly when the plugin must keep advertising. NVML requires libnvidia-ml.so.1 to be loadable in the container — bind-mount the single library file from the host driver installation (mounting the whole host library directory would drag the host glibc into the container) and give the pod access to the NVIDIA device nodes (e.g. privileged: true with /dev mounted).

Allocate contract. No new allocation logic was needed. Discovered VFs are added to the same iommuMap/bdfToIommuMap used by GPU passthrough, and dispatched through the existing GenericDevicePlugin (grouped by profile name instead of PCI device id) rather than GenericVGpuDevicePlugin, which implements the mdev-specific env-var contract. This was verified against KubeVirt's own built-in plugin: its PCIDevicePlugin.Allocate/formatVFIODeviceSpecs mount /dev/vfio/vfio plus /dev/vfio/<iommu_group> unconditionally, regardless of whether the underlying driver is vfio-pci or nvidia — the classic group-based VFIO device node is what KubeVirt itself expects for these VFs, not the per-device vfio-dev cdev. This plugin's GenericDevicePlugin.Allocate already mounts that same group node, and additionally mounts the per-device /dev/vfio/devices/vfioN cdev plus /dev/iommu when iommufd is supported host-wide — a strict superset of KubeVirt's own contract, reused here unmodified.

mdev path. createVgpuIDMap, GenericVGpuDevicePlugin, and the MDEV_PCI_RESOURCE_* env contract are untouched. The two discovery paths populate disjoint maps (vGpuMap vs vfioVGpuMap) and are expected to be mutually exclusive in practice — mdev and the vendor-specific VFIO framework apply to different GPU generations, so no de-duplication between them was implemented.

Testing

Added pkg/device_plugin/vfio_vgpu_test.go covering: profile name parsing/sanitization from creatable_vgpu_types (including the sibling-VF and conflicting-entry cases), current_vgpu_type/sriov_totalvfs file reads, full discovery against a faked sysfs tree (PF, multiple VFs, unconfigured VF, unresolvable type id, an unrelated vfio-pci passthrough device, and a non-Nvidia device all present at once), the host-wide catalog fallback for a fully consumed card (and its refusal to guess when cards disagree on an id), and dispatch through createDevicePlugins.

Hardware validation

Run on an 8x H200 SXM system (driver 580.159.01 vGPU host manager, KubeVirt v1.9.0-beta.0, externalResourceProvider: true): 36 MIG-backed VFs (nvidia.com/NVIDIA_H200X-1-18C) and 2 whole-card VFs (nvidia.com/NVIDIA_H200X-141C) discovered, resolved and registered with kubelet as separate per-profile extended resources, with running VMs consuming them through KubeVirt. Empirical findings folded back into the design above: the PF exposes no nvidia/ sysfs directory on this system (the sibling-VF catalog source is load-bearing, not a bonus path); a fully consumed card reduces creatable_vgpu_types to the header on every function; and on the fully allocated node NO sysfs catalog anywhere lists the configured type ids, making the NVML fallback the only working resolution source in that state.

Open questions

• The NVML fallback initializes and shuts down NVML once per PF per discovery scan; if discovery ever becomes hot-path, a single Init/Shutdown around the scan would be the obvious optimization.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[fanzhangio]

Thanks for this PR and hardware validation. The overall approach makes sense to me
I have two concerns before merge:

1. The host-wide type catalog can assign the wrong profile name.

   The code correctly documents that numeric vGPU type IDs are only card-local and may map to different profiles on different cards, even for the same GPU model. However, when the target PF’s catalog is reduced, createVfioVGpuMap() consults globalTypeNames before the authoritative per-PF NVML lookup.

   If the target card has no catalog entry and one other card maps the same numeric ID to a different profile, the entry appears “unambiguous” because the target card contributed no value. The VF can then be advertised under the wrong extended resource.

   Could we prefer the per-PF NVML result before any host-wide inference? If NVML is unavailable, I think skipping the VF is safer than borrowing another PF’s mapping unless there is an enforceable invariant proving the catalogs are compatible. A test covering “target PF unknown, one other PF has the same ID” would help lock this down.

2. The NVML fallback is not usable with the shipped deployment manifest.

   The fallback requires libnvidia-ml.so.1 and access to the necessary NVIDIA device nodes, but the checked-in DaemonSet only mounts /dev/vfio; the image does not include the host driver library. Consequently, on the fully allocated-node scenario that requires NVML, a restart using the default manifest can fail to resolve every profile and advertise no VFs.

   Please update the deployment and README with a supported, minimally privileged way to provide the library and required device nodes, or use a fallback that works with the default deployment. The exact configuration used for the hardware validation would be especially useful.

A couple of smaller follow-ups:

• Please document that profiles must be configured before plugin startup, or add rediscovery for later profile creation/change.
• The function comment around createVfioVGpuMap() still says the fully consumed-card case is not handled and suggests NVML as future work, although this PR now implements it.

The discovery structure, per-PF grouping, IOMMU-map integration, and test coverage otherwise look solid to me.

[lexfrei]

Thanks for the careful review — all four points are addressed.

1. Wrong-profile risk from the host-wide catalog. Removed. The host-wide catalog and its ambiguity tracking are gone. A configured VF's profile name is now resolved strictly by (1) its own card's sysfs creatable_vgpu_types, then (2) the parent PF's own NVML supported-vGPU list, then (3) skipped with a log. Another card's catalog is never consulted, so an id that only a different card maps can no longer be borrowed. Added the test you suggested — "target PF unknown, another PF maps the same id to a different profile": with NVML available it resolves to the per-PF NVML value (never the other card's), and with NVML unavailable the VF is skipped.

2. NVML fallback usable from the manifest. Added manifests/nvidia-kubevirt-gpu-device-plugin-nvml.yaml, which provides libnvidia-ml.so.1 (single-file bind) and the NVIDIA device nodes, and documented it. The default manifest stays minimal and unprivileged. The README describes two supported ways to provide the library and nodes: runtimeClassName: nvidia (minimally privileged, recommended when the NVIDIA Container Toolkit is present) and the hostPath + privileged variant. On device nodes: the plugin's NVML calls (DeviceGetHandleByPciBusId, GetSupportedVgpus, GetName) need /dev/nvidiactl and the per-GPU /dev/nvidiaN; they don't open a CUDA context or MIG capabilities, so /dev/nvidia-uvm* and /dev/nvidia-caps aren't required. To be precise about verified vs inferred: the single-file libnvidia-ml.so.1 bind and the nvidiactl + per-GPU nodes were confirmed on hardware (the validation mounted the whole host /dev under privileged; the shipped manifest narrows it to those nodes); the "uvm/caps not needed" scope is inferred from the NVML calls used, and the NVML path is not exercised in CI.

3. Profile timing. Documented: the plugin scans the vendor-VFIO VFs once at startup and does not re-scan, so all profiles must be configured before it starts; the README also suggests ordering the plugin after whatever recreates the VFs on reboot. Dynamic rediscovery of profiles created or changed while the plugin runs is a separate follow-up PR — I'll link it here once it's open.

4. Stale comment. The createVfioVGpuMap() comment no longer says the fully-consumed-card case is unhandled; it now describes the NVML fallback and the skip.