feat(device_plugin): discover vGPU VFs via vendor-specific VFIO framework

README.md

@@ -171,6 +171,37 @@ total 0
 lrwxrwxrwx. 1 root root 0 Nov 24 13:33 aa618089-8b16-4d01-a136-25a0f3c73123 -> ../../../devices/pci0000:00/0000:00:03.0/0000:03:00.0/0000:04:09.0/0000:06:00.0/aa618089-8b16-4d01-a136-25a0f3c73123
 ```
 
+--------------------------------------------------------------
+### Preparing a GPU to be used in vGPU mode (Ada Lovelace / Hopper and newer)
+
+Starting with vGPU release 17, GPUs based on Ada Lovelace and newer architectures (for example L40S, H100, H200) no longer expose vGPU instances through mdev. `/sys/bus/mdev` does not exist on these hosts. Instead, vGPU profiles are assigned directly on each SR-IOV Virtual Function through a vendor-specific VFIO sysfs interface, and this plugin discovers vGPUs on such hosts through that interface instead of mdev. The steps below replace the mdev steps above; do not use both on the same GPU.
+
+##### 1. Enable SR-IOV Virtual Functions on the physical GPU.
+```shell
+$ /usr/lib/nvidia/sriov-manage -e 0000:41:00.0
+```
+**0000:41:00.0** -- The PCI BDF of the physical GPU (Physical Function).
+
+##### 2. Find the vGPU types that can be created on a Virtual Function.
+```shell
+$ cat /sys/bus/pci/devices/0000\:41\:00.4/nvidia/creatable_vgpu_types
+1428 : NVIDIA H200X-141C
+1414 : NVIDIA H200X-1-18C
+```
+Each line lists a numeric type id and the corresponding profile name.
+
+##### 3. Create a vGPU on the Virtual Function by writing its type id.
+```shell
+$ echo 1414 > /sys/bus/pci/devices/0000\:41\:00.4/nvidia/current_vgpu_type
+```
+
+##### 4. Confirm the vGPU was created.
+```shell
+$ cat /sys/bus/pci/devices/0000\:41\:00.4/nvidia/current_vgpu_type
+1414
+```
+A non-zero value confirms the Virtual Function now has a vGPU profile configured; this plugin advertises it as a `nvidia.com/<profile-name>` resource, grouped separately from Virtual Functions configured with a different profile.
+
 ## Docs
 ### Deployment
 The Daemonset creation yaml can be used to deploy the device plugin. 
@@ -180,6 +211,24 @@ kubectl apply -f nvidia-kubevirt-gpu-device-plugin.yaml
 
 Example YAML files for creating VMs with GPU/vGPU are in the `examples` folder
 
+#### vGPU profile timing (vendor-specific VFIO)
+
+The plugin scans the vendor-specific VFIO vGPU Virtual Functions once at startup and does not re-scan while running. Every profile a node should advertise must therefore be configured on its Virtual Functions (step 3 of "Preparing a GPU to be used in vGPU mode (Ada Lovelace / Hopper and newer)") **before** the device plugin pod starts. A Virtual Function whose profile is created or changed after startup is not picked up until the pod restarts. After a node reboot, order the plugin after whatever recreates the Virtual Functions (for example an init container that waits until the count of configured Virtual Functions is non-zero and stable) so it discovers the full set. Dynamic rediscovery of profiles created or changed while the plugin is running is tracked as a separate follow-up (PR link to be added).
+
+#### NVML fallback (fully consumed nodes)
+
+The profile name of a configured Virtual Function is normally read from its card's `creatable_vgpu_types` catalog. On a fully consumed card that catalog is reduced to its header on every function, so the plugin resolves the configured type ids through NVML instead (`GetSupportedVgpus`, whose list does not shrink as capacity is allocated). This path is reached only on fully consumed nodes; the default manifest does not enable it.
+
+NVML needs two things inside the container:
+
+- **`libnvidia-ml.so.1`** from the host driver, loadable by the dynamic linker. Bind the **single file**, not the host library directory: putting the host lib directory on `LD_LIBRARY_PATH` drags the host glibc into the container and breaks the dynamic linker. This single-file requirement was confirmed on hardware.
+- **The NVIDIA device nodes** the queries touch: `/dev/nvidiactl` (the NVML control device) and the per-GPU `/dev/nvidiaN` for each physical card. The management-only queries this plugin makes (`DeviceGetHandleByPciBusId`, `GetSupportedVgpus`, `GetName`) do not open a CUDA context or MIG capabilities, so `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools` and `/dev/nvidia-caps` are not needed. `/dev/nvidiactl` plus the per-GPU nodes were confirmed on hardware; the narrower "uvm/caps not required" scope is inferred from the set of NVML calls above, not separately tested.
+
+There are two supported ways to provide these:
+
+1. **NVIDIA Container Toolkit (`runtimeClassName: nvidia`) — minimally privileged, recommended.** When the toolkit is installed on the node, set `runtimeClassName: nvidia` and the container env `NVIDIA_VISIBLE_DEVICES=all` and `NVIDIA_DRIVER_CAPABILITIES=utility` (the `utility` capability is the one that provides NVML). The toolkit's runtime hook then injects `libnvidia-ml.so.1` and the device nodes and adds the matching device-cgroup rules, so the container stays non-privileged. This is the standard toolkit mechanism; the exact `runtimeClassName` form was not part of this feature's hardware validation.
+2. **hostPath — privileged.** `manifests/nvidia-kubevirt-gpu-device-plugin-nvml.yaml` single-file-binds `libnvidia-ml.so.1`, sets `LD_LIBRARY_PATH`, mounts only `/dev/nvidiactl` and the per-GPU `/dev/nvidiaN` nodes (not the whole host `/dev`) and runs `privileged: true`. A hostPath device node is not added to the container's device cgroup, so a non-privileged container is denied when it opens the node regardless of the mount, and a plain pod spec has no field to grant a single host device node; `privileged: true` is the only pod-spec-native way to make host device nodes usable without the toolkit. The hardware validation of the fallback used this hostPath + privileged mechanism with the whole host `/dev` mounted; this manifest narrows that to `/dev/nvidiactl` plus the per-GPU nodes, which are the only device nodes the NVML calls above use. Point the `libnvidia-ml.so.1` hostPath at wherever the host driver installed it (find it with `ldconfig -p | grep libnvidia-ml`).
+
 ### Build
 
 Build executable binary using make:

go.mod

@@ -3,6 +3,7 @@ module kubevirt-gpu-device-plugin
 go 1.25.0
 
 require (
+	github.com/NVIDIA/go-nvml v0.13.0-1
 	github.com/NVIDIA/gpu-monitoring-tools v0.0.0-20211102125545-5a2c58442e48
 	github.com/fsnotify/fsnotify v1.9.0
 	github.com/onsi/ginkgo/v2 v2.28.1

go.sum

@@ -1,6 +1,8 @@
 github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
 github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
 github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
+github.com/NVIDIA/go-nvml v0.13.0-1 h1:OLX8Jq3dONuPOQPC7rndB6+iDmDakw0XTYgzMxObkEw=
+github.com/NVIDIA/go-nvml v0.13.0-1/go.mod h1:+KNA7c7gIBH7SKSJ1ntlwkfN80zdx8ovl4hrK3LmPt4=
 github.com/NVIDIA/gpu-monitoring-tools v0.0.0-20211102125545-5a2c58442e48 h1:JO/JF5CBte9mvATbhoh32swu9erf07ZdLgwFj8u21UQ=
 github.com/NVIDIA/gpu-monitoring-tools v0.0.0-20211102125545-5a2c58442e48/go.mod h1:oKPJa5eOTkWvlT4/Y4D8Nds44Fzmww5HUK+xwO+DwTA=
 github.com/NVIDIA/gpu-monitoring-tools/bindings/go/dcgm v0.0.0-20210325210537-29b4f1784f18/go.mod h1:8qXwltEzU3idjUcVpMOv3FNgxxbDeXZPGMLyc/khWiY=

manifests/nvidia-kubevirt-gpu-device-plugin-nvml.yaml

@@ -0,0 +1,99 @@
+# DaemonSet variant that enables the NVML name-resolution fallback.
+#
+# The default manifest (nvidia-kubevirt-gpu-device-plugin.yaml) mounts only
+# /dev/vfio and is enough for every node where at least one function of each
+# card still lists its creatable_vgpu_types catalog. On a fully consumed node
+# every card's sysfs catalog is reduced to the header, and the plugin then
+# resolves configured vGPU type ids through NVML instead. NVML needs the host
+# driver library (libnvidia-ml.so.1) and the NVIDIA device nodes, which the
+# default manifest does not provide.
+#
+# This variant provides them via hostPath. Because a device node mounted via
+# hostPath is not added to the container's device cgroup, opening it requires
+# privileged: true. If the NVIDIA Container Toolkit is installed on the node,
+# prefer the runtimeClassName: nvidia approach documented in the README, which
+# injects the library and the device nodes (with the matching device-cgroup
+# rules) without privileged. See the README section "NVML fallback (fully
+# consumed nodes)" for the trade-offs.
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: nvidia-kubevirt-gpu-dp-daemonset
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      name: nvidia-kubevirt-gpu-dp-ds
+  template:
+    metadata:
+      labels:
+        name: nvidia-kubevirt-gpu-dp-ds
+    spec:
+      priorityClassName: system-node-critical
+      tolerations:
+      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
+      # This, along with the annotation above marks this pod as a critical add-on.
+      - key: CriticalAddonsOnly
+        operator: Exists
+      containers:
+      - name: nvidia-kubevirt-gpu-dp-ctr
+        image: nvcr.io/nvidia/kubevirt-gpu-device-plugin:v1.5.0
+        securityContext:
+          # The NVML fallback opens /dev/nvidiactl and the per-GPU /dev/nvidiaN
+          # nodes. A hostPath device node is not placed in the container's
+          # device cgroup, so a non-privileged container is denied access to it
+          # regardless of the mount, and a plain pod spec has no field to grant
+          # a single host device node. privileged: true is used here as the
+          # only pod-spec-native way to make host device nodes accessible
+          # without the NVIDIA Container Toolkit; for a non-privileged
+          # alternative use runtimeClassName: nvidia (see the README).
+          privileged: true
+        env:
+          # Make the single-file bind of the host driver library loadable by
+          # the dynamic linker. Do NOT add the host lib directory itself to the
+          # search path: it drags the host glibc into the container and breaks
+          # the linker.
+          - name: LD_LIBRARY_PATH
+            value: /host-driver-lib
+        volumeMounts:
+          - name: device-plugin
+            mountPath: /var/lib/kubelet/device-plugins
+          - name: vfio
+            mountPath: /dev/vfio
+          # Single file, not the whole host lib directory - see LD_LIBRARY_PATH.
+          - name: nvml-lib
+            mountPath: /host-driver-lib/libnvidia-ml.so.1
+            readOnly: true
+          # NVIDIA device nodes for NVML: the control node plus one node per
+          # physical GPU. Only these are needed - the plugin's NVML queries do
+          # not open /dev/nvidia-uvm* or /dev/nvidia-caps, and the whole host
+          # /dev is deliberately NOT mounted. This lists /dev/nvidia0 for a
+          # single-GPU node; add a CharDevice volume and mount for each
+          # additional GPU (/dev/nvidia1, /dev/nvidia2, ...).
+          - name: nvidiactl
+            mountPath: /dev/nvidiactl
+          - name: nvidia0
+            mountPath: /dev/nvidia0
+      volumes:
+        - name: device-plugin
+          hostPath:
+            path: /var/lib/kubelet/device-plugins
+        - name: vfio
+          hostPath:
+            path: /dev/vfio
+        - name: nvidiactl
+          hostPath:
+            path: /dev/nvidiactl
+            type: CharDevice
+        - name: nvidia0
+          hostPath:
+            path: /dev/nvidia0
+            type: CharDevice
+        # Point this at wherever the host driver installed libnvidia-ml.so.1.
+        # Debian/Ubuntu use /usr/lib/x86_64-linux-gnu; other distributions
+        # differ (find it with: ldconfig -p | grep libnvidia-ml).
+        - name: nvml-lib
+          hostPath:
+            path: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
+            type: File

manifests/nvidia-kubevirt-gpu-device-plugin.yaml

@@ -1,3 +1,9 @@
+# This manifest mounts only /dev/vfio and runs unprivileged. It covers every
+# node where each card still exposes its creatable_vgpu_types catalog on at
+# least one function. On a fully consumed node the plugin resolves vGPU type
+# names through NVML instead, which needs the host driver library and the
+# NVIDIA device nodes - use nvidia-kubevirt-gpu-device-plugin-nvml.yaml (see
+# the README section "NVML fallback (fully consumed nodes)") on such nodes.
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:

pkg/device_plugin/device_plugin.go

@@ -89,8 +89,11 @@ var stop = make(chan struct{})
 func InitiateDevicePlugin() {
 	//Identifies GPUs and represents it in appropriate structures
 	createIommuDeviceMap()
-	//Identifies vGPUs and represents it in appropriate structures
+	//Identifies mdev-backed vGPUs and represents it in appropriate structures
 	createVgpuIDMap()
+	//Identifies vGPUs exposed through the vendor-specific VFIO framework
+	//(Ada/Hopper+ GPUs, which do not support mdev)
+	createVfioVGpuMap()
 	//Creates and starts device plugin
 	createDevicePlugins()
 }
@@ -104,6 +107,7 @@ func createDevicePlugins() {
 	log.Printf("Device Map %v", deviceMap)
 	log.Println("vGPU Map ", vGpuMap)
 	log.Println("GPU vGPU Map ", gpuVgpuMap)
+	log.Println("VFIO vGPU Map ", vfioVGpuMap)
 
 	//Iterate over deivceMap to create device plugin for each type of GPU on the host
 	for k, gpuDevices := range deviceMap {
@@ -162,6 +166,33 @@ func createDevicePlugins() {
 			vGpuDevicePlugins = append(vGpuDevicePlugins, dp)
 		}
 	}
+	//Iterate over vfioVGpuMap to create a device plugin for each vGPU profile
+	//exposed through the vendor-specific VFIO framework. These devices are
+	//grouped by profile name (unlike deviceMap, which groups by raw PCI
+	//device id) but otherwise use the same PCI-passthrough-style contract as
+	//deviceMap, since vendor-VFIO VFs are ordinary PCI/IOMMU-group devices.
+	for k, gpuDevices := range vfioVGpuMap {
+		devs = nil
+		for _, gpuDev := range gpuDevices {
+			devs = append(devs, &pluginapi.Device{
+				ID:     gpuDev.addr,
+				Health: pluginapi.Healthy,
+				Topology: &pluginapi.TopologyInfo{
+					Nodes: []*pluginapi.NUMANode{
+						{ID: gpuDev.numaNode},
+					},
+				},
+			})
+		}
+		log.Printf("DP Name %s", k)
+		dp := NewGenericDevicePlugin(k, "/dev/vfio/", devs)
+		err := startDevicePlugin(dp)
+		if err != nil {
+			log.Printf("Error starting %s device plugin: %v", dp.deviceName, err)
+		} else {
+			devicePlugins = append(devicePlugins, dp)
+		}
+	}
 
 	<-stop
 	log.Printf("Shutting down device plugin controller")