feat(device_plugin): rediscover vendor-specific VFIO vGPU profiles at runtime#194
Open
lexfrei wants to merge 7 commits into
Open
feat(device_plugin): rediscover vendor-specific VFIO vGPU profiles at runtime#194lexfrei wants to merge 7 commits into
lexfrei wants to merge 7 commits into
Conversation
…work Ada/Hopper+ GPUs (vGPU 17+) no longer expose vGPU instances through mdev; instead, SR-IOV Virtual Functions stay bound to the "nvidia" host driver and are configured directly on the PCI function via sysfs. The existing vGPU discovery only walks /sys/bus/mdev/devices, so it finds nothing on these GPUs and every vGPU VF ends up indistinguishable from a whole-card passthrough device, since both share the same PCI vendor:device id. Add a second discovery path that walks /sys/bus/pci/devices looking for Nvidia functions bound to the "nvidia" driver, skips Physical Functions (identified by sriov_totalvfs), and treats a Virtual Function as a configured vGPU when its current_vgpu_type is non-zero. The profile name is resolved by matching that type id against the creatable_vgpu_types catalog, merged per physical card (scoped by each VF's physfn parent) since a configured VF's own list is frequently empty or reduced to the active entry, and since vGPU type ids are only unique within a single physical card. Because these VFs are ordinary PCI/IOMMU-group devices, discovered functions are added to the same iommu/BDF maps used by GPU passthrough and dispatched through the existing GenericDevicePlugin, grouped by vGPU profile name instead of PCI device id. No changes were needed to the allocate path or device contract; the mdev-backed vGPU discovery is untouched. Document the new hardware path in the README alongside the existing mdev instructions, since it applies to a different GPU generation and neither replaces the other. Signed-off-by: Aleksei Sviridkin <f@lex.la>
A fully consumed card reduces the creatable_vgpu_types list of every one of its functions down to the header, so the per-card catalog cannot resolve the types of already-configured VFs and the plugin advertised nothing exactly when all capacity was allocated. Fall back to a host-wide catalog merged across all cards; numeric ids that different cards map to different names stay ambiguous and are still skipped rather than guessed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
On a host where every card is fully consumed, no creatable_vgpu_types catalog lists anything at all — sysfs alone cannot resolve configured type ids to profile names no matter how catalogs are merged. Resolve through NVML as the last resort: the supported-vGPU list of a card does not shrink as capacity is allocated. Scoped per physical card like the sysfs catalogs, cached per scan, and requires the host driver library to be loadable in the container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
…other cards When a configured VF's card catalog is reduced and cannot resolve its type id, resolution fell back to a host-wide catalog merged across all cards. Numeric vGPU type ids are only unique within one physical card, so if the target card contributed no entry for the id and another card mapped the same id to a different profile, that entry looked unambiguous and the VF could be advertised under the wrong extended resource. Drop the host-wide catalog entirely. Resolve strictly card-local sysfs, then the parent Physical Function's own NVML supported-type list, then skip the VF with a log. NVML is authoritative per card and its supported list does not shrink as capacity is allocated, so it covers the fully consumed card without ever borrowing a sibling card's mapping. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ments The NVML name-resolution fallback needs libnvidia-ml.so.1 and the NVIDIA device nodes. The default DaemonSet mounts only /dev/vfio and runs unprivileged, so on a fully consumed node a restart could resolve no profile names and advertise no VFs. Add a dedicated manifest that provides them via a single-file bind of libnvidia-ml.so.1 plus the NVIDIA device nodes under privileged: true, leaving the default manifest minimal and unprivileged. Document in the README both the minimally privileged runtimeClassName: nvidia path and the hostPath variant, why the library must be bound as a single file rather than the whole host lib directory, and which device nodes the plugin's NVML queries actually require. Also document that vGPU profiles must be configured before the plugin starts, since it scans the Virtual Functions once at startup. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
…U profiles The vendor-specific VFIO vGPU discovery ran only once at startup, so a vGPU profile configured or changed on a Virtual Function after the plugin started was not advertised until the pod restarted. Add opt-in periodic rediscovery, enabled by setting VFIO_VGPU_RESCAN_INTERVAL to a Go duration (disabled when unset, so existing deployments are unchanged; a positive value below five seconds is clamped up). On each tick the plugin re-runs the same scan and reconciles the running device plugins: it starts advertising a newly configured profile, updates the device list of a profile whose set of Virtual Functions changed, and stops advertising a profile once none of its Virtual Functions remain configured. A periodic rescan is used rather than an inotify watch because inotify does not fire reliably for the sysfs attribute writes that (re)configure a vGPU, and SR-IOV Virtual Function creation/removal moves whole device directories. To support this the one-shot scan is split into a pure discovery function and a locked publish step so startup and rediscovery share it, the shared IOMMU maps are snapshotted under a mutex so a rescan can update them while Allocate and the health check read them, and each device plugin gains a non-blocking device-list update that preserves per-VF health and extends the health-check watches to newly added Virtual Functions. Stopping the plugin for a profile that is no longer configured closes a per-plugin done channel so its health-check goroutine and watcher terminate instead of leaking, and the plugin lifecycle is serialized so a concurrent stop and restart cannot race the server field or double-close that channel. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe the VFIO_VGPU_RESCAN_INTERVAL opt-in in the README's vGPU profile timing section and add a commented-out env example to both the default and the NVML-fallback manifests. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
marked this pull request as ready for review
July 11, 2026 10:41
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds opt-in periodic rediscovery of vendor-specific VFIO vGPU profiles, so a profile configured or changed on a Virtual Function after the plugin starts is picked up without restarting the pod. Enabled by setting
VFIO_VGPU_RESCAN_INTERVALto a Go duration (for example30s); disabled when unset, so existing deployments are unchanged.This answers the follow-up on #192 ("document that profiles must be configured before plugin startup, or add rediscovery for later profile creation/change"): #192 added the documentation half, and this PR adds the rediscovery half.
Stacked on #192
This is stacked on #192 (
feat/vendor-vfio-vgpu). Until #192 merges, this PR's diff and commit list include #192's commits; only the last two commits (thefeatanddocscommits) belong to this PR. I will rebase once #192 lands.How it works
On each tick the plugin re-runs the same scan #192 uses and reconciles the running device plugins against the freshly observed profiles:
ListAndWatchloop;The shared IOMMU maps are updated before any change is advertised, so a new VF is allocatable the moment kubelet sees it.
Why a ticker, not fsnotify
A periodic rescan is used rather than an inotify/fsnotify watch on sysfs. inotify does not fire reliably for the sysfs attribute writes that (re)configure a vGPU (
current_vgpu_type,creatable_vgpu_typesare kernel-backed synthetic files, not regular files), and SR-IOV Virtual Function creation/removal moves whole device directories that are awkward to watch correctly. Re-running the exact scan already validated at startup is simpler and more robust. The interval has a 5s floor so a misconfiguration cannot spin sysfs (and NVML on fully consumed cards) in a tight loop.Lifecycle choice
Per-profile add/remove uses the existing per-instance
GenericDevicePluginstart/stop rather than a full re-init of the plugin set, so adding or removing one profile does not churn kubelet registration for the others. Making a single plugin stoppable at runtime (previously plugins were only stopped at process shutdown) required a per-plugindonechannel so its health-check goroutine and fsnotify watcher terminate instead of leaking, and serializing the plugin lifecycle so a concurrent stop and a health-check-driven restart cannot race. Composition changes within a profile carry over per-VF health so an already-unhealthy VF is not reset to healthy.Testing and CI limits
Unit tests cover the reconciliation diff (VF added/removed within a profile, profile appears/disappears, no-op rescan, GPU-passthrough isolation), health carry-over, non-blocking updates, the interval parser contract, the health-check goroutine teardown, and concurrent stop safety.
go build,go vet,gofmt, andgo test ./pkg/device_plugin/...are green; the new tests are race-clean under-race.CI cannot exercise the live path end to end: there is no GPU, no vendor VFIO sysfs, and no way to churn vGPU profiles on the runner. The tests drive the reconciliation and lifecycle logic through injected discovery results and fakes; the sysfs/NVML scan itself is unchanged from #192 and validated there.