feat: Fabric Manager partition-aware GetPreferredAllocation for Shared NVSwitch multi-tenancy

[inerplat]

Fixes #83. Related to #24, which is about the Service-VM flavor and exposing NVSwitch devices; this PR is about GPU allocation in the host-managed FM flavor.

Background

On NVSwitch-based HGX systems running Fabric Manager in the Shared NVSwitch model, the host keeps the NVSwitches and tenant VMs get GPUs by VFIO passthrough. NVLink between a VM's GPUs only works if those GPUs are exactly the members of an activated FM fabric partition. FM programs the switch routing so traffic stays within a partition. The set of supported partitions is fixed per baseboard and differs across platforms, so it has to be read from FM at runtime rather than assumed. On the HGX H100 8-GPU board we tested, FM reports 15 partitions: the full 8, two 4-GPU halves, four 2-GPU pairs, and eight singles. There the 2-GPU pairs happen to be diagonal by physical id, {1,3} {2,4} {5,7} {6,8}, which is exactly the kind of layout a count-based or ordinal allocator gets wrong.

Today the plugin advertises nvidia.com/pgpu as a plain counted resource and GetPreferredAllocation is a stub, so it never tells kubelet which GPU sets form a partition. Kubelet's devicemanager then picks devices without regard to the partition topology, and only a few of the possible combinations are valid partitions (on H100, 4 of the 28 possible 2-GPU sets). Two things go wrong. A multi-GPU pod often gets a set that matches no supported partition, so there is no NVLink between its GPUs. And if the partition covering the assigned GPUs was not activated, the guest driver blocks on NVLink fabric registration until nvidia-persistenced's 120s timeout and the VM fails to boot.

No fixed device-ordering rule fixes this in general. The plugin's device ids follow vfio/BDF enumeration, and the BDF-to-module-id mapping is board-specific and not derivable from that order. On our test box 0000:19:00.0 is physical GPU 2 and 0000:4c:00.0 is physical GPU 1, so which device sets happen to be valid partitions is a property of the board. The allocator has to consult FM and a real BDF-to-module mapping.

Design

The device plugin already knows each GPU by a device id derived from its vfio group / iommufd number. Fabric Manager identifies GPUs by physical (module) id. Partitions are expressed in physical ids. So the allocator needs to translate a device id to a physical id, ask FM which partitions exist, and translate a chosen partition back to device ids. Three pieces:

1. FM client. A thin wrapper over libnvfm exposing one call, GetSupportedPartitions() []Partition, where Partition{ID, IsActive, GPUPhysicalIDs}. It connects lazily and drops the handle on any API error so the next call reconnects, so FM bouncing does not wedge the plugin. Built behind the nvfm tag, with a stub compiled otherwise (see Build).

2. BDF-to-module-id mapping. FM speaks module ids, the plugin speaks device ids that map to a BDF, and the two orderings differ, so a mapping is required. Loaded from a JSON file, { "<pci-bdf>": <module-id> }, path /run/nvidia-fabricmanager/gpu-pci-module-mapping.json by default. Both integer and string module-id values are accepted and BDFs are normalized to the sysfs 0000:bb:dd.f form. This is the same file convention the kubevirt plugin uses.

3. Selection. For each container request of size N, SelectPreferred(available, mustInclude, N):

• translate available device ids to physical ids through the mapping, dropping any that don't map (logged);
• keep supported partitions whose numGpus == N, whose physical ids are all in the available set, and that contain every mustInclude device;
• among those, prefer partitions that are already active, breaking ties by lowest partition id for determinism;
• translate the chosen partition's physical ids back to device ids and return them.

If no partition fits, it returns nothing. Active-first is the important bit: partition activation is the operator's steering wheel, so pre-activating the partitions of the layout you intend to run makes the plugin route tenants onto exactly those GPUs, with no extra scheduler component.

Failure handling

GetPreferredAllocation is advisory in the kubelet contract. A preferred set is a hint, and returning none just lets the devicemanager fall back to its normal allocation. Every failure mode here takes that path: FM unreachable, a device missing from the mapping, or no partition of the requested size available all resolve to "no preference". The feature can log a problem but it can never fail a pod's allocation, so enabling it cannot regress the plain counted-resource behavior. GetDevicePluginOptions only advertises GetPreferredAllocationAvailable when the manager initialized, and only for the GPU resource, not NVSwitch.

What this PR does

• adds pkg/fabric_manager: FM client (nvfm_client_linux.go, tag nvfm) + stub (nvfm_client_stub.go), mapping loader (mapping.go), and the selection logic (partition_manager.go);
• implements GetPreferredAllocation and the capability advertisement in pkg/device_plugin, and wires the mapping/manager in cmd/main.go;
• makes it opt-in with ENABLE_FABRIC_MANAGER=true (off by default; with it unset the plugin is unchanged). A binary built without -tags=nvfm refuses to enable FM even when the env var is set, rather than advertising a capability that would always fail. FM_ADDRESS, FM_ADDRESS_TYPE, and GPU_PCI_MODULE_MAPPING_PATH cover the deployment knobs;
• documents the feature and the deployment requirements in the README.

Tests: partition_manager_test.go drives SelectPreferred against the real H100 partition table with a fake FM client: size-2 requests pick a diagonal pair and never an adjacent {1,2}-style pair; must-include steers the choice; a partly-consumed node still finds a fully available partition; active partitions win and ties break by id; unmappable devices and FM errors both degrade to no preference. mapping_test.go covers integer/string values, domain normalization, and malformed input. Device-plugin tests cover the capability gating and the fall-back-on-error path.

The approach follows the same direction as the kubevirt-gpu-device-plugin work, adapted to this plugin's vfio/iommufd device model:

• feat: Add Fabric Manager partition support for NVLink-enabled multi-GPU VMs kubevirt-gpu-device-plugin#158
• Fabric Manager shared-nvswitch virtualization model support kubevirt-gpu-device-plugin#166

This PR is scoped to the allocation logic. Container image build, deployment manifests, and partition-activation lifecycle are out of scope; see "Out of scope" below and the README To Do.

Build

The libnvfm client is behind the nvfm build tag and needs the nvidia-fabric-manager-devel package:

CGO_ENABLED=1 go build -tags=nvfm ./...

The default build compiles a stub instead, so go build ./..., go vet ./... and go test ./... work with no FM SDK present. The CGO connection parameters follow the FM SDK sample (inet or unix address, non-zero timeout, versioned request structs).

How it was tested

Built with -tags=nvfm and run on an HGX H100 8-GPU box: driver 580.126.20, kernel 6.17, Kata 3.32 with the kata-qemu-nvidia-gpu runtime classes, GPU Operator v26 sandbox workloads, k8s 1.35, host-managed FM in FABRIC_MODE=1.

Four 2-GPU kata pods at once, with the four 2-GPU partitions pre-activated: each landed on a proper diagonal pair and all four guests showed NV18. The plugin log from that run, where the available pool shrinks as pairs get taken and the device ids are non-ordinal:

[pgpu] container request 0: available=[7 0 1 2 3 4 5 6] mustInclude=[] size=2
fabric_manager: selected partition 3 (active: true, GPUs: [1 3]) from 4 candidate(s)
[pgpu] preferred devices for container request 0: [2 3]
[pgpu] container request 0: available=[6 7 0 1 4 5] mustInclude=[] size=2
fabric_manager: selected partition 4 (active: true, GPUs: [2 4]) from 3 candidate(s)
[pgpu] preferred devices for container request 0: [0 1]
[pgpu] container request 0: available=[5 6 7 4] ... selected partition 5 (GPUs [5 7]) -> [5 4]
[pgpu] container request 0: available=[6 7]     ... selected partition 6 (GPUs [6 8]) -> [6 7]

A mixed 1+1+2+4 layout in one shot also worked: a 4-GPU pod with a full NV18 mesh (devices [2 0 3 1], the mapping resolving a non-ordinal partition), a 2-GPU pod with an NV18 pair, and two 1-GPU pods, all correct at the same time.

For interference, a 4-GPU tenant and a 2-GPU tenant ran sustained cudaMemcpyPeer sweeps through the same physical switches simultaneously. Both held a flat ~370 GB/s for every sample across the whole overlap window, so partitions don't take bandwidth from each other.

With ENABLE_FABRIC_MANAGER unset the plugin behaves as before, and killing FM while pods schedule just loses the preference without failing allocations.

Out of scope

• Building the container image with libnvfm on all target arches. The FM SDK repo is x86_64, so the image and CI changes for that are a follow-up (noted in the README To Do).
• Generating the mapping file. On vfio-bound hosts nvidia-smi cannot see the GPUs; what worked for us was booting one 1-GPU kata pod per GPU and reading Module ID from nvidia-smi -q in the guest, matched to the QEMU host=<BDF> argument. Having the driver container emit the file would be a good follow-up.
• Partition activate/deactivate lifecycle stays out-of-band. Automatic activation on Allocate would pair well with active-first selection but is left out to keep this change small.
• Rack-scale GB200 NVL72, which is NMX-managed with multi-node partitions, is a different model. The FMClient interface is the seam where an NMX-backed implementation could slot in.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.