Multi-GPU pods get GPUs that don't match any FM fabric partition (no NVLink in Shared NVSwitch mode)
I'm running Kata pods on an HGX H100 8-GPU box with Fabric Manager in Shared NVSwitch mode (FABRIC_MODE=1 on the host, GPUs on vfio-pci, plugin v0.0.3, kata 3.32, driver 580).
In this mode NVLink only works between GPUs of the same activated fabric partition. The supported partition set is fixed per baseboard and differs across platforms. On the HGX H100 box I'm on, FM reports the 2-GPU partitions as {1,3} {2,4} {5,7} {6,8} by physical id, so only 4 of the 28 possible 2-GPU combinations are valid. Since GetPreferredAllocation is a TODO stub, the plugin advertises nvidia.com/pgpu as a plain count and never tells kubelet which GPU sets form a partition. Kubelet's devicemanager then picks devices without regard to that, so a multi-GPU pod often comes up on a set that isn't a partition and has no NV18 between its GPUs. And if the partition covering the assigned GPUs isn't active, the guest driver blocks on fabric registration (nvidia-persistenced, 120s) and the VM fails to boot.
No fixed device-ordering rule fixes this in general, because the BDF-to-module-id mapping is board-specific and not derivable from device order. On my node vfio device 0 is physical GPU 2 (0000:19:00.0 = module id 2) and device 2 is physical GPU 1, so which device pairs happen to be valid partitions is a property of this board, not something an allocator can assume. It has to ask FM.
Repro: activate the four 2-GPU partitions, create several pgpu: 2 kata pods, and check nvidia-smi topo -m in the guests. Since the allocation isn't partition-aware, you'll usually find pods placed on GPU pairs that aren't a partition and show no NV18.
I have a working implementation of GetPreferredAllocation that queries fmGetSupportedFabricPartitions via libnvfm and prefers device sets matching an active partition of the requested size, following the same direction as the kubevirt-gpu-device-plugin work:
Tested on the box above: four concurrent 2-GPU pods each get a proper NVLink pair, and mixed 1+1+2+4 works too. I'll open a PR. It's related to #24, but this one is about the host-managed FM flavor and GPU allocation specifically.
Multi-GPU pods get GPUs that don't match any FM fabric partition (no NVLink in Shared NVSwitch mode)
I'm running Kata pods on an HGX H100 8-GPU box with Fabric Manager in Shared NVSwitch mode (FABRIC_MODE=1 on the host, GPUs on vfio-pci, plugin v0.0.3, kata 3.32, driver 580).
In this mode NVLink only works between GPUs of the same activated fabric partition. The supported partition set is fixed per baseboard and differs across platforms. On the HGX H100 box I'm on, FM reports the 2-GPU partitions as {1,3} {2,4} {5,7} {6,8} by physical id, so only 4 of the 28 possible 2-GPU combinations are valid. Since
GetPreferredAllocationis a TODO stub, the plugin advertisesnvidia.com/pgpuas a plain count and never tells kubelet which GPU sets form a partition. Kubelet's devicemanager then picks devices without regard to that, so a multi-GPU pod often comes up on a set that isn't a partition and has no NV18 between its GPUs. And if the partition covering the assigned GPUs isn't active, the guest driver blocks on fabric registration (nvidia-persistenced, 120s) and the VM fails to boot.No fixed device-ordering rule fixes this in general, because the BDF-to-module-id mapping is board-specific and not derivable from device order. On my node vfio device 0 is physical GPU 2 (0000:19:00.0 = module id 2) and device 2 is physical GPU 1, so which device pairs happen to be valid partitions is a property of this board, not something an allocator can assume. It has to ask FM.
Repro: activate the four 2-GPU partitions, create several
pgpu: 2kata pods, and checknvidia-smi topo -min the guests. Since the allocation isn't partition-aware, you'll usually find pods placed on GPU pairs that aren't a partition and show no NV18.I have a working implementation of
GetPreferredAllocationthat queriesfmGetSupportedFabricPartitionsvia libnvfm and prefers device sets matching an active partition of the requested size, following the same direction as the kubevirt-gpu-device-plugin work:Tested on the box above: four concurrent 2-GPU pods each get a proper NVLink pair, and mixed 1+1+2+4 works too. I'll open a PR. It's related to #24, but this one is about the host-managed FM flavor and GPU allocation specifically.