Build images for Flatcar

[robinschneider]

This PR adds support for Flatcar for the Nvidia GPU operator.
It features both pre-compiled images as well as a built container to build the driver image at runtime for arbitrary Flatcar releases.
Both image versions provide support for the fabric manager, required for multi GPU nodes such as HGX H100 with 4 or 8 GPUs on a single board.
All images are built using GItHub Actions and published to ghcr.io.

Fixes flatcar/Flatcar#1928 flatcar/Flatcar#1962

Testing

build container (driver compiled at runtime)

Deploy the GPU operator with the base image tag (the operator appends the
-flatcar<VERSION_ID> suffix derived from the node labels):

export BASE_TAG=flatcar-test-dba25cf1-595.71.05-flatcar
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.repository=ghcr.io/robinschneider \
  --set driver.image=driver \
  --set driver.version="${BASE_TAG}" \
  --set driver.imagePullPolicy=Always \
  --set toolkit.installDir=/var/lib/nvidia

The driver container downloads the Flatcar kernel sources and compiles the kernel
modules on the node. This is visible in the driver pod logs (takes a few minutes):

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
Checking NVIDIA driver packages...
Installing the Flatcar kernel sources into the development environment...
...
Building kernel modules:
  [##############################] 100%
Loading NVIDIA driver kernel modules...
Done, now waiting for signal

pre-compiled image

The pre-compiled image ships kernel modules built for one specific Flatcar release
(tag suffix flatcar<VERSION_ID>, e.g. flatcar4593.2.0 / kernel 6.12.81-flatcar).
Deploy it by using the version prefix without the -flatcar suffix:

export BASE_TAG=flatcar-test-dba25cf1-595.71.05
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.repository=ghcr.io/robinschneider \
  --set driver.image=driver \
  --set driver.version="${BASE_TAG}" \
  --set driver.imagePullPolicy=Always \
  --set toolkit.installDir=/var/lib/nvidia

That the pre-compiled path is actually used (no download, no compilation) can be
verified in the driver pod logs — the packaged kernel interface is matched against
the running kernel's /proc/version and installed directly, the pod becomes Ready
within seconds:

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
Checking NVIDIA driver packages...
kernel interface matches.
Found NVIDIA driver package nvidia-modules-6.12.81
Installing NVIDIA driver kernel modules...
Loading NVIDIA driver kernel modules...
Done, now waiting for signal

If the node runs a Flatcar release that does not match the image, the container
falls back to building the modules at runtime (same behavior as the build container).

fabric manager (NVSwitch systems, e.g. HGX H100 8-GPU)

On NVSwitch based nodes the driver container detects the switch devices
(/proc/driver/nvidia-nvswitch/devices) and starts the NVIDIA Fabric Manager
automatically — with both image types:

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
...
Starting NVIDIA persistence daemon...
Starting NVIDIA fabric manager daemon...
Done, now waiting for signal

A successfully initialized fabric is reported by nvidia-smi:

$ kubectl -n gpu-operator exec ds/nvidia-driver-daemonset -- nvidia-smi -q | grep -A2 "^    Fabric"
    Fabric
        State                                          : Completed
        Status                                         : Success

Verified on an HGX H100 8-GPU node (Flatcar 4593.2.0): all 8 GPUs visible via
nvidia-smi -L, CUDA validator pods complete, cuda-vectoradd sample passes.
On nodes without NVSwitch (e.g. L40S) the fabric manager start is skipped.

Example workload

Applies to both image types — schedule a CUDA sample on a GPU node:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: Never
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

$ kubectl logs cuda-vectoradd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

[chewi]

Looks good! I haven't gone over it too closely, as I've seen it basically work before, and I know most of it is based on NVIDIA's code anyway.

One thought though. I don't think any of this depends on anything outside the flatcar directory. Maintenance would be far simpler if we just put these files at the top-level of an empty repo. What do you think?

[chewi]

Didn't you write this? Or AI did? Or it's heavily based on theirs?

[robinschneider]

It is not heavily based on theirs, I am unsure if we should drop the license header or not, other files I introduced have no license header.
I changed it to Flatcar Maintainers here.
Not sure how to deal with that.

[chewi]

If you're unsure, it's common to include both.

[chewi]

Don't we want a less ambiguous image name?

[robinschneider]

I am aware, but that way the CI works for my fork as well to test builds, but we can change it.

[chewi]

Ooooooh. 😮

[robinschneider]

One thought though. I don't think any of this depends on anything outside the flatcar directory. Maintenance would be far simpler if we just put these files at the top-level of an empty repo. What do you think?

That way we should also be able to move it to the flatcar org, right?
@t-lo This is not what we initially planned, what do you think?

[robinschneider]

One thought though. I don't think any of this depends on anything outside the flatcar directory. Maintenance would be far simpler if we just put these files at the top-level of an empty repo. What do you think?

I like the idea, then we could also host it in the flatcar org instead.
@chewi What do you think, should we keep the name of the repository (maybe for users searching for it) or do we want to rename it as well`

[chewi]

I guess keeping the name would help users. It is unlikely we would need anything like this for any other GPU.

[robinschneider]

@chewi @t-lo I created a new repo here and created a PR there: flatcar/gpu-driver-container#1