Helm chart: add volumeMounts for nvidiaDriverRoot + custom env. for NFD pod

[plevart]

This is a proposed fix for #1677

With this patch, I can specify something like the following in the Helm chart custom values.yaml:

deviceListStrategy: cdi-cri
nvidiaDriverRoot: "/"
gfd:
  enabled: true
  env:
    # when deviceListStrategy=cdi-cri, there is no runc wrapper to inject the library
    - name: LD_PRELOAD
      value: /driver-root/usr/lib64/libnvidia-ml.so.1

...and the gfd Pod is happy.

I also cleaned-up and aligned the logic of this part of template with device plugin template.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[elezar]

Although it may be useful to set the envvars for the GFD pod, I would argue that using this to set the location of libnvidia-ml.so.1 is not the correct mechanism. We already support loading a SPECIFIC path to libnvidia-ml.so.1 and the initialisation of the library here

k8s-device-plugin/cmd/gpu-feature-discovery/main.go

Line 182 in 51f7ece

nvmllib := nvml.New()

should be updated.

See for example how this is done in the DRA driver here https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/ceb2f34a763def3531062efef01593f40861258e/cmd/gpu-kubelet-plugin/nvlib.go#L54-L69

Note that in the case of the GPU operator, the Device Plugin and GFD are consisdered "management" containers and are explicitly run using a specific runtime class to ensure that they have access to the requried devices and libraries. In cases where native CDI is used, we have an NRI plugin that ensures that these pods have GPU access based on pod annotations.