[BUG]: nvrtc-builtins failed to load on windows and system install on linux

[ZzEeKkAa]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.pathfinder

Describe the bug

Failed to load nvrtc-builtins from system install on windows. On linux works fine. nvrtc works so:

(venv) PS C:\Users\yhavrylko> pip install cuda-pathfinder
Collecting cuda-pathfinder
  Downloading cuda_pathfinder-1.1.0-py3-none-any.whl.metadata (2.3 kB)
Downloading cuda_pathfinder-1.1.0-py3-none-any.whl (17 kB)
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.1.0
(venv) PS C:\Users\yhavrylko> python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc'))"
LoadedDL(abs_path='C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvrtc64_120_0.dll', was_already_loaded_from_elsewhere=False, _handle_uint=140730218643456)
(venv) PS C:\Users\yhavrylko> python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))
                                             ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
  File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\load_nvidia_dynamic_lib.py", line 44, in _load_lib_no_cache
    found.raise_if_abs_path_is_None()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\find_nvidia_dynamic_lib.py", line 194, in raise_if_abs_path_is_None
    raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "nvrtc-builtins*.dll": No such file: nvrtc-builtins*.dll, No such file: nvrtc-builtins*.dll
  listdir("C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin"):
    __nvcc_device_query.exe
    bin2c.exe
    compute-sanitizer.bat
    crt
    cu++filt.exe
    cublas64_12.dll
    cublasLt64_12.dll
    cudafe++.exe
    cudart64_12.dll
    cufft64_11.dll
    cufftw64_11.dll
    cuinj64_129.dll
    cuobjdump.exe
    curand64_10.dll
    cusolver64_11.dll
    cusolverMg64_11.dll
    cusparse64_12.dll
    fatbinary.exe
    nppc64_12.dll
    nppial64_12.dll
    nppicc64_12.dll
    nppidei64_12.dll
    nppif64_12.dll
    nppig64_12.dll
    nppim64_12.dll
    nppist64_12.dll
    nppisu64_12.dll
    nppitc64_12.dll
    npps64_12.dll
    nvJitLink_120_0.dll
    nvblas64_12.dll
    nvcc.exe
    nvcc.profile
    nvdisasm.exe
    nvfatbin_120_0.dll
    nvjpeg64_12.dll
    nvlink.exe
    nvprof.exe
    nvprune.exe
    nvrtc-builtins64_129.dll
    nvrtc64_120_0.alt.dll
    nvrtc64_120_0.dll
    nvvp.bat
    ptxas.exe

How to Reproduce

0. Clean Windows install with latest nvidia driver, vs-redist C++ build tools 2022, ctk 12.9.1.
1. Create python virtual env and activate it
2. pip install cuda-pathfinder
3. python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"

Expected behavior

Same as on linux, where it manages to find the library.

Operating System

Windows 11 24H

nvidia-smi output

n/a

Update

On linux it also fails to find sysctk install:

python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 40, in _load_lib_no_cache
    loaded = load_with_system_search(libname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 104, in load_with_system_search
    abs_path = abs_path_for_dynamic_library(libname, handle)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 47, in abs_path_for_dynamic_library
    for symbol_name in EXPECTED_LIB_SYMBOLS[libname]:
                       ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'nvrtc-builtins'

( most likely related to #833 )

However it is able to find with nvidia-cuda-nvrtc-cu12:

python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
LoadedDL(abs_path='/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv/lib/python3.11/site-packages/nvidia/cuda_nvrtc/lib/libnvrtc-builtins.so.12.9', was_already_loaded_from_elsewhere=False, _handle_uint=94212168521552)

On windows it still does not work with pip install nvidia-cuda-nvrtc-cu12. (nvrtc works so)

[ZzEeKkAa]

There is nvrtc-builtins64_129.dll in the list. Could it be because suffix is 129, not 12?

[leofang]

Sorry, but why is this a bug? I already said that load_nvidia_dynamic_lib('nvrtc-builtins') is not needed (and in fact is not meant to be supported by the pathfinder). We just need load_nvidia_dynamic_lib('nvrtc') and that's all!

Look at the pathfinder output, it does know where it is

cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "nvrtc-builtins*.dll": No such file: nvrtc-builtins*.dll, No such file: nvrtc-builtins*.dll
  listdir("C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin"):
    (...)
    nvrtc-builtins64_129.dll
    nvrtc64_120_0.alt.dll
    nvrtc64_120_0.dll
    nvvp.bat
    ptxas.exe

and it's NVRTC's job to load this dll when JIT compilation starts. The NVRTC location has been added to the search path after the call load_nvidia_dynamic_lib('nvrtc'), as described in the linked comment.

@ZzEeKkAa could you provide a better reproducer to the true failure?

[rwgk]

@leofang, @ZzEeKkAa and I discussed this offline, we agree.

[leofang]

Thanks guys, let's close this for now.