[CI][C++][Release] Flaky conda dylib load failure in macOS verify-rc build

[tadeja]

Describe the bug, including details regarding any error messages, version, and platform.

The nightly job Verify release candidate on macOS intermittently aborts the C++ build:

dyld[24719]: Library not loaded: @rpath/libLLVM.22.1.dylib
  Referenced from: .../miniforge/pkgs/llvm-tools-22-22.1.8-hc181bea_1/bin/llvm-ranlib-22

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM.
Not caused by LLVM 22.1.8 (initial incorrect assessment) and also not LLVM-specific.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.

There are two such failing jobs:

• https://github.com/ursacomputing/crossbow/actions/runs/27960474749/job/82740300712 (mimalloc)
• https://github.com/ursacomputing/crossbow/actions/runs/27960469209/job/82740279170 (Gandiva)

Component(s)

C++, Continuous Integration

[kou]

Is this a problem of the LLVM conda package? If so, we need to report this to the LLVM conda package.

[raulcd]

@h-vetinari I see you also maintain the llvmdev-feedstock . Is this a conda issue? I am also not sure what is going on but it seems like a link on macOS libLLVM.22.1.dylib might be missing?

[h-vetinari]

This is unfortunately a long-standing issue that no-one managed to debug thus far. 🥲 (not with llvm specifically; the issue occurs irregularly with basically any shared library on osx that inexplicably doesn't find its dependencies on the rpath even though the libraries are present in the environment)

Usually the incidence is far less than 1-in-10, which so far managed below the threshold where someone feels compelled to spend a couple of days on this.

[tadeja]

Investigated in #50243: the library is present and loadable in the env. The failure is an intermittent rpath miss when the conda llvm-tools binary is invoked from the package cache (@loader_path/../lib -> no libLLVM).

So #50243 works around it by adding $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the build.

[tadeja]

So this flaky build failure is not LLVM-specific.
I've checked available failing crossbow nightly logs of the last 3 months for
verify-rc-source-cpp-macos-conda-amd64,
verify-rc-source-integration-macos-conda-amd64 and
verify-rc-source-python-macos-conda-amd64

The same @rpath miss has aborted macOS build via three conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc):

Date	Library not loaded: @rpath/__	Binary	Build site	Logs
2026-04-21	libprotoc.25.3.0.dylib	grpc_cpp_plugin (libgrpc 1.62.2)	Flight.pb.cc	(log)
2026-05-04	libLLVM.22.1.dylib	llvm-ranlib-22 (llvm-tools-22 22.1.4)	mimalloc libmimalloc.a	(log)
2026-05-30	libabsl_log_initialize.2401.0.0.dylib	protoc-25.3.0 (libprotobuf 4.25.3)	Flight.pb.cc	(log)
2026-06-14	libabsl_log_initialize.2401.0.0.dylib	protoc-25.3.0 (libprotobuf 4.25.3)	Flight.pb.cc	(log)
2026-06-18	libprotoc.25.3.0.dylib	grpc_cpp_plugin (libgrpc 1.62.2)	Flight.pb.cc	(log)
2026-06-20	libLLVM.22.1.dylib	llvm-link-22 (llvm-tools-22 22.1.8)	Gandiva irhelpers.bc	(log)
2026-06-22	libLLVM.22.1.dylib	llvm-link-22 (llvm-tools-22 22.1.8)	Gandiva irhelpers.bc	(log)
2026-06-22	libLLVM.22.1.dylib	llvm-ranlib-22 (llvm-tools-22 22.1.8)	mimalloc libmimalloc.a	(log)

All are build time, matching @h-vetinari's indication that any osx shared library is affected - conda-forge/cmake-feedstock#230.

PR #50243 works around it consumer-side by adding $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the cmake --build step, which should resolve any such conda dependency on @rpath miss.