new feature: Explore if OpenDAL can support KvikIO (aka Nvidia GPUDirect Storage)

[Xuanwo]

Feature Description

GPUDirect Storage

Nvidia GPUDirect Storage is a technology introduced by Nvidia that enables GPUs to read files directly.

With GPUDirect Storage:

• Higher Bandwidth: GDS offers 2x-8x higher bandwidth by enabling direct data transfers between storage and GPU, bypassing CPU involvement.
• Lower Latency: Explicit and direct transfers reduce latency by 3.8x and ensure stable latency even as GPU concurrency increases.
• Reduced CPU Load: DMA engines near storage reduce CPU load, maintaining higher bandwidth with minimal CPU utilization, and GPU utilization remains near zero during data transfers.
• Superior IO Bandwidth: GPUs achieve higher IO bandwidth (215 GB/s) compared to CPUs (50 GB/s).
• Scalable Storage Access: GDS allows fast access to large data sets, whether stored locally or remotely, nearly saturating GPU bandwidth by combining data transfers from various sources (CPU memory, local storage, remote storage).
• Efficient Data Caching: Large distributed data sets can be cached in local storage, and working tables can be cached in CPU memory for collaborative use with the CPU, enhancing overall system performance.

cuFile

Nvidia exposes GDS support via cuFile which included in CUDA toolkit.

KvikIO

KvikIO is a High Performance File IO lib.

KvikIO is a Python and C++ library for high performance file IO. It provides C++ and Python bindings to cuFile, which enables GPUDirect Storage (GDS). KvikIO also works efficiently when GDS isn't available and can read/write both host and device data seamlessly. The C++ library is header-only making it easy to include in existing projects.

Problem and Solution

Perhaps OpenDAL could implement a service based on kvikio, allowing our users to interact with GDS using the familiar OpenDAL API.

kvikio currently offers only C++ and Python APIs. Those interested should first create a Rust binding for kvikio. To explore the performance of GDS, one should also have the CUDA toolkit and a compatible graphics card.

Additional Context

No response

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[yuchanns]

Interesting! What a shame I don't have a Nvidia GPU.

[Xuanwo]

Interesting! What a shame I don't have a Nvidia GPU.

AMD supports similar technology called DirectGMA, but I'm not sure if it can read files.

[morristai]

This is definitely an ambitious idea, and I would like to explore it further. However, after some research, I found that even if we create a Rust binding for KvikIO, it might not be sufficient.

The reason is that to utilize GDS, we need to use the CUDA API to manage GPU memory. Unfortunately, there is not yet a mature Rust library for CUDA. In KvikIO’s examples, to create a GPU memory buffer for KvikIO to read/write, they use APIs from cuda_runtime_api.h in C++ and CuPy in Python. These APIs are fully supported by Nvidia, and creating a Rust binding for them would be challenging.

If we use KvikIO without CUDA, as demonstrated in the example in the KvikIO repo (cpp/examples/basic_no_cuda.cpp), it won't use GPU memory at all. See the discussion in the issue for more details.

We could still try some third-party libraries like cudarc; maybe it will be sufficient for our use case. Perhaps others might have better ideas?

[morristai]

Followed up on this issue — after some testing, I found that GDS over TCP/IP isn’t supported yet. I tested with both a remote S3 bucket and a local MinIO cluster. Using the latest cudf-cu12 25.6.0 to read a remote S3 Parquet file (~1 GiB), it first reads the data into host RAM, and only after streaming the entire Parquet file does it copy the data into GPU VRAM. This means that even if we create a Kvikio Rust FFI binding, GDS for S3 services won’t work if the underlying library doesn’t support GDS over TCP/IP. However, fs should work though.

By the way, MinIO is working on their proprietary solution to support GDS: https://youtu.be/YvKP6kBW4YY?si=X5jb0nTlUUYoYaib&t=457. So I guess S3 and open-source MinIO should support GDS in the near future.

file_url = "s3://morris-testing-s3/00000-0-c5b97e11-c15c-4dcc-bab8-4f227a34a669-0-00001.parquet"
df = cudf.read_parquet(file_url)

gds-test.mp4

[weiji14]

Did you try with CUDF_KVIKIO_REMOTE_IO=ON (available since cudf=24.12)? From reading https://developer.nvidia.com/blog/high-performance-remote-io-with-nvidia-kvikio/, kvikIO should support Remote IO from s3 already, though GDS can be tricky to set up...

We could still try some third-party libraries like cudarc; maybe it will be sufficient for our use case. Perhaps others might have better ideas?

FYI, someone's trying to add cuFile bindings to cudarc - chelsea0x3b/cudarc#403

[morristai]

Did you try with CUDF_KVIKIO_REMOTE_IO=ON (available since cudf=24.12)? From reading https://developer.nvidia.com/blog/high-performance-remote-io-with-nvidia-kvikio/, kvikIO should support Remote IO from s3 already, though GDS can be tricky to set up...

@weiji14 It works! The setup process is very straightforward. (Although it keeps looking for /etc/pki/tls/certs/ca-bundle.crt even if you set CURL_CA_BUNDLE) But everything works like a charm. Thank you!

[weiji14]

Awesome, glad that it worked 😄

Although it keeps looking for /etc/pki/tls/certs/ca-bundle.crt even if you set CURL_CA_BUNDLE

Maybe related to this issue - rapidsai/kvikio#711? It seems like @TomAugspurger had a fix for this in TomAugspurger/kvikio@73842e2.

However, after some research, I found that even if we create a Rust binding for KvikIO, it might not be sufficient.

The reason is that to utilize GDS, we need to use the CUDA API to manage GPU memory. Unfortunately, there is not yet a mature Rust library for CUDA.

Just to somewhat go back to the core topic of "how to create Rust bindings to GDS", I'm wondering if we want to wrap just the low-level cuFile (which cudarc is getting to at chelsea0x3b/cudarc#403), or the higher-level kvikIO (which wraps cuFile, but also has a non-CUDA compatibility mode for CPU fallback). I've been reading rapidsai/kvikio#686 (comment), and my impression is that kvikIO's cuFile (Python & C++?) bindings will eventually be moved out of kvikIO. Not sure how that would happen in practice, but maybe best to exercise caution on which part of kvikIO's API to wrap in Rust until then.

[Xuanwo]

Thank you @weiji14 for the help! It seems we can work on cudarc first? What do you think @morristai ?

[vyasr]

Hi there, I'm one of the kvikio developers. There's a lot in flux as we improve our tooling here, and you're right to note that things are being moved into places that we expect to be better long-term homes. I don't know much about the Rust side of things, but we expect that in the long run the home for C++ cufile bindings will be in CCCL, see NVIDIA/cccl#6149. Meanwhile, Python bindings are moving into cuda-python, see NVIDIA/cuda-python#684. Depending on the timeline that you're looking to add these features, those are the libraries I would point you to.

[weiji14]

Thanks for the info @vyasr, very useful! The Rust cuFile bindings are available in cudarc>=0.17.1, and opendal can probably just rely on cudarc keeping those bindings updated if they move to CCCL. Not sure where that leaves kvikIO though, what will remain in the API in the long term?

[vyasr]

The main value-add that kvikio provides on top of lower-level cuFile APIs is that it detects GDS availability at runtime and provides transparent fallback to POSIX-based I/O when GDS is not available that uses multithreaded host reads along with coordinated H2D data transfers. GDS remains a bit of a bear to set up on many machines, so having the ability to run on all machines is useful. The core cufile bindings in both Python and C++ will wind up outside of kvikio, but we'll likely also retain some higher-level wrappers around those to provide a better UX. For example, in Python we'll probably try to offer something Pythonic that looks like standard Python io file types, whereas cuda-python will (at least initially) be focused more on providing raw cuFile Python bindings.