| Documentation |
XLLM OPS is a scalable and high-performance operator library designed for large language models.
With the widespread application of large language models, developers face challenges such as low computational efficiency and high resource consumption during model training and inference. These issues have created an urgent need for an efficient operator library to enhance performance and reduce costs.
To address this, we designed XLLM OPS, a domestic chip operator library focused on performance optimization, aiming to provide faster computation speeds and lower resource consumption.
- Compile the operator library
bash build.shcd test
export VCPKG_ROOT=/path/to/vcpkg
bash build.sh
./bin/group_gemm_gtest
- The optimized GroupMatmul operator shows significant advantages in computation time, especially when k=128 and m=64. As shown in the figure, the optimized operator's time consumption is reduced by 50%.
- After using the topKtopP fusion operator, in the qwen2-0.5B model, TTOT decreased by 37%, and TTFT improved by 10%.
If you encounter any issues along the way, you are welcomed to submit reproducible steps and log snippets in the project's Issues area, or contact the xLLM Core team directly via your internal Slack. In addition, we have established official WeChat groups. You can access the following QR code to join. Welcome to contact us!
This project was made possible thanks to the following open-source projects:
- cann-ops-dev - Adopted the engineering construction in cann-ops-adv