CLIP inference using TensorRT

[dhaivatd]

I want to run CLIP vision and text model inference usning TensorRT on NVIDIA GPU on linux environment. I don't see many successful attempts on the web.

Is there a support for CLIP on TensorRT-LLM?

[Advait251206]

TensorRT-LLM is primarily designed for autoregressive transformer models (LLMs), so there isn't dedicated support for standalone CLIP inference in the same way there is for models like Llama, Mistral, or Qwen.

That said, CLIP itself consists of two transformer-based encoders (a vision encoder and a text encoder), so there are a couple of options:

• Standard TensorRT is generally the recommended choice if your goal is to accelerate standalone CLIP image/text embedding inference. You can export the vision and text encoders (e.g., via ONNX) and build TensorRT engines for each.

• TensorRT-LLM may come into play if CLIP is being used as part of a larger multimodal pipeline (for example, alongside an LLM). In those cases, the vision encoder may be accelerated separately, while TensorRT-LLM handles the language model portion of the inference pipeline. However, this is different from native support for standalone CLIP inference.

So if your objective is simply to generate image and text embeddings with CLIP, I would recommend using TensorRT rather than TensorRT-LLM.

One question for the TensorRT-LLM team:

• Are there any plans to officially support standalone encoder-only transformer models like CLIP (vision/text encoders) within TensorRT-LLM, or is the intended approach to continue using standard TensorRT for encoder inference and reserve TensorRT-LLM for autoregressive language models?

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206