Best way to serve a 4bit model with qlora adapters in TRT

[lodm94]

Hey there!
My goal is to deploy a Mixtral-7x8B with couple of QLoRA fine-tuned adapters using TRT.
The corollary is porting a dev environment made with bitsandbytes and hugging face to Triton Server.

This is definetely my situation.

I wonder if this apply to a Mixtral-7x8 as well as llama.

According to this line, it should not apply. In this case, support to Mixtral-7x8 is in roadmap?

Still, according to this section, it seems you can quantize a Mistral with default settings from quantize.py, that is gptnext. Is this correct? Does it apply to Mixtral-7x8 as well?

Any further int on the topic is welcome!

Cheers

[Advait251206]

Based on the documentation around that time, there are two separate questions here:

1. Can Mixtral be quantized?
2. Can Mixtral be served with QLoRA adapters?

These are related but not the same feature.

From my understanding:

• Yes, Mixtral models can be quantized using the TensorRT-LLM quantization workflow (e.g., via quantize.py with the appropriate model configuration), assuming the model architecture is supported. The gptnext configuration is a generic decoder-only configuration used for several GPT-like architectures, so seeing Mistral support there does not necessarily imply that every Mistral-derived model (such as Mixtral) has identical support for all downstream features.

• QLoRA adapter support is a separate capability. Even if the base Mixtral model can be quantized and built successfully, that does not automatically mean LoRA/QLoRA runtime adapter support is available for Mixtral. Historically, LoRA support has been introduced on a per-architecture basis, so support for Llama does not necessarily extend to Mixtral.

Because Mixtral is a Mixture-of-Experts (MoE) architecture, applying LoRA adapters is also more involved than for a standard decoder-only transformer, as there are additional considerations around expert layers and routing.

So, if the documentation explicitly excludes Mixtral from LoRA support, I would interpret that as:

• ✅ Base Mixtral inference may be supported.
• ✅ Quantized Mixtral inference may also be supported.
• ❌ Runtime LoRA/QLoRA adapter support for Mixtral is not necessarily supported.

One question for the TensorRT-LLM team:

• Is runtime LoRA/QLoRA support for Mixtral (e.g., Mixtral-8x7B) on the roadmap, or is the current recommendation to merge the adapters into the base model prior to conversion? Clarifying whether this is a current limitation or a planned feature would be helpful for users migrating Hugging Face + bitsandbytes workflows to TensorRT-LLM/Triton.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206