is it possible to use quantize.py on encoder models like BERT ?

[mistborn17]

is it possible to use quantize.py on encoder models like BERT ?

In the script during export, it asks for the decoder_type parameter.

[Advait251206]

Short answer: not directly, and not in the intended way.

The quantize.py script in TensorRT-LLM is primarily designed around decoder-only autoregressive transformer models (GPT-style / LLaMA-style). That’s why it requires a decoder_type parameter during export.

Why BERT doesn't fit this pipeline

BERT is an encoder-only model, and it differs in a few fundamental ways:

• No autoregressive decoding loop
• No KV cache
• No incremental token generation
• Full-sequence bidirectional attention instead of causal attention
• No “decoder stack” abstraction that TensorRT-LLM expects

Because of this, the quantization + engine-building flow in quantize.py is tightly coupled to decoder assumptions (especially around:

• past-key-values
• generation step loops
• sampling / beam search kernels)

So encoder models like BERT don’t map cleanly into that pipeline.

---

What does work for BERT?

For BERT-style encoder models, the recommended approach is:

• Use TensorRT directly (not TensorRT-LLM)
• Export to ONNX
• Use:
  • FP16 optimization, or
  • INT8 calibration via TensorRT (PTQ), or
  • Q/DQ quantization (where applicable)

This gives you:

• faster inference
• reduced memory footprint
• supported encoder execution graph

---

Why decoder_type exists in quantize.py

That argument is used to select among supported decoder architectures, such as:

• LLaMA-like
• GPT-like
• Mistral-like variants

It configures:

• attention kernels
• KV cache layout
• layer structure assumptions
• runtime generation logic

None of these are relevant for BERT.

---

Practical takeaway

Model type	Recommended tool
GPT / LLaMA / Mistral	TensorRT-LLM (quantize.py)
BERT / RoBERTa / encoder models	TensorRT (ONNX → engine)
CLIP encoders	TensorRT

---

Bottom line

You can’t meaningfully use quantize.py for BERT in TensorRT-LLM without significant modifications, because the script is not designed for encoder-only architectures.

---

If you're trying to deploy BERT specifically, the best path is:

ONNX export → TensorRT engine build → optional INT8 calibration

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206