Quantize llama3 70B on 3x 4090

[teis-e]

I right now run inference thru Transformers with on the fly 4 bit quantization.

Can i create an engine with 4 bit quantization without having to fit the whole unquantized model on the gpu's?

I already have the 8b model running with trition and a working environment! I just need some help with the commands for 70B

[Advait251206]

Yes, but it depends on which quantization workflow you're planning to use.

TensorRT-LLM supports several quantization methods (e.g., AWQ, GPTQ, FP8, INT8), and the memory requirements during quantization vary accordingly. In general, you do not necessarily need to fit the full FP16/BF16 model on the GPUs at once to end up with a 4-bit TensorRT-LLM engine.

If you already have a pre-quantized checkpoint (for example, an AWQ or GPTQ checkpoint), you can typically convert/build the engine directly from that checkpoint without first loading the full-precision model onto the GPUs.

If, however, you're starting from the original Llama 3 70B weights and want TensorRT-LLM to perform the quantization itself, the conversion step may still require access to the full model (or enough CPU/GPU memory depending on the workflow and quantization algorithm).

For a setup with 3× RTX 4090, a common workflow would be:

1. Obtain or generate a supported 4-bit quantized checkpoint (e.g., AWQ or GPTQ).
2. Convert the checkpoint to the TensorRT-LLM format (if required by the workflow).
3. Build the engine with the desired tensor parallel configuration (e.g., tp=3 if appropriate for your setup).

The exact commands depend on:

• The TensorRT-LLM version you're using.
• Whether you're starting from:
  • the original Hugging Face Llama 3 70B weights,
  • an AutoAWQ checkpoint,
  • a GPTQ checkpoint,
  • or another quantized format.

Could you clarify:

• Which TensorRT-LLM release or commit you're using?
• Whether you're starting from the original Meta/Hugging Face checkpoint or an already quantized checkpoint?
• Which 4-bit quantization method you're targeting (AWQ, GPTQ, etc.)?

With that information, it would be possible to provide the exact conversion and trtllm-build commands for your setup.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206