[OMNIML-5562] Add FAR3D ONNX PTQ and accuracy evaluation example

.pre-commit-config.yaml

@@ -126,6 +126,7 @@ repos:
               examples/llm_eval/lm_eval_hf.py|
               examples/llm_eval/mmlu.py|
               examples/llm_eval/modeling.py|
+              examples/onnx_ptq/far3d/evaluate.py|
               examples/llm_qat/train.py|
               examples/llm_sparsity/weight_sparsity/finetune.py|
               examples/specdec_bench/specdec_bench/models/specbench_medusa.py|

LICENSE

@@ -223,6 +223,7 @@ the following copyright holders, licensed under the Apache License, Version 2.0
   Copyright 2023 Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li
   Copyright (c) 2024 Heming Xia
   Copyright 2025 The Qwen team, Alibaba Group and the HuggingFace Inc. team
+  Copyright (c) OpenMMLab. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not
 use these files except in compliance with the License. You may obtain a copy

examples/onnx_ptq/README.md

@@ -129,6 +129,10 @@ The top5 accuracy of the model is <accuracy score between 0-100%>
 Inference latency of the model is <X> ms
 ```
 
+### FAR3D 3D object detection
+
+The [FAR3D example](./far3d/) demonstrates an end-to-end workflow that exports and quantizes the FAR3D ONNX image encoder, builds TensorRT engines, and evaluates 3D object detection mAP on the Argoverse 2 validation set.
+
 ## Advanced Features
 
 ### Per node calibration of ONNX models

examples/onnx_ptq/far3d/Dockerfile

@@ -0,0 +1,80 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM nvcr.io/nvidia/pytorch:25.06-py3
+
+ENV LD_LIBRARY_PATH=/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
+
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        libgl1 \
+        libglib2.0-0 && \
+    rm -rf /var/lib/apt/lists/*
+
+ENV UV_PYTHON_INSTALL_DIR=/opt/python
+
+RUN python -m pip install --no-cache-dir uv && \
+    uv python install 3.8 && \
+    uv venv --seed --python 3.8 /opt/far3d
+
+RUN env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        torch==1.13.1+cu117 \
+        torchvision==0.14.1+cu117 \
+        --extra-index-url https://download.pytorch.org/whl/cu117
+
+RUN env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        "numpy<1.24" \
+        opencv-python==4.5.5.64 \
+        yapf==0.32.0 \
+        mmcv-full==1.7.0 \
+        -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13.0/index.html
+
+RUN env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        av2==0.2.1 \
+        einops \
+        "ipython<9" \
+        kornia==0.6.12 \
+        mmdet==2.28.2 \
+        mmsegmentation==0.30.0 \
+        onnx \
+        onnx-graphsurgeon==0.6.1 \
+        onnxruntime \
+        onnxsim \
+        refile \
+        tensorrt-cu12-bindings==10.11.0.33 \
+        --extra-index-url https://pypi.nvidia.com && \
+    mkdir -p /opt/far3d/lib/python3.8/site-packages/tensorrt && \
+    cp /opt/far3d/lib/python3.8/site-packages/tensorrt_bindings/__init__.py \
+        /opt/far3d/lib/python3.8/site-packages/tensorrt/__init__.py && \
+    cp /opt/far3d/lib/python3.8/site-packages/tensorrt_bindings/tensorrt.so \
+        /opt/far3d/lib/python3.8/site-packages/tensorrt/tensorrt.so
+
+RUN env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        "setuptools<81" && \
+    env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        --no-build-isolation \
+        mmdet3d==1.0.0rc6
+
+COPY . /opt/modelopt
+RUN env -u PIP_CONSTRAINT python -m pip install --no-cache-dir \
+        lief \
+        ml_dtypes \
+        omegaconf \
+        "onnx-graphsurgeon>=0.6.1" \
+        "onnx~=1.21.0" \
+        "onnxruntime-gpu~=1.24.2" \
+        onnxscript \
+        "onnxslim>=0.1.76" \
+        "setuptools>=80" && \
+    env -u PIP_CONSTRAINT python -m pip install --no-cache-dir --no-deps -e /opt/modelopt

examples/onnx_ptq/far3d/README.md

@@ -0,0 +1,148 @@
+# FAR3D ONNX PTQ and Argoverse 2 evaluation
+
+This example quantizes the FAR3D image encoder and decoder to INT8 or FP8 with Model Optimizer and evaluates the complete pipeline on the Argoverse 2 validation set. It follows the [NVIDIA DL4AGX FAR3D workflow](https://github.com/NVIDIA/DL4AGX/tree/master/AV-Solutions/far3d-trt).
+
+FAR3D uses a legacy PyTorch/MMCV environment that is incompatible with the current Model Optimizer Python dependencies. The provided image uses `nvcr.io/nvidia/pytorch:25.06-py3` for the complete workflow and isolates the legacy FAR3D packages in a Python 3.8 virtual environment.
+
+## 1. Prepare FAR3D and Argoverse 2
+
+Clone DL4AGX, initialize its submodules, and apply its FAR3D patch:
+
+```bash
+git clone https://github.com/NVIDIA/DL4AGX.git
+cd DL4AGX
+git submodule update --init --recursive
+cd AV-Solutions/far3d-trt/dependencies/Far3D
+git apply ../../patch/far3d.patch
+git apply /path/to/Model-Optimizer/examples/onnx_ptq/far3d/far3d_pytorch_25_06.patch
+cd ../..
+```
+
+The second patch makes the unused CUDA 11-only FlashAttention implementation optional; the reference configuration uses MMCV `MultiheadAttention`.
+
+Download the [Argoverse 2 sensor validation set](https://www.argoverse.org/av2.html), the [reference FAR3D checkpoint](https://github.com/NVIDIA/DL4AGX/tree/master/AV-Solutions/far3d-trt#pytorch-model-to-onnx), and its configuration. The remaining commands assume:
+
+```text
+far3d-trt/
+├── data/av2/val/
+├── dependencies/Far3D/projects/configs/far3d.py
+└── weights/iter_82548.pth
+```
+
+Build the example image from the Model Optimizer checkout:
+
+```bash
+docker build \
+  -f /path/to/Model-Optimizer/examples/onnx_ptq/far3d/Dockerfile \
+  -t far3d-modelopt \
+  /path/to/Model-Optimizer
+```
+
+Start the image and mount the FAR3D checkout:
+
+```bash
+docker run --rm -it --network=host --gpus=all --shm-size=80G --privileged \
+  -v /data/av2:/data/av2 \
+  -v /path/to/far3d-trt:/workspace/far3d-trt \
+  far3d-modelopt
+```
+
+Use `/opt/far3d/bin/python` for data preparation, export, and evaluation. It selects the isolated legacy FAR3D environment:
+
+```bash
+export PYTHONPATH=/workspace/far3d-trt/dependencies/Far3D
+cd /workspace/far3d-trt
+/opt/far3d/bin/python /opt/modelopt/examples/onnx_ptq/far3d/prepare_metadata.py data/av2
+```
+
+## 2. Export the ONNX models
+
+```bash
+/opt/far3d/bin/python tools/export_onnx.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  weights/iter_82548.pth
+```
+
+This produces `far3d.encoder.onnx` and `far3d.decoder.onnx`.
+
+## 3. Prepare calibration batches
+
+Build temporary engines from the exported models. They run the reference pipeline while collecting representative encoder and decoder inputs:
+
+```bash
+trtexec \
+  --onnx=far3d.encoder.onnx \
+  --saveEngine=far3d.encoder.fp16.engine \
+  --fp16 \
+  --skipInference
+trtexec \
+  --onnx=far3d.decoder.onnx \
+  --saveEngine=far3d.decoder.fp16.engine \
+  --stronglyTyped \
+  --skipInference
+```
+
+Extract 512 batches sampled every 20 frames from the Argoverse 2 validation loader:
+
+```bash
+/opt/far3d/bin/python /opt/modelopt/examples/onnx_ptq/far3d/prepare_calibration.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  data/far3d_calibration \
+  --encoder-engine far3d.encoder.fp16.engine \
+  --decoder-engine far3d.decoder.fp16.engine \
+  --num-samples 512 \
+  --sample-skip-interval 20
+```
+
+The calibration directory contains separate `encoder/` and `decoder/` batches. Decoder batches include the image features, camera geometry, and temporal state seen by the reference decoder.
+
+## 4. Quantize the models
+
+Use the base Python environment for Model Optimizer:
+
+```bash
+python /opt/modelopt/examples/onnx_ptq/far3d/quantize.py \
+  far3d.encoder.onnx \
+  far3d.decoder.onnx \
+  data/far3d_calibration
+```
+
+Both models use max calibration. INT8 is the default; use `--quantization-mode fp8` to produce `far3d.encoder.fp8.onnx` and `far3d.decoder.fp8.onnx` instead. FP8 deployment requires an FP8-capable GPU.
+
+The quantizer preserves the accuracy-sensitive exclusions used by the DL4AGX reference: the `OSA4_5` block and nodes downstream of `lateral_convs` remain in high precision.
+
+To keep the decoder in its original mixed FP16/FP32 precision, add `--fp16-decoder`; decoder calibration batches are not required in that mode. This flag can be combined with either quantization mode.
+
+Build both engines in the same container. Serialized TensorRT engines are not portable across TensorRT versions or GPU architectures.
+
+Set the precision to the quantization mode used above:
+
+```bash
+precision=int8  # Use fp8 for FP8 models.
+trtexec \
+  --onnx=far3d.encoder.${precision}.onnx \
+  --saveEngine=far3d.encoder.${precision}.engine \
+  --stronglyTyped \
+  --skipInference
+trtexec \
+  --onnx=far3d.decoder.${precision}.onnx \
+  --saveEngine=far3d.decoder.${precision}.engine \
+  --stronglyTyped \
+  --skipInference
+```
+
+When using `--fp16-decoder`, build `far3d.decoder.onnx` as `far3d.decoder.fp16.engine` instead.
+
+## 5. Evaluate accuracy
+
+```bash
+precision=int8  # Use fp8 for FP8 models.
+/opt/far3d/bin/python /opt/modelopt/examples/onnx_ptq/far3d/evaluate.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  far3d.encoder.${precision}.engine \
+  far3d.decoder.${precision}.engine
+```
+
+The evaluator reports the dataset metrics, including mAP. The DL4AGX reference reports 0.230 mAP for its INT8 encoder and mixed FP16/FP32 decoder, compared with 0.232 mAP for FP16 encoder and decoder. It reports neither an INT8 decoder nor FP8 results; validate the selected precision on the target TensorRT version and GPU.
+
+Use `--max-samples N` for an inference smoke test. Dataset metrics are skipped when only part of the validation set is processed.