CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR–Camera Calibration with Iterative and Attention-Driven Post-Refinement
Accurate multi-sensor calibration is essential for deploying robust perception systems in applications such as autonomous driving and intelligent transportation. Existing LiDAR-camera calibration methods often rely on manually placed targets, preliminary parameter estimates, or intensive data preprocessing, limiting their scalability and adaptability in real-world settings. In this work, we propose a fully automatic, targetless, and online calibration framework, CalibRefine, which directly processes raw LiDAR point clouds and camera images. Our approach is divided into four stages: (1) a Common Feature Discriminator that leverages relative spatial positions, visual appearance embeddings, and semantic class cues to identify and generate reliable LiDAR-camera correspondences, (2) a coarse homography-based calibration that uses the matched feature correspondences to estimate an initial transformation between the LiDAR and camera frames, serving as the foundation for further refinement, (3) an iterative refinement to incrementally improve alignment as additional data frames become available, and (4) an attention-based refinement that addresses non-planar distortions by leveraging a Vision Transformer and cross-attention mechanisms. Extensive experiments on two urban traffic datasets demonstrate that CalibRefine achieves high-precision calibration with minimal human input, outperforming state-of-the-art targetless methods and matching or surpassing manually tuned baselines. Our results show that robust object-level feature matching, combined with iterative refinement and self-supervised attention-based refinement, enables reliable sensor alignment in complex real-world conditions without ground-truth matrices or elaborate preprocessing.
Train.py→ train the Common Feature Discriminator modelTest.py→ test the accuracy of the Common Feature Discriminator
After training the Common Feature Discriminator, you can now use:
python calibration/lidar_to_camera_calibration.pyThis performs the coarse calibration.
After obtaining the coarse calibration matrix, refine it by running:
python calibration/Online_Refine_Calibration_IT_finetune.py📺 To visualize the calibration results:
python calibration/Video_Visualization.pyAfter coarse calibration, you may keep refining the calibration matrix by performing Attention-based Refinement in the folder /Attention_refinement_model/:
Train_finetune.py→ train the Attention modelTest_finetune.py→ test the performance of the Attention modelAdaptive_Correction.py→ apply Attention-based Refinement
If you have the camera intrinsic matrix, you may choose to do Intrinsic-Rectified Calibration instead of Attention-based Refinement.
First run:
python rectify_image_bbox.pyThis employs the intrinsic matrix to rectify image distortion.
Then, you can repeat the above calibration steps (Coarse Homography Calibration and Iterative Refinement), where the .py files ending with _rectify are specifically designed for this purpose.