Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement
Aziz Al-Najjar, Marzieh Amini, James R. Green, Felix Kwamena · 2026
Read Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement by Aziz Al-Najjar, Marzieh Amini, James R. Green, Felix Kwamena (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31576.txt.
Introduction
Al-Najjar and team present a two-stage system for joint camera and LiDAR extrinsic calibration. Most methods treat each camera and LiDAR pair alone, ignoring rigid coupling across cameras, so estimates may be accurate yet inconsistent. First, CMRNext runs on each camera to produce initial extrinsic estimates plus dense 2D-3D correspondences. Those predictions are then refined through multi-frame bundle adjustment with reprojection and pose prior terms. Tests on KITTI and Walkley show better accuracy and consistency, with 0.89 cm error on KITTI and a sharp drop on Walkley.
Catalog record
| CREATOR | Aziz Al-Najjar, Marzieh Amini, James R. Green, Felix Kwamena |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35222 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31576
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement Aziz Al-Najjar, Marzieh Amini, James R. Green, Felix Kwamena Abstract Most learning-based camera–LiDAR calibration methods treat each camera–LiDAR pair independently, ignoring the rigid geometric coupling in multi-camera platforms. As a result, per-camera estimates may be individually accurate yet inconsistent at the system level. We present a two-stage framework for joint multi-camera LiDAR extrinsic calibration that combines learned pairwise matching with geometric refinement. First, CMRNext is applied independently to each camera to produce initial extrinsic estimates and dense 2D–3D correspondences. These predictions are then jointly refined through a multi-frame bundle adjustment with reprojection, per-camera prior, and relative-pose prior terms. This approach converts pairwise predictions into a globally consistent multi-camera calibration. Experiments on KITTI (in-domain for CMRNext) and Walkley (out-of-domain) datasets show improved per-camera accuracy and inter-camera consistency. On KITTI, the method achieves 0.89 cm translation error and 0.038 ∘ rotation error. On Walkley, it reduces translation error from 108.6 cm to 3.1 cm, highlighting the benefit of explicit multi-camera coupling when single-camera predictions are less reliable. Keywords: camera-LiDAR calibration, multi-camera calibration, extrinsic calibration, sensor fusion, bundle adjustment, geometric refinement, cross-modal matching. 1 Introduction Modern perception systems rely on combining the geometric accuracy of LiDAR with the rich visual information of cameras to support tasks such as object detection, semantic segmentation, and scene understanding 2 ; 7 . This fusion critically depends on accurate extrinsic calibration, which defines the mapping that projects 3D LiDAR points into the camera image plane. Even small calibration errors can lead to significant reprojection misalignment, degrading downstream performance 17 ; 1 . While single camera-LiDAR calibration has been studied extensively 23 ; 20 ; 16 ; 15 ; 18 ; 11 ; 13 ; 6 , many real-world platforms deploy multiple cameras around a single LiDAR sensor to increase field of view and robustness 8 ; 3 . In such systems, the calibration problem acquires an additional geometric structure: all cameras are rigidly attached to the same platform, and their extrinsics are therefore not independent. Instead, they are coupled through the shared LiDAR coordinate frame, which fully determines the relative pose between cameras. Ignoring this coupling can result in calibrations that are individually plausible but jointly inconsistent, leading to misalignment artifacts across views 10 . This raises the central question addressed in this work: can learned pairwise camera–LiDAR calibration be improved by jointly optimizing all cameras under the rigid geometric constraints of a shared multi-camera platform? To answer this question, we propose a two-stage framework for joint multi-camera LiDAR extrinsic calibration. In Stage 1, a single-camera learning-based method, CMRNext, is applied independently to each camera to produce initial extrinsic estimates alongside dense 2D–3D correspondences. In Stage 2, these outputs are jointly refined through a multi-frame bundle adjustment that combines a reprojection term with per-camera and relative-pose priors to enforce inter-camera consistency. We evaluate the proposed method on two datasets with complementary characteristics. On KITTI 8 , one of the datasets on which CMRNext was originally trained and performed well. On Walkley 3 , a custom outdoor multi-camera LiDAR dataset with a different sensor configuration and image resolution, the CMRNext predictions are substantially less reliable, and the benefits of joint refinement become much larger. This contrast allows us to study whether multi-camera coupling helps, and under which operating conditions it provides improvements. The main contributions of this work are: • A joint formulation for multi-camera LiDAR extrinsic calibration that uses learned single-camera predictions as initialization and refines all camera extrinsics together, rather than independently. • An explicit relative-pose prior that enforces inter-camera geometric consistency during optimization and enables information to propagate across cameras. • A study on in-domain and out-of-domain datasets showing when reprojection-only refinement is sufficient and when adding per-camera and relative-pose priors yields additional gains in both per-camera accuracy and system-level consistency. 2 Related Work Camera-LiDAR calibration methods can be broadly divided into target-based, targetless, and learning-based approaches. Target-based calibration: Target-based methods use known geometric structures such as checkerboards or planar targets to establish explicit 2D-3D correspondences. For example, Zhou et al. 23 exploit line and plane constraints extracted from a calibration board, while Verma et al. 20 combine point and plane correspondences to solve for the rigid transform. These methods typically reduce calibration to a Perspective- n -Point (PnP) or least-squares problem with well-conditioned constraints, yielding high accuracy. However, they require controlled data collection and manual setup, limiting their practicality in dynamic or large-scale deployments. Targetless calibration: Targetless methods avoid calibration objects by leveraging natural scene structure. A classical family of approaches formulates calibration as an information-theoretic alignment problem, maximizing mutual information between LiDAR-derived attributes (e.g., depth or reflectance) and image intensities 16 ; 15 . Other methods rely on geometric consistency, such as aligning planes or edges across modalities, or minimizing reprojection error through bundle-adjustment-style optimization 4 ; 12 . While these methods eliminate the need for calibration targets, they often depend on scene geometry, good initialization, or handcrafted features, and can struggle in sparse or ambiguous environments. Learning-based calibration: Learning-based approaches aim to improve robustness through data-driven feature extraction. Regression-based methods such as RegNet, CalibNet, and LCCNet directly predict the extrinsic transformation from RGB images and projected LiDAR inputs 18 ; 11 ; 13 . These models typically use convolutional encoders and optimize geometric consistency losses, but they implicitly learn dataset-specific cues and may generalize poorly across sensor configurations 10 . More recent matching-based methods decouple correspondence estimation from geometric optimization. CMRNext 6 follows this paradigm by predicting dense optical flow between a rendered LiDAR depth map and the camera image, from which 2D-3D correspondences are derived and used to estimate the extrinsic. This formulation improves generalization, but it remains limited to single camera-LiDAR pairs and does not enforce consistency across multiple cameras. Multi-sensor calibration: Multi-sensor calibration methods extend the problem beyond a single camera-LiDAR pair and aim to recover a globally consistent set of extrinsics. Tu et al. 19 formulate multi-camera-LiDAR calibration as a joint structure-from-motion problem. UniCal 21 learns calibration in a unified framework across heterogeneous sensors, while continuous-time formulations jointly model calibration and sensor motion over trajectories 14 . Multi-Calib 9 further studies scalable calibration under varying sensor configurations. These methods address global consistency, but they do not exploit the recent strength of learned pairwise camera-LiDAR matching as the front end of a rig-aware refinement stage. 3 Methodology Figure 1 : Overview of the proposed joint multi-camera LiDAR calibration framework Our proposed pipeline first obtains per-camera extrinsic initialization from CMRNext and then jointly refines all extrinsics using multi-frame BA (Fig. 1 ). 3.1 Problem Formulation Let \mathcal{F}_{L} denote the LiDAR coordinate frame, and let \{\mathcal{F}_{C_{1}},\ldots,\mathcal{F}_{C_{N}}\} denote the coordinate frames of N rigidly mounted cameras, where C_{i} is the i -th camera. The goal is to estimate the set of extrinsic transformations \mathcal{T}=\bigl\{T_{L\to C_{1}},\,\ldots,\,T_{L\to C_{N}}\bigr\},\quad T_{L\to C_{i}}\in SE(3) (1) where SE(3) denotes the special Euclidean group of 3D rigid-body transformations, and each T_{L\to C_{i}} maps a 3D point from the LiDAR frame to the coordinate frame of camera C_{i} . Given a synchronized LiDAR point cloud \mathbf{P}^{t}=\{\mathbf{p}_{k}^{t}\} and camera images \{I_{i}^{t}\} at time t , a 3D point \mathbf{p}_{k}^{t} projects into camera C_{i} through the corresponding camera extrinsic and