PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image
Yuanchao Yue, Zhengxin Li, Wei Zhang, Huiyuan · 2025
Read PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image by Yuanchao Yue, Zhengxin Li, Wei Zhang, Huiyuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15285.txt.
Introduction
Yue and co-authors present a method, published in 2025, for aligning laser scans of a scene with camera photos of it. Self-driving systems fuse these two views, but the sensors must first be matched precisely. Classic matching needs a calibration board or special scenery, which is slow. Their method, named EEPNet-V2 in the text, aligns the data directly. It projects the three-dimensional laser points into several flat views that a matcher can compare with the photo.
The matcher works in stages, from rough patches down to single pixels. That patch-to-pixel design aims for fine accuracy at real-time speed. Past methods usually traded one for the other. Careful alignment is the base on which sensor fusion stands. Errors there ripple into every map and detection that follows. The authors test their scheme against earlier approaches on standard data.
Catalog record
| CREATOR | Yuanchao Yue, Zhengxin Li, Wei Zhang, Huiyuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAPI-Reg: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image Yuanchao Yue, Zhengxin Li, Wei Zhang, Huiyuan Abstract The primary requirement for cross-modal data fusion is the precise alignment of data from different sensors. However, the calibration between LiDAR point clouds and camera images is typically time-consuming and needs external calibration board or specific environmental features. Cross-modal registration effectively solves this problem by aligning the data directly without requiring external calibration. However, due to the domain gap between the point cloud and the image, existing methods rarely achieve satisfactory registration accuracy while maintaining real-time performance. To address this issue, we propose a framework that projects point clouds into several 2D representations for matching with camera images, which not only leverages the geometric characteristic of LiDAR point clouds more effectively but also bridge the domain gap between the point cloud and image. Moreover, to tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network to effectively extract features from both camera images and the projection maps of LiDAR point cloud. Additionally, we propose a patch-to-pixel matching network to provide more effective supervision and achieve higher accuracy. We validate the performance of our model through experiments on the KITTI and nuScenes datasets. Our network achieves real-time performance and extremely high registration accuracy. On the KITTI dataset, our model achieves a registration accuracy rate of over 99%. 1 Introduction In autonomous driving and robot systems, the fusion between data collected from different sensors now plays an important role in many downstream tasks, such as object detection Bai et al. 2022 ; Chen et al. 2017 ; Ku et al. 2018 and segmentation Caltagirone et al. 2021 ; Wu et al. 2018 ; Wang and Neumann 2018 . We can achieve better accuracy and efficiency by using these fused data, thereby ensuring safety and robustness in complex environments. Among these methods, the most common process is to perform a fusion of the LiDAR (solid-state LiDAR or rotating LiDAR, etc.) point cloud and the camera (high-definition cameras or infrared cameras, etc.) image. Figure 1 : The problem about the domain gap in the cross-modal registration task. Due to the dimensional differences between point clouds and images, the matching failure rate between them is high, which in turn reduces the registration accuracy. However, the primary condition of a successful fusion is an accurate alignment between these data. A commonly used method to deal with the alignment task between sensors is to calculate the external parameters between sensors using a checkerboard Kim and Park 2019 , which we call off-line calibration. We can make a high-accuracy alignment between sensors by using these types of methods and unavoidable time costs. In addition, some online calibration methods Schneider et al. 2017 ; Iyer et al. 2018 ; Yuan et al. 2020 are proposed to correct external parameter errors caused by bumps or vibrations. These online calibration methods can modify the deviations within a limited error range. However, they will lose effectiveness when the errors are too large. Although these methods can achieve high-accuracy alignment, these complicated calibration processes make data fusion preprocessing time-consuming and inefficient. Whenever the deployment of sensors has changed, these processes have to be repeated. Why not propose a method to bypass these calibration processes and save manpower and resources by directly doing a registration between the collected data? Cross-modal registration methods have emerged to address this challenge. Recent cross-modal registration methods Feng et al. 2019 ; Pham et al. 2020 can solve this cross-modal data alignment task by using specific deep learning networks. However, the cross-modal registration process typically involves comparing the similarity of the high-dimensional features between point clouds and images to establish correspondences, and the domain gap between 3D and 2D data always influence this process, as we showed in Fig. 1 . Specifically, there is a dimensional and informational difference between the 3D spatial data of the point cloud and the 2D texture information of the image. Although some existing methods attempt to decline these influence by exchanging feature information Ren et al. 2022 or using feature extraction networks with more similar structures Zhou et al. 2024 , this domain gap still affect the accuracy of the registration, especially leading to a high mismatching rate. Figure 2 : The challenge we faced in the matching task between projection map and camera image. The cross-modal difference between the camera image and the projection map will make it difficult to the feature matching. Additionally, the scarcity of positive samples caused by the limited overlapping region will lead to the network’s supervision problem. Besides, the commonly used point cloud feature extraction network in the cross-modal registration task (e.g., PointNet++ Qi et al. 2017b , sparse convolution network Choy et al. 2019 ) always treats the LiDAR point cloud simply as a series of unordered points or voxels, which fails to effectively use the spatial and geometric characteristics of the point cloud. Therefore, to make better use of the scanning characteristics of LiDAR to bridge the domain gap, we introduce a method to project point clouds into 2D representations: a range map and a reflectance map. Although using these projection methods is not new in other downstream tasks involving LiDAR point clouds, they have not been fully leveraged in cross-modal registration methods. Moreover, mapping point clouds to pixels in 2D representations offers a data format that implicitly contains more geometric information, making it more effective and accurate than using unordered sparse point clouds. This projection approach will enable more effective feature extraction and bridge the domain gap difference between point clouds and images. Furthermore, we combined the use of the two projection maps containing different types of information to enhance the system’s robustness for registration across various scenarios. After the projection, we transform the cross-modal registration task to an image registration task between the camera image and the projection maps. Common image registration methods are generally accomplished through key point matching. Due to the cross-modal differences between the projected 2D map and the camera image, traditional key point registration methods cannot be directly applied to this task, as we showed in Fig. 2 . Furthermore, the overlapping region between the 2D map and the camera image is often limited in practical scenarios, therefore the proportion of truly matched point pairs becomes extremely small, leading to positive pair scarcity. For existing deep learning-based image matching methods, this scarcity can cause supervision difficulties, making the network hard to converge and increasing the number of outliers, which in turn raises the likelihood of registration failures. To address these challenges, we designed a multi-scale feature extraction network to mitigate the cross-modal differences between images and projection maps, effectively extracting features. Additionally, we introduced a patch-to-pixel feature matching framework. Specifically, patch-level searching addresses the supervision and success rate issues caused by positive sample scarcity, while pixel-level matching enhances registration accuracy. Our experiment part has proved that our network design achieves high accuracy while maintaining low computational complexity. In conclusion, our contributions are as follows. 1. We propose a robust registration framework that leverages range map and reflectance map