DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion

Zixuan Yin, Han Sun 🖂, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen · 2025

Read DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion by Zixuan Yin, Han Sun 🖂, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00220.txt.

Introduction

Zixuan Yin, Han Sun, Ningzhong Liu, Huiyu Zhou, and Jiaquan Shen present DecoratingFusion in 2024, a network that joins LiDAR and camera data for 3D spotting in self-driving cars. Most fusion tools lean on learned soft links between point clouds and images, which are hard to read and skip firm links.

Firm links from sensor setup guide the hunt for objects here. Image traits paint the point cloud early on, and twin sparse filters pull out the mixed traits. Search seeds start from a center heat map with class tags mixed in. Tests on KITTI and Waymo beat prior tools.

Catalog record

CREATORZixuan Yin, Han Sun 🖂, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion Zixuan Yin, Han Sun 🖂, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen Abstract Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion. 1 Introduction In recent years, lidar-camera fusion methods have been increasingly applied in 3D object detection for autonomous driving scenarios [ 16 ] . Point cloud data provides spatial geometric information, describing object shape, position, and size, while image data provides color, texture, and visual features. However, due to significant domain gaps between the two modalities, feature alignment has become a key challenge. After undergoing the development of late-fusion at the result level and early-fusion at the point level, the current state-of-the-art fusion method is mid-level feature fusion [ 19 ] . This kind of method maximizes the complementarity between the two types of data but is also the most challenging to implement. Due to the high abstraction level of the features, the key challenge lies in how to match the data from the two modalities in the feature space effectively. Existing feature fusion methods mostly rely on learning-based approaches to obtain soft correlations between point clouds and images. However, these soft correlations lack interpretability, making it difficult to ensure their reliability. In contrast, the hard correlations commonly used in point-level fusion methods (such as calibration matrices between LiDAR and cameras, which are usually provided by data collectors) are intuitive, ensuring the alignment between point clouds and images. However, hard correlations are rarely applied in feature fusion methods because they become feature-to-feature mappings in the feature space, rendering the use of hard correlations impractical. To introduce hard correlations into feature fusion methods, we combine early fusion and mid-level fusion and propose an efficient multi-modal fusion strategy called DecoratingFusion. Specifically, we utilize a 2D CNN to extract image features and employ them to decorate the original point cloud data. These convolutional layers, along with other network components, are trained in an end-to-end manner. Considering the domain gaps between the two modalities, two independent sparse convolutions are used to extract the decorated point cloud features, which are then concatenated. These concatenated features are utilized to generate object queries and fuse point cloud and image features through cross-attention mechanisms. Additionally, to optimize the initialization process of object queries, we first predict a center heatmap using the decorated point cloud features and select the initialization position of the query from the heatmap. Finally, the fused features are fed into the prediction head to obtain the final results. In brief, our contributions can be summarized as follows: • We combine the early fusion and mid-level fusion approaches, utilizing the hard correlations established through calibration matrices to guide the generation of object queries, and fuse the point cloud and image features using cross-attention mechanisms. • We use two independent sparse convolutions to extract the decorated point cloud features. Additionally, we initialize the object queries using a center heatmap and embed the predicted class from the center heatmap as auxiliary information into the object queries. • We validate our method on two prominent autonomous driving datasets, namely KITTI and Waymo. The experimental results demonstrate the effectiveness of our proposed approach. 2 Related Work According to the fusion stage, existing fusion methods can be divided into three categories: early fusion, mid-level fusion, and late fusion. Below, we will introduce each of them in chronological order of their development. 2.1 Late Fusion Late fusion methods refer to fusion performed at the later stages of the network, also known as result-level fusion. Early multi-modal fusion methods are mostly late fusion methods, where each modality’s data is independently predicted, and the detection results are subsequently fused. This category of methods has the simplest fusion approach and high reliability because they can still operate normally under single-modality conditions. However, their fusion granularity is the lowest among the three categories, often resulting in poor performance. Classic late fusion methods include F-PointNets [ 7 ] , CLOCs [ 6 ] . In recent years, early fusion methods and mid-level fusion methods have gained more attention, while research on late fusion methods has been relatively limited. 2.2 Early Fusion Early fusion methods refer to fusion performed at the early stages of the network, where different modalities of data are fused to create a new modality, which serves as the input to the feature extraction network. For example, PointPainting [ 11 ] utilizes pixel-level semantic segmentation scores to decorate point cloud data as camera features, while PointAugmenting [ 12 ] decorates point cloud data with image features. Although PointPainting and PointAugmenting provide novel approaches for multi-modal fusion, researchers quickly discover the bottleneck in them. While these fusion approach allow for the alignment of 3D and 2D coordinates using calibration matrices, existing feature extractors struggle to directly process the fused data. Typical 3D backbone networks are designed specifically to extract sparse point cloud features, and as point cloud and image data have significant differences in characteristics, using existing 3D backbone networks to extract features from the fused data is not suitable. 2.3 Mid-level Fusion Mid-level fusion methods refer to fusion performed during the feature extraction stage of the network. Compared to early fusion, which can only fuse data at the raw data level, mid-level fusion methods can operate at the feature level. Therefore, in theory, mid-level fusion methods can maximize the advantages of fusion. However, most existing mid-level fusion methods rely on learned soft correlations between point clouds and images. We consider this approach to be unreliable because it lacks interpretability, and existing feature-level fusion methods often overlook the presence of hard correlations. For example, TransFusion [ 1 ] introduces the SMCA (Spatially Modulated Cross Attention) module, which allows the network to adaptively determine which parts of the image features are more important and suitable for fusion. DeepFusion [ 5 ] enables a voxel in the point cloud to match multiple pixels in the image and assigns weights to the pixels using cross-attention mechanisms. 3 DecoratingFusion 3.1 Motivations and Pipeline Before our work, early fusion methods and mid-level fusion methods are completely different approaches. We aimed to leverage the advantages of both by incorporating the hard correlations established through calibration matrices to achieve better detection results. To utilize the hard correlations between point clouds and images to guide the fusion of the two

Read the complete text