CoDiff: Robust Collaborative 3D object deteciton via Diffusion-Based Feature Realignment

Zhe Huang, Shuo Wang, Yongcai Wang*, Deying Li, Zhaoxin Fan, Lei Wang · 2025

Read CoDiff: Robust Collaborative 3D object deteciton via Diffusion-Based Feature Realignment by Zhe Huang, Shuo Wang, Yongcai Wang*, Deying Li, Zhaoxin Fan, Lei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14891.txt.

Introduction

Zhe Huang, Shuo Wang, Yongcai Wang, and Lei Wang present CoDiff, in a 2025 preprint, a system that lets self-driving cars see together without being fooled by noise. Cars that share sensor findings spot more than one car alone. In practice, position errors and delays smear the shared features, and detection suffers. Diffusion models are natural cleaners of noisy data. The authors use one to realign the fused features from several cars. The work is, they say, the first to bring diffusion models to joint perception.

Features are first compressed into the latent space of a pre-trained autoencoder. Each car's data then guides a diffusion sampling that scrubs the shared map. Step by step, the fused features grow clearer and fuller. On simulated and real road data, CoDiff beat prior methods. It held up especially well when position and timing noise ran high.

Catalog record

CREATORZhe Huang, Shuo Wang, Yongcai Wang*, Deying Li, Zhaoxin Fan, Lei Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoDiff: Robust Collaborative 3D object deteciton via Diffusion-Based Feature Realignment Zhe Huang, Shuo Wang, Yongcai Wang*, Deying Li, Zhaoxin Fan, Lei Wang Abstract Collaborative 3D object detection holds significant importance in the field of autonomous driving, as it greatly enhances the perception capabilities of each individual agent by facilitating information exchange among multiple agents. Most collaborative perception systems rely on an idealized noise-free assumption, ignoring delays and pose inconsistencies during interaction. However, real-world deployment inevitably introduces noise, causing severe feature misalignments and performance degradation. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to explore the use of diffusion models to address feature misalignment in multiagent systems caused by noise. In this work, we propose CoDiff, a novel robust collaborative perception framework that leverages the potential of diffusion models to generate more comprehensive and clearer feature representations. Specifically, CoDiff consists of two core modules: a Pose Calibration Module (PCM), which establishes associations between objects detected by different agents and optimizes their relative poses by minimizing alignment errors; and a Time Calibration Module (TCM), which leverages single-agent features as conditions to remove delay-induced noise and generate clearer and more comprehensive collaborative representations. Experimental study on both simulated and real-world datasets demonstrates that the proposed framework CoDiff consistently outperforms existing relevant methods in terms of the collaborative object detection performance and exhibits highly desired robustness when the pose and delay information of agents is with high-level noise. Ablation studies further validate the effectiveness of key components and design choices within the framework. I Introduction 3D object detection [ 1 , 2 , 3 , 4 , 5 , 6 ] is a fundamental task in autonomous driving. It has gained significant interest in domains including drones, robotics, and the metaverse [ 7 , 8 , 9 , 10 , 11 , 12 ] . However, 3D object detection with a single agent has inherent limitations, such as occlusion and missing detection of distant objects [ 13 , 14 , 15 , 16 ] . Recently, researchers have addressed these limitations by enabling multiple agents to share complementary perception information through communication, leading to more comprehensive and accurate detection. Cooperative 3D object detection [ 17 , 18 , 19 , 20 ] is an important field that has made significant progress in terms of high-quality datasets and methods. However, it also faces many challenges, including time delays [ 21 ] , pose inconsistencies [ 19 ] , communication bandwidth limitations [ 18 ] , and adversarial attacks [ 22 ] . Existing methods have considered various methods to address these challenges. CoAlign [ 23 ] introduces a novel agent-object pose graph modeling approach to improve pose consistency between collaborative agents. RoCo [ 19 ] designs a matching and optimization strategy to correct pose inconsistencies. V2XViT [ 24 ] incorporates latency as an input for feature compensation, while CoBEVFlow [ 21 ] introduces the concept of feature flow to handle irregular time delays. The above methods work well when the noise is minimal and the number of vehicles is limited. However, in crowded and complex scenarios, as noise increases, misalignment in object information transmitted by different agents increases, making it increasingly difficult to align features at the same location. This situation ultimately reduces the accuracy of 3D object detection. Fig. 1: Illustration of collaborative perception under noise and the perception results w.o./w. CoDiff. Red boxes are detection results and green boxes are the ground truth. To effectively share information, multiple agents need to transmit precisely aligned real-time features to synchronize their data within a consistent timestamp and spatial coordinate system, which is the foundation for maintaining effective collaboration. However, in real-world scenarios, we observe that the transmitted information may suffer from pose and time misalignment. As illustrated in Fig. 1 , two key factors significantly affect the performance of collaborative perception. The first issue is time delays, as shown in Fig. 1 (a), when agent j sends information to agent i , unstable inter-agent communication (e.g. congestion or interruptions) may introduce delays, resulting in misaligned timestamps between agents. Secondly, the 6-DoF pose estimation conducted by each agent’s localization module is inherently imperfect, inevitably leading to relative pose inconsistencies. For instance, in Fig. 1 (b), a noticeable positional deviation between object 3 detected by agent i (orange box) and object 2 detected by agent j (blue box), which adversely affects subsequent feature fusion. When pose inconsistencies or time delays are large, this misalignment becomes even more pronounced. Therefore, this work focuses on addressing the inter-agent feature misalignment in 3D object detection caused by time delays and pose inconsistencies, with the goal of mitigating the negative impact of such noise on collaborative detection systems. To address this limitation, we propose a novel hybrid collaborative framework called CoDiff, which can address pose and time noise in information sharing, mitigate feature misalignment, and ultimately enhance detection accuracy. In CoDiff, we treat the time delays between multiple agents as a unified noise to be learned and handled, and employ conditional diffusion models for feature fusion, replacing the existing methods based on regression models and attention mechanisms. CoDiff primarily consists of two components: a Pose Calibration Module (PCM) and a Time Calibration Module (TCM). PCM establishes correspondences among multiple objects and constructs a pose optimization graph to effectively adjust agent pose inconsistencies. TCM first compresses the high-dimensional features of agents into a latent space and then employs a conditional diffusion model to progressively synthesize denoised, higher-quality features, as illustrated in Fig. 2 . CoDiff has three main advantages: i) The pose adjustment method in CoDiff is unsupervised, requiring no ground-truth pose annotations for either agents or objects, and it exhibits strong adaptability to various pose inconsistencies; ii) Unlike existing methods that design specialized fusion modules, the proposed fusion approach is a generative method based on diffusion models, which fundamentally prevents the introduction of additional noise into the features; iii) Even in crowded and complex environments, CoDiff can accurately align the agents and improve detection accuracy. We conducted extensive experiments on both simulation and real-world datasets, including V2XSet [ 24 ] , OPV2V [ 25 ] and DAIR-V2X [ 26 ] . Results show that CoDiff consistently achieves the best performance in the task of collaborative 3D object detection with the presence of pose inconsistencies and time delays. In summary, the main contributions of this work are as follows: • We propose CoDiff, a novel and powerful diffusion-based collaborative perception framework that addresses pose inconsistencies and time delays among multiple agents, alleviates feature misalignment, and improves collaborative detection performance. • CoDiff first establishes object correspondences through graph matching, refines their relative poses by minimizing alignment errors, and subsequently employs a diffusion-based generative fusion strategy to mitigate time delays, ultimately yielding clearer and more comprehensive representations. • Extensive experiments have shown that CoDiff consistently enhances the accuracy and robustness of 3D object detection, even under challenging conditions with dense traffic and significant noise. Fig. 2: Illustration of the proposed robust 3D multi-agent collaborative perception system with PCM and TCM module, where q is the diffusion process and p_{\theta} is the reverse process, massage M_{j\rightarrow i} is the condition. II Related Work Collaborative 3D Object

Read the complete text