Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining

Xiang Xu *, Lingdong Kong *, Hui Shuai, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, Qingshan Liu · 2025

Read Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining by Xiang Xu *, Lingdong Kong *, Hui Shuai, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, Qingshan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19912.txt.

Introduction

Self-driving cars read the world with LiDAR, but teaching those systems usually demands hand-labeled data. An eight-author team presents SuperFlow++ in 2025, a pretraining method that learns from raw LiDAR and camera streams together. It adds four pieces the earlier version lacked. One aligns meaning across camera views, and one keeps features steady as point clouds thin out. A third learns how scenes flow over time, and a fourth votes semantics across scans for steadier predictions.

Across eleven datasets the method beats the best earlier systems on many tasks and driving conditions. Scaling up the image and 3D backbones reveals new emergent behavior. The authors offer the work as a step toward general 3D foundation models that need little labeling.

Catalog record

CREATORXiang Xu *, Lingdong Kong *, Hui Shuai, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, Qingshan Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining Xiang Xu *, Lingdong Kong *, Hui Shuai, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, Qingshan Liu Abstract LiDAR representation learning has emerged as a promising approach to reducing reliance on costly and labor-intensive human annotations. While existing methods primarily focus on spatial alignment between LiDAR and camera sensors, they often overlook the temporal dynamics critical for capturing motion and scene continuity in driving scenarios. To address this limitation, we propose SuperFlow++ , a novel framework that integrates spatiotemporal cues in both pretraining and downstream tasks using consecutive LiDAR-camera pairs. SuperFlow++ introduces four key components: (1) a view consistency alignment module to unify semantic information across camera views, (2) a dense-to-sparse consistency regularization mechanism to enhance feature robustness across varying point cloud densities, (3) a flow-based contrastive learning approach that models temporal relationships for improved scene understanding, and (4) a temporal voting strategy that propagates semantic information across LiDAR scans to improve prediction consistency. Extensive evaluations on 11 heterogeneous LiDAR datasets demonstrate that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions. Furthermore, by scaling both 2D and 3D backbones during pretraining, we uncover emergent properties that provide deeper insights into developing scalable 3D foundation models. With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving. The code is publicly available at https://github.com/Xiangxu-0103/SuperFlow . I Introduction Robust perception is essential for autonomous driving, forming the basis for safe and efficient navigation. Advances in sensing technologies, such as LiDAR (Light Detection and Ranging) and surround-view cameras, have greatly improved 3D scene understanding, enabling comprehensive and precise environmental perception [ 1 , 2 , 3 ] . Despite these advances, achieving high-performing 3D perception remains challenging due to the heavy reliance on extensive labeled data and high computational costs [ 4 , 5 ] . Compared to 2D images, annotating 3D point clouds is more expensive and labor-intensive, posing a major scalability bottleneck for existing methods [ 6 , 7 , 8 , 9 ] . To mitigate this challenge, data representation learning has emerged as a promising alternative [ 10 ] . By designing effective pretraining objectives, models can extract meaningful features from raw, unlabeled data, reducing dependence on large-scale annotations while improving downstream performance [ 11 ] . Following this paradigm, Sautier et al. [ 12 ] introduced SLidR, a framework that transfers knowledge from 2D pretrained backbones, such as MoCo [ 13 ] and DINO [ 14 ] , to LiDAR point clouds. SLidR leverages a superpixel-driven contrastive learning mechanism to align camera and LiDAR features. Subsequent works have refined this approach with techniques such as class balancing [ 15 ] , hybrid-view distillation [ 16 , 17 ] , semantic superpixels [ 18 , 19 ] , and representation distillation [ 20 ] . While these methods have substantially advanced LiDAR representation learning, they primarily focus on spatial alignment. However, they largely overlook the temporal dynamics inherent in LiDAR data [ 21 , 22 ] , which are crucial for achieving robust, context-aware 3D perception in autonomous driving [ 23 , 24 ] . Recently, our preliminary work, SuperFlow [ 25 ] , introduced a semantic-flow framework for consecutive LiDAR scans, enforcing dynamic consistency and improving LiDAR representation learning. It comprises three key components: (1) View Consistency Alignment , which refines the segmentation head of vision foundation models (VFMs) [ 26 , 27 , 28 ] using predefined prompts, generating semantic-aware superpixels across camera views to align semantic flow across timestamps; (2) Dense-to-Sparse Consistency Regularization , which enforces feature consistency between dense and sparse point clouds, mitigating the impact of irregular LiDAR point distributions; and (3) Flow-Based Contrastive Learning , which incorporates spatial and temporal contrastive learning to jointly capture static structures and dynamic motion patterns for robust autonomous perception. However, several open challenges remain. First, the complexity and variability of real-world driving environments introduce factors such as sensor noise, motion blur, and adverse weather conditions, all of which can substantially degrade feature quality and hinder downstream performance [ 29 , 30 ] . Although SuperFlow [ 25 ] propagates LiDAR features across frames to enforce temporal consistency, its learned representations remain vulnerable to these disturbances. Second, most downstream pipelines continue to operate in a per-frame manner, ignoring the inherent temporal continuity of LiDAR sequences. This lack of temporal exploitation often results in inconsistent predictions across frames, ultimately limiting the robustness and generalizability of SuperFlow in challenging autonomous driving scenarios. To address these limitations, we propose SuperFlow++ , an enhanced framework that reinforces temporal modeling from pretraining to downstream tasks. Building upon our preliminary work [ 25 ] , it introduces the following extensions: • Refined Language-Guided Semantic Superpixels. While our previous work utilizes VFMs to generate semantic superpixels, SuperFlow++ enhances this process by incorporating LiDAR-camera calibration. This refinement ensures better alignment of superpixels across overlapping camera views, reducing semantic inconsistencies caused by viewpoint variations. By unifying semantic cues across multiple cameras, SuperFlow++ provides a more stable and structured representation of scene elements, improving cross-view consistency and facilitating more accurate LiDAR-camera fusion. • Cross-Sensor Temporal Contrastive Learning. To better capture the evolution of dynamic scenes, we introduce a cross-sensor temporal contrastive learning module that explicitly aligns temporal cues between cameras and LiDAR across different timestamps. Unlike prior approaches that focus solely on intra-modal temporal learning, our method enforces feature consistency across both spatial and temporal domains, ensuring that motion patterns and semantic structures remain coherent over time. This enables the model to leverage complementary information from different sensors, improving its robustness to motion-induced distortions and sensor noise in complex driving environments. • Temporal Voting for Downstream Tasks. To address the challenge of per-frame inconsistencies in downstream tasks, SuperFlow++ introduces a temporal voting strategy that aggregates semantic predictions from consecutive LiDAR frames. By integrating contextual from neighboring frames into current frame predictions, this strategy mitigates transient misclassifications, enhances robustness to occlusions, and smooths out segmentation artifacts caused by sparsity variations. This mechanism not only refines per-frame segmentation accuracy but also ensures temporal stability, which is critical for safety-critical applications such as autonomous driving. By integrating these newly proposed components into SuperFlow [ 25 ] , SuperFlow++ fosters a semantically and temporally enriched feature space, significantly improving data representation learning. Our framework demonstrates substantial performance gains over the baseline [ 25 ] across a wide range of downstream tasks, including segmentation, detection, and robustness evaluation. Furthermore, we investigate the scalability of both 2D and 3D network backbones during pretraining and evaluate their impact on semi-supervised learning, providing insights into the development of unified and scalable 3D perception models for diverse real-world applications. In summary, our key contributions are as follows: • We present SuperFlow++ , an enhanced framework that enforces temporal consistency across consecutive LiDAR scans, integrating it seamlessly into both pretraining and downstream tasks. • For pretraining, we introduce three key components: a view consistency alignment module, a dense-to-sparse consistency regularization mechanism, and a flow-based contrastive learning module. These components collectively unify the semantic feature space and enhance feature representation learning by effectively capturing temporal dynamics. • For downstream tasks, we introduce a temporal voting strategy that aggregates semantic information across consecutive

Read the complete text