Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations
Xiang Xu, Lingdong Kong, Song Wang, Chuanwei Zhou, Qingshan Liu, [0.8ex] Nanjing University of Aeronautics and Astronautics National University of Singapore Zhejiang University · 2025
Read Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations by Xiang Xu, Lingdong Kong, Song Wang, Chuanwei Zhou, Qingshan Liu, [0.8ex] Nanjing University of Aeronautics and Astronautics National University of Singapore Zhejiang University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05260.txt.
Introduction
Xiang Xu, Lingdong Kong, Song Wang, Chuanwei Zhou, and Qingshan Liu teach self-driving systems to read LiDAR point clouds without costly human labels. Earlier training methods ignored the time-based cues inside LiDAR sequences. Their framework, LiMA, borrows memory from camera images over long stretches to enrich what the LiDAR model learns.
One part aligns overlapping regions across neighboring camera views into a clean memory bank. A second part carries image features across many frames, keeping time coherence. A third part aligns memory across whole driving sequences, helping the model generalize to new places. The extra work happens only in pretraining, not in later use. On standard benchmarks, the method clearly improved both scene labeling and 3D object detection.
Catalog record
| CREATOR | Xiang Xu, Lingdong Kong, Song Wang, Chuanwei Zhou, Qingshan Liu, [0.8ex] Nanjing University of Aeronautics and Astronautics National University of Singapore Zhejiang University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations Xiang Xu, Lingdong Kong, Song Wang, Chuanwei Zhou, Qingshan Liu, [0.8ex] Nanjing University of Aeronautics and Astronautics National University of Singapore Zhejiang University Abstract LiDAR representation learning aims to extract rich structural and semantic information from large-scale, readily available datasets, reducing reliance on costly human annotations. However, existing LiDAR representation strategies often overlook the inherent spatiotemporal cues in LiDAR sequences, limiting their effectiveness. In this work, we propose LiMA , a novel long-term image-to- Li DAR M emory A ggregation framework that explicitly captures longer range temporal correlations to enhance LiDAR representation learning. LiMA comprises three key components: 1) a Cross-View Aggregation module that aligns and fuses overlapping regions across neighboring camera views, constructing a more unified and redundancy-free memory bank; 2) a Long-Term Feature Propagation mechanism that efficiently aligns and integrates multi-frame image features, reinforcing temporal coherence during LiDAR representation learning; and 3) a Cross-Sequence Memory Alignment strategy that enforces consistency across driving sequences, improving generalization to unseen environments. LiMA maintains high pretraining efficiency and incurs no additional computational overhead during downstream tasks. Extensive experiments on mainstream LiDAR-based perception benchmarks demonstrate that LiMA significantly improves both LiDAR semantic segmentation and 3D object detection. We hope this work inspires more effective pretraining paradigms for autonomous driving. The code has be made publicly accessible for future research. † † (*) Project lead. (\textrm{\Letter}) Corresponding author. 1 Introduction LiDAR sensors provide high-resolution spatial information and are essential for precise environmental perception and safe navigation [ 73 , 36 , 3 , 42 ] . However, achieving accurate perception relies on large-scale, densely labeled datasets, which are costly and labor-intensive to acquire, thereby limiting scalability in real-world applications [ 19 , 15 , 21 , 80 ] . To mitigate this challenge, recent advances in data representation learning leverage large-scale, easily accessible datasets to explore inherent semantic structures [ 15 , 86 , 88 , 51 ] . Among these approaches, image-to-LiDAR pretraining methods [ 68 , 63 , 54 , 104 ] utilize rich visual priors from RGB images to improve LiDAR feature learning. However, as illustrated in Fig. 1 (a), these methods primarily focus on spatial alignment while overlooking the temporal dynamics in LiDAR sequences. As a result, they struggle to capture motion patterns and scene evolution, which are crucial for robust perception in dynamic environments [ 27 , 45 , 90 , 22 ] . Figure 1 : Illustrative examples of image-to-LiDAR pretraining paradigms . (a) Spatial Alignment aligns LiDAR features with corresponding image features in the spatial domain without considering temporal consistency. (b) Short-Term methods propagate LiDAR features frame by frame, ensuring feature consistency across neighboring frames, but fail to capture long-term dependencies. (c) Our approach leverages Long-Term image sequences to enrich LiDAR representations, enabling a more comprehensive understanding of long-range dependencies and motion patterns. Recent works [ 49 , 60 , 96 ] incorporate short-term temporal modeling to mitigate this limitation. As shown in Fig. 1 (b), these approaches enforce feature consistency across adjacent frames, capturing local motion cues and ensuring smooth feature transitions. However, they rely primarily on frame-to-frame propagation, which maintains local consistency but lacks the capacity to model long-range dependencies and global scene transformations over extended durations. This limitation hinders the ability to construct stable, high-level representations crucial for understanding long-term motion trends and structural variations. Since LiDAR-based perception tasks inherently involve sequential data, long-term modeling is essential for capturing stable, high-level features that encode extended motion trajectories and scene transitions [ 79 , 44 , 50 ] . Unlike short-term methods that focus on immediate feature alignment, long-term modeling provides a more comprehensive understanding of dynamic environments [ 91 ] . This is particularly beneficial for tasks such as motion forecasting, behavior prediction, and autonomous decision-making [ 16 , 76 , 66 ] . To bridge this gap, we propose long-term image-to- Li DAR M emory A ggregation ( LiMA ), a simple yet effective pretraining framework that explicitly models temporal structures in driving sequences, as illustrated in Fig. 1 (c). LiMA introduces three key components to enhance LiDAR representation learning from long-term image sequences. • Cross-View Aggregation. To mitigate redundancy in overlapping camera views, this module identifies and unifies shared regions across multiple viewpoints, reinforcing spatial consistency in LiDAR representations. By aligning and fusing features, it constructs a high-quality memory bank for long-term feature propagation. • Long-Term Feature Propagation. To capture long-range dependencies, stored features in the memory bank are transformed into the ego-vehicle coordinate frame, ensuring feature alignment across frames for seamless temporal fusion. This process establishes a temporally coherent image representation, which is distilled into the LiDAR model to encode motion patterns from extended image sequences. The memory bank is dynamically updated in a first-in, first-out (FIFO) manner, enabling continuous temporal propagation across LiDAR frames. • Cross-Sequence Memory Alignment. To improve robustness across diverse driving conditions, this strategy synthesizes mixed scenes from distinct sequences, facilitating adaptation to varying environments. By maintaining structural coherence with sequence-specific memory banks, it ensures robust feature alignment and improves generalization across heterogeneous driving scenarios. By integrating these three components, LiMA effectively captures both spatial and temporal correlations within and across driving sequences. Extensive experiments on multiple 3D perception benchmarks validate its effectiveness, demonstrating substantial performance gains in both LiDAR semantic segmentation and object detection tasks. Notably, Our framework ensures high pretraining efficiency with an increased number of frames, requiring no more than 20 hours for pretraining. Furthermore, LiMA introduces no additional computational overhead during downstream tasks, ensuring efficient deployment. To summarize, this work makes contributions as follows: • We propose LiMA , a long-term image-to-LiDAR memory aggregation framework that captures extended temporal dependencies to improve LiDAR representations. • We introduce three key components: a cross-view aggregation module, a long-term feature propagation module, and a cross-sequence memory alignment strategy, which collectively enhance spatial consistency and temporal robustness across both intra- and inter-sequence domains. • Extensive experiments across multiple 3D perception benchmarks demonstrate the effectiveness of our method, achieving significant performance improvements in LiDAR semantic segmentation and object detection tasks. 2 Related Work LiDAR Scene Understanding. LiDAR sensors provide high-precision 3D environmental representations essential for autonomous driving [ 9 , 35 , 8 , 41 ] . However, the sparsity and irregularity of LiDAR point clouds pose challenges for accurate perception. To address this, existing methods transform point clouds into various representations, including raw points [ 64 , 65 , 26 , 77 , 71 , 70 ] , bird’s-eye view (BEV) [ 105 , 109 , 39 , 7 ] , range images [ 56 , 92 , 1 , 32 , 97 ] , sparse voxels [ 12 , 75 , 110 , 24 , 25 ] , and multi-view formats [ 93 , 111 , 48 , 11 , 100 , 94 , 62 ] . While these methods achieve strong performance, they rely heavily on large-scale labeled datasets, which are expensive and time-consuming to obtain. To mitigate this challenge, recent studies have explored semi-supervised [ 34 , 38 , 40 , 82 ] and weakly-supervised [ 46 , 78 , 95 , 55 , 81 ] learning strategies to reduce