GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Driving
William Ljungbergh, Adam Lilja, Adam Tonderski, Arvid Laveno Ling, Carl Lindström, Willem Verbeke, Junsheng Fu, Christoffer Petersson, Lars Hammarstrand, Michael Felsberg · 2025
Read GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Driving by William Ljungbergh, Adam Lilja, Adam Tonderski, Arvid Laveno Ling, Carl Lindström, Willem Verbeke, Junsheng Fu, Christoffer Petersson, Lars Hammarstrand, Michael Felsberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15672.txt.
Introduction
Self-driving cars produce huge streams of data about roads over space and time. Ljungbergh and co-authors ask whether a model can learn the structure of that world by predicting it, the way language models learn text. Their method, GASP, pre-trains a model to answer three things for any future point in space and time. It predicts whether the spot is occupied, whether the car's own path passes there, and what high-level features a vision model sees there.
By predicting occupancy fields rather than raw sensor data, the model builds a unified picture of how a scene is laid out and how it changes. The authors tested GASP on several driving benchmarks. It improved semantic occupancy forecasting, online mapping, and prediction of the car's own trajectory. They argue that continuous four-dimensional prediction of this kind is a scalable way to pre-train systems for driving.
Catalog record
| CREATOR | William Ljungbergh, Adam Lilja, Adam Tonderski, Arvid Laveno Ling, Carl Lindström, Willem Verbeke, Junsheng Fu, Christoffer Petersson, Lars Hammarstrand, Michael Felsberg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Driving William Ljungbergh, Adam Lilja, Adam Tonderski, Arvid Laveno Ling, Carl Lindström, Willem Verbeke, Junsheng Fu, Christoffer Petersson, Lars Hammarstrand, Michael Felsberg Abstract Self-supervised pre-training based on next-token prediction has enabled large language models to capture the underlying structure of text, and has led to unprecedented performance on a large array of tasks when applied at scale. Similarly, autonomous driving generates vast amounts of spatiotemporal data, alluding to the possibility of harnessing scale to learn the underlying geometric and semantic structure of the environment and its evolution over time. In this direction, we propose a g eometric a nd s emantic self-supervised p re-training method, GASP, that learns a unified representation by predicting, at any queried future point in spacetime, (1) general occupancy, capturing the evolving structure of the 3D scene; (2) ego occupancy, modeling the ego vehicle path through the environment; and (3) distilled high-level features from a vision foundation model. By modeling geometric and semantic 4D occupancy fields instead of raw sensor measurements, the model learns a structured, generalizable representation of the environment and its evolution through time. We validate GASP on multiple autonomous driving benchmarks, demonstrating significant improvements in semantic occupancy forecasting, online mapping, and ego trajectory prediction. Our results demonstrate that continuous 4D geometric and semantic occupancy prediction provides a scalable and effective pre-training paradigm for autonomous driving. For code and additional visualizations, see our project page . 1 1 footnotetext: Denotes equal contribution. 1 Introduction Autonomous driving (AD) has the potential to improve safety, accessibility, and enhance transportation efficiency. For an autonomous vehicle (AV) to operate safely and effectively, it must have a comprehensive understanding of its environment and the evolution thereof. In doing so, the AV must learn to reason about geometry and semantics in a dynamic environment. Figure 1 : GASP learns a structured, generalizable representation of the environment and its evolution and can be further trained to perform well on downstream AD tasks. We outperform SotA pre-training UnO [ 2 ] across the board, especially on primarily semantic tasks like map segmentation. No pre-training is displayed for reference. Downstream tasks requiring additional labels are post-trained using 1000 samples ( \sim 1% of pre-training scale). To develop a comprehensive understanding of the environment, most existing systems rely heavily on large datasets with human-labeled annotations. Annotations are essential for solving tasks such as object detection and forecasting [ 61 , 26 ] , online mapping [ 33 ] , and to enable multi-task frameworks with ego trajectory planning [ 24 , 25 , 13 ] . Unlabeled data is typically abundant when developing AD systems, but annotating a sufficiently diverse dataset is prohibitively expensive, limiting scalability of annotation reliant methods. In other domains, e.g . , natural language processing, self-supervised predictive learning over large datasets has been highly successful [ 43 , 15 , 9 ] . Researchers have explored predictive learning for AD, e.g . by predicting future point clouds [ 27 , 60 ] , occupancy [ 64 , 24 ] , or video [ 58 ] . These methods have shown promise but may struggle to model the continuous and dynamic nature of the driving environment, as they focus on predicting sensor observations rather than the underlying structure of the world. Recent works [ 1 , 2 ] address this by learning a representation in continuous spacetime. By predicting future occupancy from past lidar data, these methods offer a more accurate model of the inherently continuous real world. However, while future occupancy prediction provides strong geometric and temporal cues, it lacks the semantic richness needed comprehensive scene understanding and complex reasoning in downstream tasks. To overcome this limitation, we propose GASP, a self-supervised pre-training method that integrates multiple sources of readily available signals in AV development: Future lidar scans, camera images, and ego poses. By leveraging supervision from diverse sensor modalities, our method results in a richer representation of the environment and improves geometric, temporal, and semantic understanding. Specifically, GASP learns to predict occupancy, ego-path, and features from a vision foundation model (VFM) in a continuous 4D (3D + time) representation. The learned representation is useful on an array of downstream AD tasks, outperforming prior works as illustrated in Fig. 1 . Additionally, we introduce and demonstrate the efficacy of practical improvements: 1) harvesting, negative information [ 31 ] , from missing lidar rays for additional supervision, and 2) a rotation augmentation strategy that significantly improves model generalization. Our main contributions are: • Propose a self-supervised pre-training method, GASP, designed to learn a structured, generalizable 4D representation in continuous time by integrating geometric, temporal, and semantic supervision from multiple readily available signals. • Demonstrate that GASPpre-training leads to improved generalization across multiple downstream autonomous driving tasks, significantly outperforming uni-modal pre-training on tasks such as semantic occupancy forecasting, online mapping, and ego-trajectory prediction. • Provide open-source code, including custom CUDA kernels for accelerated query generation and reimplementation of previously closed-source baselines, to facilitate further research in self-supervised learning for AD. 2 Related work Figure 2 : Overview of GASP. Past lidar scans are encoded into a BEV feature map. These features are used by implicit decoders to predict DINOv2 features \hat{\mathcal{D}} , occupancy \hat{\mathcal{O}} , and ego-path \hat{\mathcal{E}} at the query points \mathcal{Q} generated from future sensor data during pre-training. We also show that the learned representation is useful when transferred to an array of downstream AD tasks. Self-supervised learning has gained traction due to its ability to capture meaningful patterns without requiring expensive labels [ 12 , 39 , 45 ] , enabling greater scalability. We apply these ideas to AD and provide an overview of the most relevant developments. Generative methods: Generative methods withhold or alter parts of an input data sample and aim to reconstruct this part from the remaining data. Such methods learn features that generalize across a multitude of tasks. Masked input models have been applied to text [ 15 , 8 , 44 ] , images [ 42 , 19 , 5 , 6 , 16 ] , videos [ 50 ] and point clouds [ 20 , 62 , 41 , 37 ] . These methods have been tailored to AD by jointly encoding multiple sensors and recovering masked inputs by neural rendering techniques [ 56 , 63 ] . Predicting future raw sensory data, such as point cloud forecasting [ 27 , 60 ] and video frame forecasting [ 58 ] , as a pre-training step can also be seen in the light of masked input modeling. While such models learn relevant patterns in the data, they are also forced to learn details that are irrelevant for AD tasks: sensor intrinsics such as the scan pattern of a lidar, and low-level stochastic information such as the lighting of each reconstructed pixel. Implicit generative methods: Alternatively, sensory data forecasting can be rephrased as generic occupancy forecasting [ 2 ] . This has two advantages compared to direct generative methods: Future occupancy depends on the dynamics of the environment but not on that of the sensors, and occupancy is directly useful for downstream tasks in AD. By encoding past sensory information ( e.g . , lidar [