Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
RAJENDRAMAYAVAN SATHYAM \authorrefmark 1 1, YUEQI LI \authorrefmark 1 1 · 2025
Read Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities by RAJENDRAMAYAVAN SATHYAM \authorrefmark 1 1, YUEQI LI \authorrefmark 1 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08302.txt.
Introduction
Rajendramayavan Sathyam and Yueqi Li survey how foundation models are changing self-driving perception in a 2025 review. The field is moving from narrow models, each built for one task, to broad models trained on huge, mixed data sets. Three flaws still block the road: weak general skill, hard scaling, and poor grace when road scenes drift from the training mix.
The survey sorts the field by four skills a car must have. It must hold broad knowledge. It must grasp space. It must fuse many sensors with grace. And it must reason over time. Each skill is checked against real driving needs. The frame gives engineers a plain test: is this model road-ready, or just large?
Catalog record
| CREATOR | RAJENDRAMAYAVAN SATHYAM \authorrefmark 1 1, YUEQI LI \authorrefmark 1 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 119 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities RAJENDRAMAYAVAN SATHYAM \authorrefmark 1 1, YUEQI LI \authorrefmark 1 1 Abstract Foundation models are revolutionizing autonomous driving perception, transitioning the field from narrow, task-specific deep learning models to versatile, general-purpose architectures trained on vast, diverse datasets. This survey examines how these models address critical challenges in autonomous perception, including limitations in generalization, scalability, and robustness to distributional shifts. The survey introduces a novel taxonomy structured around four essential capabilities for robust performance in dynamic driving environments: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal reasoning. For each capability, the survey elucidates its significance and comprehensively reviews cutting-edge approaches. Diverging from traditional method-centric surveys, our unique framework prioritizes conceptual design principles, providing a capability-driven guide for model development and clearer insights into foundational aspects. We conclude by discussing key challenges, particularly those associated with the integration of these capabilities into real-time, scalable systems, and broader deployment challenges related to computational demands and ensuring model reliability against issues like hallucinations and out-of-distribution failures. The survey also outlines crucial future research directions to enable the safe and effective deployment of foundation models in autonomous driving systems. \receiveddate XX Month, XXXX \reviseddate XX Month, XXXX \accepteddate XX Month, XXXX \publisheddate XX Month, XXXX \currentdate 27 June, 2024 \doiinfo OJVT.2024.0627000 {IEEEkeywords} Autonomous Driving, Foundation Models, Self-supervised Learning, Sensor Fusion, Vision Language Models, 3D Reconstruction, Diffusion Models † † footnotetext: * Equal contribution 1 INTRODUCTION \IEEEPARstart Autonomous driving perception constitutes a cornerstone of intelligent transportation systems, as it is responsible for enabling vehicles to interpret and respond to their surrounding environments in real time. The perception stack involves a suite of tasks such as object detection, semantic segmentation, and object tracking—each of which is essential for safe navigation and effective decision-making [ 1 , 2 , 3 ] . Traditionally, these tasks have been tackled using narrowly focused deep learning models designed explicitly for individual perception tasks, which are trained on meticulously curated, manually annotated datasets. While achieving impressive performance within controlled scenarios, such specialized models inherently suffer from limited scalability and poor generalizability. They frequently experience significant performance degradation when exposed to long-tailed scenarios, which include infrequent yet safety-critical events, or rare occurrences insufficiently represented in training data [ 4 ] . Moreover, these models often struggle to adapt effectively when faced with distributional shifts arising from changes in environmental conditions, sensor configurations, or operational contexts. Recent breakthroughs in the development of foundation models have initiated a transformative shift in autonomous driving perception. Foundation models refer to large-scale, general-purpose neural networks that are pre-trained on extensive, diverse datasets using self-supervised or unsupervised learning strategies [ 5 ] . Typically, these models are built by leveraging architectures such as transformers, which effectively model complex interactions in data, allowing them to learn universal representations and capture latent knowledge about the world without explicit supervision. The pre-training process involves exposing these models to vast quantities of data from multiple modalities—text, images, videos, and potentially other sensor data—enabling them to acquire generalized features useful across various downstream tasks. In the context of autonomous driving, foundation models provide significant advantages due to their inherent capability for broad generalization, efficient transfer learning, and reduced reliance on task-specific annotated datasets. By possessing extensive general knowledge encoded during the pre-training phase, these models can more readily adapt to diverse and dynamic real-world driving conditions, improving performance in scenarios beyond the reach of traditional supervised approaches. Moreover, the unified representation learned by foundation models facilitates seamless integration across perception tasks, promoting consistency and coherence in interpreting complex driving environments. Foundation models need extensive data and large-scale architectures to encode broad and generalizable knowledge. In the context of autonomous driving, it is important not only to scale model size and data volume, but also to apply specialized techniques and deliberate training strategies. These methods help guide the model to acquire capabilities that are particularly relevant to the unique challenges of autonomous vehicle perception. Recognizing this need, this paper identifies and elaborates on four critical aspects that are particularly essential for constructing and refining foundation models tailored specifically for autonomous driving perception: 1. Generalized knowledge. The model should be capable of adapting to a broad range of driving scenarios, including rare or previously unseen situations. It must infer plausible outcomes and reason about unfamiliar agents in a principled manner. 2. Spatial awareness. An effective model must possess a strong understanding of 3D spatial structure and relationships. This includes detecting both known and unknown objects, as well as reasoning about their physical interactions and future trajectories. 3. Multi-sensor robustness. The system should maintain high performance across a wide spectrum of environmental conditions, such as varying weather, illumination, and sensor noise. Robustness to partial sensor failures is also critical for ensuring safety. 4. Temporal understanding. Beyond instantaneous perception, the model must capture temporal dependencies and anticipate future states of the environment. This entails modeling motion patterns, recognizing occluded agents, and reasoning about object permanence. Figure 1: Overview of the four key pillars for foundation models in autonomous driving and the corresponding methods to achieve them. For each of these aspects, we first clearly outline its significance in the context of foundation models for autonomous driving perception. Subsequently, we provide a comprehensive analysis of the underlying key techniques and methodologies employed to effectively develop and enhance the respective capabilities. These techniques are illustrated in Figure 1 . We provide abundant examples throughout to clarify their practical implementation and highlight their relevance in real-world autonomous driving scenarios. Finally, we critically analyze current challenges associated with improving these aspects and propose potential directions for future research. As summarized in Table 1 , existing surveys on foundation models in autonomous driving predominantly adopt method-based or task-based taxonomies [ 6 , 7 , 8 , 9 , 10 , 11 ] . While valuable, these approaches often focus on a wide array of applications beyond perception, such as prediction and planning. Consequently, they do not provide a comprehensive treatment of all the core capabilities essential for robust perception; critical aspects like multi-sensor robustness and spatial awareness are frequently overlooked. In contrast, our survey introduces a capability-based framework tailored specifically to the perception domain. By structuring the discussion around the four fundamental capabilities we have identified, our taxonomy offers a more focused and in-depth analysis. This approach not only clarifies the essential components of perception-centric foundation models but also provides a systematic guide for researchers to identify and address specific deficiencies, thereby fostering targeted innovation in this critical area. Table 1: Survey Papers on Foundation Models for Autonomous Driving Survey Taxonomy Covered Capabilities Task(s) in Autonomous driving Forging Vision Foundation Models for Autonomous Driving [ 6 ] Method‑based Generalized knowledge, Multi‑sensor robustness, Spatial awareness, Temporal understanding Perception, simulation Applications of Large Scale Foundation Models for Autonomous Driving [ 7 ] Method‑based & task‑based Generalized knowledge, Spatial awareness, Temporal understanding Perception, prediction, planning/control, end‑to‑end, simulation A Survey for Foundation Models in Autonomous Driving [ 8 ] Method‑based & task‑based Generalized knowledge, Temporal understanding Perception, prediction, planning/control, end‑to‑end, simulation Prospective Role of Foundation Models in Advancing Autonomous Vehicles [ 9 ] Method‑based & task‑based Generalized knowledge, Temporal understanding Perception, Prediction, planning/control, simulation, end‑to‑end LLM4Drive: A