SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving
Philipp Wolters, Johannes Gilg, Torben Teepe, Gerhard Rigoll · 2025
Read SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving by Philipp Wolters, Johannes Gilg, Torben Teepe, Gerhard Rigoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10567.txt.
Introduction
Philipp Wolters and co-authors add radar to self-driving car perception in this 2025 paper. Systems that drive end to end, from sensors to steering in one model, promise better results, but camera-only setups struggle in bad weather, blocked views, and judging speed. Radar answers some of those gaps, since it senses motion directly and cares little for rain or fog.
Their system, SpaRC-AD, fuses camera and radar in a query-based design. Sparse three-dimensional feature matching and radar-based speed estimates build a scene model used to refine object anchors, map lines, and motion forecasts. Against camera-only rivals it lifted detection, tracking, and mapping scores and cut prediction and planning errors. Tests covered city driving data and a closed-loop simulator.
Catalog record
| CREATOR | Philipp Wolters, Johannes Gilg, Torben Teepe, Gerhard Rigoll |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10567
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving Philipp Wolters, Johannes Gilg, Torben Teepe, Gerhard Rigoll Abstract End-to-end autonomous driving systems promise stronger performance through unified optimization of perception, motion forecasting, and planning. However, vision-based approaches face fundamental limitations in adverse weather conditions, partial occlusions, and precise velocity estimation – critical challenges in safety-sensitive scenarios where accurate motion understanding and long-horizon trajectory prediction are essential for collision avoidance. To address these limitations, we propose SpaRC-AD , a query-based end-to-end camera-radar fusion framework for planning-oriented autonomous driving. Through sparse 3D feature alignment and Doppler-based velocity estimation, we achieve strong 3D scene representations for refinement of agent anchors, map polylines, and motion modelling. Our method achieves strong improvements over the state-of-the-art vision-only baselines across multiple autonomous driving tasks, including 3D detection ( + 4.8 % +4.8\% mAP), multi-object tracking ( + 8.3 % +8.3\% AMOTA), online mapping ( + 1.8 % +1.8\% mAP), motion prediction ( − 4.0 % -4.0\% mADE), and trajectory planning ( − 0.1 -0.1 m L2 and − 9 % -9\% TPC). We achieve both spatial coherence and temporal consistency on multiple challenging benchmarks, including real-world open-loop nuScenes , long-horizon T-nuScenes, and closed-loop simulator Bench2Drive . We show the effectiveness of radar-based fusion in safety-critical scenarios where accurate motion understanding and long-horizon trajectory prediction are essential for collision avoidance. The source code of all experiments is available at https://phi-wol.github.io/sparcad/ 1 Introduction Figure 1 : We propose SpaRC-AD , a query-based camera-radar fusion framework for autonomous driving that jointly optimizes perception, prediction and planning. Autonomous driving systems have evolved from modular, multi-stage perception pipelines to unified end-to-end learning frameworks that directly map raw sensor inputs to vehicle control commands [ 2 ] . While conventional approaches decompose the driving task into independent modules for 3D object detection [ 14 , 52 , 47 ] , multi-object tracking [ 57 , 60 , 53 , 46 ] , and online mapping [ 33 , 29 , 56 ] , recent end-to-end methods [ 13 , 59 ] demonstrate the advantages of joint optimization across perception, prediction, and planning tasks. The new optimization objective is to generate driving controls and trajectories for the ego vehicle, directly from sensor inputs of cameras, LiDARs, and radars [ 4 ] . Leveraging expert demonstrations through imitation learning, raw sensor signals are directly processed to output vehicle motion plans and intermediate representations optimized towards the final planning goal. Initially in Bird’s Eye View (BEV) representations [ 34 ] , the future trajectory of the ego vehicle is regressed from an ego-token within a transformer decoder, reducing the problem to a supervised learning setting [ 44 ] . However, state-of-the-art research has focused on vision-centric approaches, limiting their robustness in challenging scenarios such as adverse weather conditions, partial occlusions, and long-range detection. Critical for planning safety: robust depth estimation, strong motion-forecasting, stable trajectories. Song et al. have shown that especially in turning scenarios, models suffer from unstable trajectories, vulnerability to occlusions, and temporal inconsistencies [ 43 ] . The implicit depth modeling in query-based transformers lacks geometric constraints, leading to substantial localization errors in 3D perception due to unreliable depth estimation [ 48 ] . Due to noise from highly dynamic environments and following detection errors, uncertainties arise in long-time horizon and long-range planning. Moreover, causal confusion and the reliance on temporal smoothness of the ego trajectory and past motion pose a challenge [ 28 ] . Radar sensors provide critical advantages that address fundamental limitations of vision-centric approaches in end-to-end autonomous driving. Their robust long-range detection capabilities beyond 150m, direct velocity measurements through Doppler effects, and weather-independent operation enable more reliable spatial reasoning through time-of-flight range measurements [ 48 , 7 , 62 ] . Additionally, radar’s ability to measure relative velocities enhances multi-agent intent prediction, leading to more stable and consistent trajectory planning. These complementary strengths make radar fusion particularly valuable for safety-critical autonomous driving applications. While multi-modal fusion with cameras and LiDAR has shown benefits [ 4 ] , and radar fusion has proven effective for modular perception [ 32 , 49 , 5 ] , the integration of radar into end-to-end autonomous driving remains unexplored. We investigate the impact and potential of including radar into the end-to-end optimization and how to leverage the additional motion cues reflected from the environment. Due to sparsity of the radar representation and precise spatial-temporal calibration, we propose a query-based approach that iteratively refines the motion and positional charactersitcs of map and traffic agent representations. In this work, we address the critical gap in radar-based end-to-end autonomous driving by proposing SpaRC-AD , extending the sparse representation paradigm of radar points and scene instances in a coherent end-to-end framework, and creating synergies between radar data characteristics and planning requirements. Our approach iteratively refines motion and positional characteristics of both map and agent representations by leveraging spatial proximity of reflected radar points as strong inductive biases. Our main contributions are: • First radar-based end-to-end autonomous driving baseline on key benchmarks. • Extension of sparse fusion design for simultaneous detection, tracking, and planning queries. • Holistic radar-based fusion improves 3D detection (+4.8% mAP), multi-object tracking (+8.3% AMOTA), online mapping (+1.8% mAP), and motion forecasting (-4.0% mADE), optimizing trajectory prediction consistency (-9.0% TPC) and simulation success rates (+10.0%). • Extensive evaluation on multiple benchmarks of open-loop nuScenes [ 1 ] and closed-loop simulation of Bench2Drive [ 17 ] . • We provide additional qualitative analysis demonstrating superior performance through enhanced perception range, more accurate motion modeling, and increased robustness under challenging environmental conditions. 2 Related Work 2.1 Planning Oriented Autonomous Driving A new paradigm has emerged in autonomous driving research, moving from multi-stage frameworks [ 25 , 16 , 26 ] to end-to-end autonomous driving [ 2 ] . This evolution addresses the fundamental limitations of modular approaches: information loss and error accumulation across subsequent stages, which constrain optimal system performance. The goal is to strengthen generalization to complex driving scenarios in a data-driven manner. Typically the state-of-the-art methods follow an encoder-decoder principle, first encoding the sensor data into a latent representation, then decoding the intermediate representation into a driving policy [ 4 , 12 ] . The pioneering works of UniAD [ 13 ] and VAD [ 19 , 3 ] have shown that all tasks are communicated within unified query interfaces, enabling goal-oriented optimization through vectorized scene representations. VADv2 [ 3 ] extends the planner to probabilistic planning, while Hydra-MDP [ 27 ] integrates additional supervision from rule-based planning modules. SparseDrive [ 44 ] explores sparse scene representations for efficient scene modeling, discarding Birds-Eye-View (BEV) representations. 2.2 Camera-Radar 3D Perception In 3D object detection, radar-camera-based approaches have emerged as a low-cost and robust alternative to lidar-based perception. Initial works fused in the perspective view [ 36 , 22 , 37 , 35 ] , associating the sparsely projected radarpoints to the dense encoded image features. Grid-rendering approaches have adapted the BEVFusion [ 34 ] paradigm to the characteristics of radar sensors [ 49 , 23 , 32 , 21 , 20 , 41 , 40 ] , by encoding them with PointPillars [ 24 ] or Voxels [ 63 ] , that are densely parametrized, but sparse