nuCarla: A nuScenes-Style Bird’s-Eye View Perception Dataset for CARLA Simulation

Zhijie Qiao, Zhong Cao, Henry X. Liu, [0.5em] Civil, Environmental Engineering, University of Michigan · 2025

Read nuCarla: A nuScenes-Style Bird’s-Eye View Perception Dataset for CARLA Simulation by Zhijie Qiao, Zhong Cao, Henry X. Liu, [0.5em] Civil, Environmental Engineering, University of Michigan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13744.txt.

Introduction

Zhijie Qiao, Zhong Cao, and Henry X. Liu release nuCarla, a bird's-eye view perception dataset for self-driving built inside the CARLA simulator. Closed-loop driving models need interactive data where perception, planning, and control train together, but real datasets are recorded passively and cannot react. That gap leaves closed-loop models behind even simple rule-based drivers.

nuCarla mirrors the popular nuScenes format at comparable scale, so real-world perception models transfer directly, and its class balance is more even. Because it lives in simulation, the data plug straight into closed-loop deployment and testing. Strong perception backbones supplied with the release reach top detection results, giving the field an open benchmark for interactive driving research.

Catalog record

CREATORZhijie Qiao, Zhong Cao, Henry X. Liu, [0.5em] Civil, Environmental Engineering, University of Michigan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13744

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

nuCarla: A nuScenes-Style Bird’s-Eye View Perception Dataset for CARLA Simulation Zhijie Qiao, Zhong Cao, Henry X. Liu, [0.5em] Civil, Environmental Engineering, University of Michigan Abstract End-to-end (E2E) autonomous driving heavily relies on closed-loop simulation, where perception, planning, and control are jointly trained and evaluated in interactive environments. Yet, most existing datasets are collected from the real world under non-interactive conditions, primarily supporting open-loop learning while offering limited value for closed-loop testing. Due to the lack of standardized, large-scale, and thoroughly verified datasets to facilitate learning of meaningful intermediate representations, such as bird’s-eye-view (BEV) features, closed-loop E2E models remain far behind even simple rule-based baselines. To address this challenge, we introduce nuCarla, a large-scale, nuScenes-style BEV perception dataset built within the CARLA simulator. nuCarla features (1) full compatibility with the nuScenes format, enabling seamless transfer of real-world perception models; (2) a dataset scale comparable to nuScenes, but with more balanced class distributions; (3) direct usability for closed-loop simulation deployment; and (4) high-performance BEV backbones that achieve state-of-the-art detection results. By providing both data and models as open benchmarks, nuCarla substantially accelerates closed-loop E2E development, paving the way toward reliable and safety-aware research in autonomous driving. 1 Introduction In the field of autonomous driving, end-to-end (E2E) systems have attracted increasing attention. UniAD 14 represents an influential milestone, proposing a transformer-based architecture that unifies perception, prediction, and planning through a query-driven design, achieving state-of-the-art performance on the large-scale nuScenes 3 dataset. Subsequent works such as VAD 19 and UAD 12 further improved the overall architectural scheme and established new records on open-loop prediction tasks. Despite these advances, studies have highlighted that improvements in open-loop do not necessarily translate to better performance in closed-loop evaluation 44 ; 2 . In open-loop, the ego agent is reset to the ground truth position at each frame, breaking the causal relationship between actions and outcomes. In contrast, closed-loop evaluation enforces continuous control, where minor disturbances can accumulate and cause the agent to drift off the track 33 . Figure 1 : Maps of nine CARLA towns with traffic in the nuCarla dataset, shown under diverse weather conditions. Since real-world testing is risky, inefficient, and often non-reproducible, closed-loop training and evaluation for autonomous vehicles are extensively conducted on simulation environments. Although there have been advances in open-loop E2E modeling, primarily driven by non-interactive real-world datasets 3 ; 10 ; 41 , these developments have not yet been reflected in the simulation domain. To date, there remains a lack of standardized, thoroughly verified, and large-scale datasets specifically designed for simulation-based closed-loop research. Among existing platforms, CARLA 9 has become widely used for autonomous driving simulation, offering a high-fidelity, physics simulator with diverse sensor suites and realistic environments. The CARLA autonomous driving leaderboard 4 provides an open framework for evaluating autonomous agents on predefined routes. Despite its popularity, the leaderboard has been criticized for focusing on basic driving skills and lacking rigorous evaluation under complex conditions 5 . To address this, Bench2Drive 17 introduced a large-scale dataset with well-defined metrics and provided pretrained E2E models based on UniAD 14 and VAD 19 architectures. However, the reported driving success rates (SR) for these models remain notably low, at only 16.36% and 15.00%, respectively. Subsequent works have improved performance on the benchmark. For example, MomAD 35 achieved an SR of 16.71%, VeteranAD 45 33.85%, DriveTransformer 18 35.01%, and Orion 11 54.62%. Nevertheless, even the best-performing framework, Orion, which leverages the advanced Vision-Language Model (VLM) Qwen2 38 for scene understanding and navigation reasoning, succeeds in only about half of the scenarios, each lasting merely 20 seconds. In contrast, a simple rule-based algorithm, PDM-Lite 34 , originally designed for Graph Visual Question Answering tasks, achieves a remarkable 92.27% SR when utilizing ground-truth perception information 1 . We argue that the suboptimal performance of existing models does not necessarily stem from the E2E paradigm itself, but rather from limitations in the available data. Most simulation datasets provide only raw sensor inputs and direct vehicle control outputs, restricting E2E systems’ ability to learn meaningful intermediate representations, such as bird’s-eye-view (BEV) features, which are critical for improved generalization and stability . Yet, such intermediate-level datasets and pretrained perception backbones remain largely unexplored in the literature, making it difficult to develop robust closed-loop systems. To address this gap, we introduce nuCarla, a nuScenes-style, camera-based BEV perception dataset built within the CARLA simulator. Following the standard nuScenes protocol, nuCarla contains 1,000 driving scenarios (700 for training, 150 for validation, and 150 for testing), each consisting of 40 frames sampled at 0.5-second intervals. The dataset strictly aligns with nuScenes in naming conventions, annotation structure, file hierarchy, and API compatibility, enabling direct transfer of existing BEV models to the CARLA environment without modification. To thoroughly validate the nuCarla dataset, we train and evaluate four state-of-the-art BEV perception models: BEVFormer 25 ; 42 , PETR 26 ; 27 , BEVDet 15 ; 16 , and FastBEV 22 . All models achieve stable convergence and demonstrate strong detection performance on the nuCarla validation set, as measured by the official nuScenes metrics. We release the full dataset along with pretrained weights for each BEV architecture. In the same way that modern perception frameworks adopt ResNet 13 or VoVNet 20 as standard visual backbones, we envision nuCarla as a BEV-level perception backbone for the development of robust E2E autonomous driving systems. Finally, we resolve version conflicts by upgrading the legacy mmdetection3d-1.0 7 frameworks (widely adopted in perception models and E2E systems) to ensure full compatibility with the latest PyTorch and GPU architectures. This resolves a persistent pain point in the research community 37 ; 32 ; 31 . The main contributions of this work are as follows: 1. We provide a nuScenes-style BEV perception dataset in the CARLA simulator to facilitate development of perception models. 2. We validate the dataset by training four BEV models, achieving competitive performance under the official nuScenes metrics. We also upgrade the legacy mmdetection3d framework to be compatible with the latest PyTorch and GPU architectures. 3. We release pretrained weights for all evaluated BEV architectures, providing strong perception backbones to support future E2E autonomous driving research. 2 Related Work 2.1 Open-Loop Trajectory Prediction Early E2E autonomous driving frameworks are primarily implemented and evaluated on open-loop trajectory prediction tasks. UniAD 14 introduces a unified transformer-based architecture that formulates perception, prediction, and planning as interdependent query-based tasks, thereby mitigating error propagation across subtasks and achieving leading results on multiple benchmarks of the nuScenes 3 dataset. VAD 19 proposes a vectorized approach that replaces rasterized inputs with instance-level vector representations, modeling agents and map features as explicit geometric entities for improved interpretability and inference speed. GenAD 46 is a generative framework that produces driving plans from raw sensor inputs by encoding scenes into instance tokens, learning trajectory priors in a latent space, and modeling agent and ego dynamics. Para-Drive 40 explores differentiable modular architectures and designs a fully parallel structure that enhances safety and runtime. Hydra-MDP 24 uses knowledge distillation from both human and rule-based experts and generates diverse trajectory candidates through a multi-head decoder that accounts for multiple evaluation metrics; this method achieved first place in the Navsim challenge 8 . Hydra-MDP++ 21 extends Hydra-MDP 24 by introducing richer behavioral evaluation metrics

Read the complete text