A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator
Elahe Delavari, Feeza Khan Khanzada, Jaerock Kwon · 2025
Read A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator by Elahe Delavari, Feeza Khan Khanzada, Jaerock Kwon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08221.txt.
Introduction
Reinforcement learning for self-driving, as practised inside one popular simulator, is surveyed by Elahe Delavari and 2 co-authors in a 2025 review. Around one hundred peer-reviewed papers that train, test, or validate driving policies in the open-source CARLA simulator are analysed systematically, because the field lacked a clear account of its own habits.
Papers are sorted by algorithmic family: model-free, model-based, hierarchical, and hybrid. More than four in five studies still rely on model-free methods. Benchmarking and evaluation practices are charted alongside the algorithms. A map like this shows newcomers where the crowded roads are, and it exposes how much of the field's confidence rests on a single simulator's version of reality.
Catalog record
| CREATOR | Elahe Delavari, Feeza Khan Khanzada, Jaerock Kwon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator Elahe Delavari, Feeza Khan Khanzada, Jaerock Kwon Abstract Autonomous‑driving research has recently embraced deep Reinforcement Learning (RL) as a promising framework for data‑driven decision making, yet a clear picture of how these algorithms are currently employed, benchmarked and evaluated is still missing. This survey fills that gap by systematically analysing around 100 peer‑reviewed papers that train, test or validate RL policies inside the open‑source CARLA simulator. We first categorize the literature by algorithmic family—model‑free, model‑based, hierarchical, and hybrid—and quantify their prevalence, highlighting that more than 80% of existing studies still rely on model‑free methods such as DQN, PPO and SAC. Next, we explain the diverse state, action and reward formulations adopted across works, illustrating how choices of sensor modality (RGB, LiDAR, BEV, semantic maps, and carla kinematics states), control abstraction (discrete vs. continuous) and reward shaping are used across various literature. We also consolidate the evaluation landscape by listing the most common metrics (success rate, collision rate, lane deviation, driving score) and the towns, scenarios and traffic configurations used in CARLA benchmarks. Persistent challenges including sparse rewards, sim‑to‑real transfer, safety guarantees and limited behaviour diversity are distilled into a set of open research questions, and promising directions such as model‑based RL, meta‑learning and richer multi‑agent simulations are outlined. By providing a unified taxonomy, quantitative statistics and a critical discussion of limitations, this review aims to serve both as a reference for newcomers and as a roadmap for advancing RL‑based autonomous driving toward real‑world deployment. keywords Reinforcement Learning, Autonomous Driving, CARLA Simulator † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Autonomous driving represents one of the most transformative technological frontiers of the 21st century, with the potential to revolutionize transportation by enhancing safety, efficiency, and accessibility. By reducing human error—widely recognized as the leading cause of traffic accidents National Highway Traffic Safety Administration 2015 — Autonomous Vehicles (AVs) promise safer roads, reduced congestion, and improved mobility. However, the development of robust and reliable AV systems remains a formidable challenge, primarily due to the complexity of real-world driving environments. These environments present dynamic traffic patterns, intricate multi-agent interactions, and rare but critical long-tail scenarios that traditional rule-based approaches struggle to handle effectively. Reinforcement Learning (RL) has emerged as a powerful paradigm for training autonomous driving policies, offering an adaptive framework where agents learn by interacting with their environment and refining their decision-making strategies over time Sallab et al. 2017 . Unlike conventional supervised learning approaches that require extensive labeled datasets, RL enables AVs to optimize actions dynamically based on rewards, making it particularly suited for sequential decision-making tasks such as lane-keeping, obstacle avoidance, intersection handling, and route planning. Despite its potential, deploying RL-based autonomous systems in the real world poses several challenges, including safety risks, computational demands, and the need for active environmental interaction for training. To address these challenges, simulation platforms have become an indispensable tool for RL research in autonomous driving. Among these, CARLA (Car Learning to Act) Dosovitskiy et al. 2017 has established itself as a premier open-source simulator designed specifically for AV research. Built on Unreal Engine, CARLA provides high-fidelity urban and highway driving environments, supporting diverse road layouts, dynamic traffic conditions, various weather scenarios, and an extensive suite of sensors (e.g., RGB cameras, LiDAR, depth sensors, and semantic segmentation). This flexibility enables safe, controlled, and scalable RL training, allowing researchers to benchmark and refine driving policies before real-world deployment. Over the years, CARLA has become a cornerstone for RL-driven autonomous driving research, fostering advancements across multiple dimensions. Researchers have leveraged a wide array of RL techniques Wu et al. 2022 ; Goel and Chauhan 2021 ; Youssef and Houda 2019 ; Carton et al. 2021 ; Mohammed et al. 2024 ; Zhang and Nakamoto 2022 ; Yang et al. 2022 ; Wang et al. 2023 ; Wei et al. 2023 ; Zhu et al. 2023 ; Li et al. 2024 ; Pan et al. 2023 ; Cai et al. 2021 ; Chen et al. 2021 ; Wang et al. 2024 ; Huang et al. 2021 ; Yurtsever et al. 2020 , including model-free and model-based approaches for handling variety of driving tasks. Furthermore, multimodal RL frameworks have been explored to incorporate rich sensor data, improving decision-making under uncertainty. While these methods have significantly advanced the field, challenges such as sparse reward signals, generalization across diverse environments, and the sim-to-real transfer gap remain open problems that demand further investigation. This paper provides a comprehensive review of RL research within the CARLA simulator, analyzing the evolution of RL techniques and their applications in autonomous driving. The key contributions of this review are as follows: • Survey of RL Formulations – We explore various RL approaches employed in CARLA, including model-free methods such as the Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG); model-based techniques; hierarchical RL; and hybrid strategies that integrate multiple learning paradigms. • Comparison of State and Action Spaces – We examine how different sensor modalities (camera, LiDAR, multi-modal fusion) and control abstractions (low-level throttle/steering vs. high-level discrete maneuvers) impact policy learning and performance. • Reward Engineering and Terminal condition in RL for Autonomous Driving – We highlight the critical role of reward function design, analyzing how different formulations balance safety, comfort, and efficiency in various driving tasks. • Evaluation metrics and Scenarios for RL Policies – We review the evaluation methodologies used in CARLA, including key performance metrics such as collision rates, lane deviation, speed regulation, and success rates in navigation tasks along with the different scenarios used for training and testing. • Challenges and Future Directions – We discuss persistent challenges in RL-driven autonomous driving often mentioned in the literature, including generalization to unseen environments, robustness in high-density traffic scenarios, and interpretability of learned strategies. Additionally, we outline promising avenues for future research to bridge these gaps. By synthesizing recent advancements and identifying key challenges, this review aims to serve as a valuable resource for researchers and practitioners in the field. As simulation platforms like CARLA continue to evolve, they will play an increasingly critical role in the pursuit of safe and reliable autonomous driving systems. Through this work, we hope to provide clarity on the state of RL in CARLA, fostering further innovation and accelerating progress toward real-world deployment. This review focuses on single-agent RL approaches in the CARLA simulator. While Multi-Agent RL (MARL) is a relevant and growing area of research—particularly for urban traffic scenarios and coordination among multiple vehicles—it is beyond the scope of this review. we consider around 100 papers from IEEE and ACM. 2 CARLA simulator CARLA is an open-source simulator explicitly designed to support research in autonomous driving and ADAS. Built on the Unreal Engine 4 1 1 1 https://www.unrealengine.com/ , CARLA provides a photorealistic 3D environment with configurable urban and highway scenes that include diverse road layouts, dynamic weather, and day–night cycles. Crucially, it exposes a flexible Python/C++ API for scene manipulation, traffic generation, and sensor configuration, enabling researchers to script complex scenarios and integrate with ROS Quigley et al. 2009 , Autoware