Building reliable sim driving agents by scaling self-play
Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, Eugene Vinitsky · 2025
Read Building reliable sim driving agents by scaling self-play by Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, Eugene Vinitsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14706.txt.
Introduction
Daphne Cornelisse and four co-authors trained dependable simulated drivers by massive self-play in a 2025 paper. Simulation drivers test self-driving systems, but only if they behave as intended: stray crashes pollute the experiments. Their agents learned from scratch, on one graphics card, across thousands of traffic scenarios drawn from a real-world motion dataset, under human-like limits on perception and control.
Within a day the agents solved nearly the whole training set. On ten thousand unseen scenes they reached their goals 99.8 percent of the time, with collisions and off-road slips together under 0.8 percent. They partly handled scenes beyond the training distribution, and minutes of fine-tuning restored near-perfect play there. The trained agents and simulator are open-sourced.
Catalog record
| CREATOR | Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, Eugene Vinitsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building reliable sim driving agents by scaling self-play Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, Eugene Vinitsky Abstract Simulation agents are essential for designing and testing systems that interact with humans, such as autonomous vehicles (AVs). These agents serve various purposes, from benchmarking AV performance to stress-testing system limits, but all applications share one key requirement: reliability. To enable sound experimentation, a simulation agent must behave as intended. It should minimize actions that may lead to undesired outcomes, such as collisions, which can distort the signal-to-noise ratio in analyses. As a foundation for reliable sim agents, we propose scaling self-play to thousands of scenarios on the Waymo Open Motion Dataset under semi-realistic limits on human perception and control. Training from scratch on a single GPU, our agents solve almost the full training set within a day. They generalize to unseen test scenes, achieving a 99.8% goal completion rate with less than 0.8% combined collision and off-road incidents across 10,000 held-out scenarios. Beyond in-distribution generalization, our agents show partial robustness to out-of-distribution scenes and can be fine-tuned in minutes to reach near-perfect performance in such cases. We open-source the pre-trained agents and integrate them with a batched multi-agent simulator. Demonstrations of agent behaviors can be viewed at https://sites.google.com/view/reliable-sim-agents , and we open-source our agents at https://github.com/Emerge-Lab/gpudrive . 1 Introduction Simulation agents are a core part of safely developing and testing systems that interact with humans, such as autonomous vehicles (AVs). In the context of self-driving, these agents, also referred to as road user behavior models, serve two primary purposes: establishing benchmarks for AV behavior ( Engström et al., 2024 ) , and representing other road users in simulators to enable statistical safety testing in both nominal and rare, long-tail scenarios ( Corso et al., 2021 ; Montali et al., 2024 ) . While each use case brings particular requirements, reliability is an important one that they share. A reliable simulation agent consistently behaves as intended by the designer, minimizing unintended actions. For instance, agents designed to stress-test AVs should reliably initiate realistic near-collision events, generating safety-critical scenarios to provide meaningful information about the system’s behavior in edge cases. Conversely, nominal agents must focus on replicating typical road behavior to simplify experiments that vary other environmental factors, such as weather. In either case, unreliable sim agents introduce noise into the evaluation process by producing trajectories that crash too infrequently in the stress-test case and too frequently in the nominal case. How can we build sim agents that are close enough 1 1 1 Here, close enough is emphasized because what constitutes an acceptable model of human behavior depends highly on the use case. to reality while maximizing designer specifications, i.e., reliability? One approach relies on generative models, which have shown remarkable progress in producing diverse, human-like behaviors through imitation learning from demonstrations ( Xu et al., 2023 ; Philion et al., 2024 ; Huang et al., 2024 ) . However, it is uncertain whether they meet the reliability standards of a fully automated AV development pipeline. This is highlighted by the best-performing models in the Waymo Open Sim Agent Challenge ( Montali et al., 2024 , WOSAC) , a well-known benchmark for realistic nominal road user behavior. While state-of-the-art models in the 2024 challenge closely replicate logged human trajectories and achieve high scores on various distributional metrics, they still fall short in critical areas. Ground-truth human trajectories in the dataset rarely or never involve collisions or off-road movements; the best aggregate collision and off-road rates among the top WOSAC 2024 are approximately 4% and 1-2%, respectively ( Zhang et al., 2024b ; Huang et al., 2024 ; Zhou et al., 2024 , CATK, VBD, BehaviorGPT) . This is well below the capabilities of human drivers. Figure 1 : Overview of approach. Left : We define several criteria to guide the learning of simulation agents through rewards. The reward function is a weighted combination of these criteria: r(o^{i}_{t})=\sum_{i}c_{i}\cdot\mathbb{I}[\text{criteria}_{i}] . Here, we focus on achieving goal-directed nominal sim agent behavior—ensuring agents stay on the road and avoid collisions while navigating to a target position. Right : Over 24 hours on a single GPU, we iterate through 10,000 scenarios (green curve) from the Waymo Open Motion Dataset in GPUDrive ( Kazemkhani et al., 2024 ) , reaching high performance (blue curve, reliability) on the defined criteria after 2 billion agent steps by self-play PPO. The example scenarios illustrate agent behavior at different stages of training. Initially, agents display random behavior and frequently collide with each other and the road edges (marked in orange and red), but their behavior becomes streamlined over many iterations. Importantly, these relatively high failure rates may limit the scalability of AV evaluation and development, especially as generative models are increasingly used to create rare safety-critical scenarios that are underrepresented in real-world data ( Mahjourian et al., 2024 ) . When trajectories deviate unpredictably, researchers or engineers must find out: Is the observed outcome a signal or an artifact of simulator noise? For example, if 1 in 10 scenarios reflects unintended behavior, distinguishing meaningful failures from artifacts becomes a time-consuming task. As such, making sim agents more reliable seems to be a key pillar to further scale AV evaluation and development. The question becomes: How can we close this reliability gap for state-of-the-art sim agents? Assuming we can precisely define what the agent should adhere to (e.g., stay on the road), there is reason to believe that self-play reinforcement learning (RL) is a piece of the puzzle. Evidence from a broad body of recent literature on games shows that self-play RL, combined with well-defined criteria (e.g., maximize score X) can produce agents capable of perfect, superhuman gameplay in the large compute and data regime ( Silver et al., 2018 ; Bakhtin et al., 2023 ; Berner et al., 2019 ) . We study whether self-play at scale improves the reliability of sim agents. Specifically, we ask: 1. How does the reliability (as measured by performance on the test set of metric X) of sim agents through self-play scale as a function of the data available? 2. How well do these agents generalize to unseen scenarios and out-of-distribution events? To explore these questions, we train agents via self-play using a semi-realistic human perception framework in a data-driven simulator ( Kazemkhani et al., 2024 ) . We evaluate performance across thousands of scenarios from the Waymo Open Motion Dataset ( Ettinger et al., 2021 ) . We find that self-play PPO scales effectively with on-policy data and compute. After sufficient training, models generalize well to 10,000 unseen test traffic scenarios, effectively closing the train-test gap. At scale, self-play PPO sim agents consistently achieve the specified criteria (Section 2.2 ): staying on the road, avoiding collisions, and reaching a target position. This establishes a framework where agents can be tuned to achieve specific collision rates, enabling both nominal and safety-critical traffic simulation. By improving the reliability standards of sim agents, our approach supports the continued scaling and automation of AV development and evaluation pipelines. Finally, we take a step towards fine-tuning these agents for behaviors underrepresented in the dataset, a useful capability for safety-critical applications. To facilitate further research, we open-source the pre-trained agents