Heterogeneous Self-Play for Realistic Highway Traffic Simulation

Jinkai Qiu, Alessandro Saviolo, Chaojie Wang, Mingke Wang, Xiaoyu Huang · 2026

Read Heterogeneous Self-Play for Realistic Highway Traffic Simulation by Jinkai Qiu, Alessandro Saviolo, Chaojie Wang, Mingke Wang, Xiaoyu Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16406.txt.

Introduction

Qiu and colleagues present PHASE, a self-play system for generating believable highway traffic to test self-driving cars safely. Each simulated driver gets its own vehicle profile and context, covering cars and articulated trucks within one policy, while training stabilizes play through early stopping of hopeless states and fault-aware collision credit. Trained only on synthetic data, the system transferred without retraining to 512 unseen real high-interaction scenarios, succeeding in about 96 percent of them. Its trajectories also looked more realistic than a standard traffic model.

Catalog record

CREATORJinkai Qiu, Alessandro Saviolo, Chaojie Wang, Mingke Wang, Xiaoyu Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37613
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16406

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heterogeneous Self-Play for Realistic Highway Traffic Simulation Jinkai Qiu, Alessandro Saviolo, Chaojie Wang, Mingke Wang, Xiaoyu Huang Abstract Realistic highway simulation is critical for scalable safety evaluation of autonomous vehicles, particularly for interactions that are too rare to study from logged data alone. Yet highway traffic generation remains challenging because it requires broad coverage across speeds and maneuvers, controllable generation of rare safety-critical scenarios, and behavioral credibility in multi-agent interactions. We present PHASE , P olicy for H eterogeneous A gent S elf-play on E xpressway, a context-aware self-play framework that addresses these three requirements through explicit per-agent conditioning for controllability, synthetic scenario generation for broad highway coverage, and closed-loop multi-agent training for realistic interaction dynamics. PHASE further supports different vehicle profiles, for example, passenger cars and articulated trailer trucks, within a single policy via vehicle-aware dynamics and context-conditioned actions, and stabilizes self-play with early termination of unrecoverable states, at-fault collision attribution, highway-aware reward shaping, coupled curricula, and robust policy optimization. Despite being trained only on synthetic data, PHASE transfers zero-shot to 512 unseen high-interaction real scenarios in exiD, achieving a 96.3% success rate and reducing ADE/FDE from 6.57/12.07 m to 2.44/5.25 m relative to a prior self-play baseline. In a learned trajectory embedding space, it also improves behavioral realism over IDM, reducing Fréchet trajectory distance by 13.1% and energy distance by 20.2%. These results show that synthetic self-play can provide a scalable route to controllable and realistic highway scenario generation without direct imitation of expert logs. 1 Introduction Highway driving is one of the most safety-critical settings for autonomous vehicles: interactions unfold at high speed, small prediction errors can escalate quickly, and rare events such as aggressive cut-ins or dense merging are difficult to capture in sufficient quantity from real-world logs alone. This makes simulation a core tool for development and validation. A highway simulator, however, must do more than replay recorded traffic. It must expose safety-critical interactions at scale, remain stable in closed-loop rollouts, and support stress testing across diverse traffic regimes. Existing approaches only partially satisfy these requirements. Log replay and supervised trajectory prediction preserve realistic behavior, but are limited by the behaviors present in the data and can drift under closed-loop execution 7 ; 2 . Rule-based traffic models offer controllability, but often rely on simplified assumptions that fail to capture the richness of real multi-agent interactions. Recent self-play reinforcement learning (RL) methods offer a promising alternative by learning interactive behavior directly from closed-loop experience, and have shown strong results in urban driving 3 ; 4 , including on benchmarks derived from the Waymo Open Motion Dataset (WOMD) 5 . Yet these methods have been developed primarily for urban regimes and do not directly address the demands of highway traffic. Highway simulation poses a distinct challenge. Compared with urban driving, highway traffic spans a broader speed range, exhibits greater variation in density and interaction style, and includes stronger physical heterogeneity across vehicle classes. A practical highway simulator must therefore satisfy three requirements simultaneously: broad coverage across speeds and maneuvers, controllable generation of rare safety-critical scenarios, and realistic multi-agent behavior in closed-loop interaction. Figure 1 : Effect of context conditioning on closed-loop highway behavior. The three panels in each row share the same initial scenario; only the context-conditioning variables \mathcal{C} (Section 3.1 ) are changed across columns. Despite identical initialization, the conditioned policy generates distinct closed-loop trajectory patterns within each triplet. Joint conditioning on \mathcal{C} and each agent’s Cartesian goal g_{i} enables explicit control over scenario outcomes, as illustrated by vehicle 0 (orange), which alternates between overtaking and being overtaken. We address these challenges with PHASE, a self-play framework for controllable heterogeneous highway traffic generation. We formulate the problem as a conditioned goal-reaching Partially Observable Stochastic Game (POSG), in which each agent is assigned a Cartesian goal together with context variables such as target speed, longitudinal action range, and vehicle type. This conditioning provides explicit behavioral control, while a single policy handles both passenger cars and articulated tractor-trailers. As illustrated in Figure 1 , varying the context variables while holding the base scene fixed produces distinct interaction patterns and allows users to steer scenario outcomes directly. To achieve broad coverage without relying on expert trajectory imitation, we train entirely on synthetic scenarios generated by an offline–online pipeline: lane-graph search constructs diverse start–goal pools offline, and online sampling randomizes agent count, lane-change composition, kinematics, and geometry. To preserve realistic behavior under self-play, we further combine simulation mechanisms, highway-aware reward design, coupled curricula, and robust policy optimization. We evaluate PHASE in two complementary settings. On exiD 13 , PHASE transfers zero-shot to unseen real highway scenarios and substantially outperforms a prior self-play baseline in both success rate and displacement error. In a learned trajectory embedding space built from proprietary real highway logs, it also produces trajectory distributions that align more closely with real traffic than IDM 21 . Our main contributions are as follows: • A conditioned formulation for controllable heterogeneous highway simulation. We formulate highway traffic generation as a conditioned goal-reaching POSG in which a single policy controls both passenger cars and articulated tractor-trailers across a 0–40 m/s operating range through joint goal and context conditioning. • A scalable synthetic scenario generation pipeline for broad highway coverage. We introduce an offline–online generator that combines lane-graph endpoint search with controllable world sampling to produce diverse, map-consistent highway scenes without direct imitation from expert trajectory logs. • A stable self-play training recipe for highway traffic. We combine simulation mechanisms, highway-aware reward shaping, coupled curricula, and robust optimization choices to make learning-from-scratch self-play practical in heterogeneous highway regimes. • Strong real-world transfer and improved behavioral realism. PHASE generalizes zero-shot to exiD and produces trajectory distributions that more closely match real highway driving than prior self-play and classical controller baselines. 2 Related Work Self-play RL for driving and autonomy. Self-play RL has been effective in complex multi-agent settings, from board games 20 ; 19 to large-scale strategy environments 16 . Recently, this paradigm has been adapted to autonomous driving, where interaction is inherently multi-agent and closed-loop. Methods such as 3 ; 4 show that large-scale self-play can learn robust driving policies in urban environments. However, existing driving applications have focused primarily on lower-speed urban regimes, where the dominant challenges are negotiation and topological complexity at intersections. In contrast, we study high-speed highway traffic, where smaller control errors can have larger consequences and where stable self-play must handle broader speed ranges and stronger vehicle heterogeneity. Large-scale driving simulators. GPU-accelerated simulators have made large-scale training for interactive driving practical. Platforms such as Waymax 6 , Nocturne 23 , and GPUDrive 11 support parallel simulation and enable millions of agent steps per second. These systems provide the computational substrate for modern self-play training, but they typically assume access to logged trajectory data or homogeneous vehicle dynamics. Our work builds on GPUDrive and extends this setting in two directions: synthetic scenario generation without expert trajectory imitation, and heterogeneous dynamics that support both passenger cars and articulated trucks within the same training framework. Imitation learning for behavior and interaction modeling. Imitation-learning approaches such as Versatile Behavior Diffusion 9 and BehaviorGPT 24 have set strong benchmarks for realistic traffic generation by modeling the multimodality of road-user trajectories. Their

Read the complete text