RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning

Ehsan Ahmadi, Hunter Schofield, Behzad Khamidehi, Fazel Arasteh, Jinjun Shan, Lili Mou, Dongfeng Bai, Kasra Rezaee · 2026

Read RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning by Ehsan Ahmadi, Hunter Schofield, Behzad Khamidehi, Fazel Arasteh, Jinjun Shan, Lili Mou, Dongfeng Bai, Kasra Rezaee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19033.txt.

Introduction

Ehsan Ahmadi and seven colleagues build RLFTSim for multi agent traffic play. Plain open loop train misses live give and take in hard drive scenes. RLFTSim lines sim runs up with real runs and brews goal led control into scenes. Built on a prior sim model, its prize weighs truth and control. It tops Waymo Open Motion realism, needs far fewer tries than hunt tunes, and shows goal led control.

Catalog record

CREATOREhsan Ahmadi, Hunter Schofield, Behzad Khamidehi, Fazel Arasteh, Jinjun Shan, Lili Mou, Dongfeng Bai, Kasra Rezaee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77968
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning Ehsan Ahmadi, Hunter Schofield, Behzad Khamidehi, Fazel Arasteh, Jinjun Shan, Lili Mou, Dongfeng Bai, Kasra Rezaee Abstract Supervised open-loop training has been widely adopted for training traffic simulation models; however, it fails to capture the inherently dynamic, multi-agent interactions common in complex driving scenarios. We introduce RLFTSim, a reinforcement-learning-based fine-tuning framework that enhances scenario realism by aligning simulator rollouts with real-world data distributions and provides a method for distilling goal-conditioned controllability in scenario generation. We instantiate RLFTSim on top of a pre-trained simulation model, design a reward that balances fidelity and controllability, and perform comprehensive experiments on the Waymo Open Motion Dataset. Our results show improvements in realism, achieving state-of-the-art performance. Compared with other heuristic search-based fine-tuning methods, RLFTSim requires significantly fewer samples due to a proposed low-variance and dense reward signal, and it directly addresses the realism alignment issue by design. We also demonstrate the effectiveness of our approach for distilling traffic simulation controllability through goal conditioning. The project page is available at https://ehsan-ami.github.io/rlftsim . 1 Introduction Simulation plays a major role in autonomous driving by providing a controllable virtual environment for the test and development of autonomous vehicles (AVs). This is particularly significant for rare accident scenarios that an AV should be capable of handling. Based on a statistical estimate in 15 , verifying the safety aptitude of an AV requires it to be capable of driving 2 million kilometers without fatal accidents to claim that it is as safe as humans. Considering the costs of such verification, the use of simulation is justified to make the verification and development of AVs feasible. In this work, we focus on the problem of microscopic multi-agent traffic simulation, as we elaborate in § 3 . This problem has previously been addressed with rule-based simulators, which can provide various feedback signals from kinematics and physics simulations 10 ; 2 . These simulators control the agents either via replaying logged trajectories or using simple rule-based models, such as the constant-speed actor model or the Intelligent Driver Model 30 . This, arguably, leads to a significant sim-to-real gap. Even for the case of ground-truth data log replay, since the agents’ behavior is not reactive, the simulation becomes unrealistic when it faces closed-loop deployment. Figure 1 : Post-training with RLFTSim. For a seed scenario from the dataset, multiple rollouts are generated. The main reward function is defined based on time-independent distribution matching of the simulated scenarios and the expert demonstration in several feature spaces. This closed-loop on-policy optimization enhances realism beyond what open-loop imitation learning achieves alone. Recently, learning-based simulation models have been introduced to bridge the realism gap and generate a reactive multi-agent traffic simulation rollout, which is an episode of the simulation for all of the agents in a traffic scenario 34 ; 44 . However, previous models are mostly trained in an open-loop setting with imitation learning objectives 34 ; 44 . We argue that to make simulation models capable of overcoming distribution shifts caused by error accumulation in closed-loop simulation and avoiding the causal confusion problem 8 ; 1 , the training (or post-training) of a simulation model needs to be done in a closed-loop environment. An effective simulation model needs to be capable of generating realistic data that accurately represents daily driving scenarios to make its adoption worthwhile. While still imitating human agents, it needs to be aligned with traffic rules and the physical constraints of the environment. In the past, a combination of open-loop supervision via imitation learning and closed-loop training via reinforcement learning has been used for ego-centric motion planning 19 . However, effective scaling of this approach for multi-agent simulation models remains an open area of research. In this work, inspired by the successful application of reinforcement learning with verifiable rewards for new skill learning in foundation models 25 , we present a reinforcement learning-based fine-tuning approach for traffic simulation (RLFTSim) to address the physics and traffic-rules alignment problem for a pre-trained model with an open-loop imitation learning objective. We deal with the alignment problem with a closed-loop post-training approach (Fig. 1 ). Specifically, we focus on the Waymo Open Simulation Challenge (WOSAC) and use its definition of realism meta-metric (RMM) as the optimization objective function for RLFT. However, RMM in its original form is a sparse population-based signal calculated per group of 32 rollouts, making it sample-inefficient for direct use as a reward in RL. To overcome this issue, we present realism Meta-metric Leave-One-Out (MLOO) as a low-variance and dense adaptation of the RMM for RL fine-tuning. To the best of our knowledge, this is the first successful application of the RMM for RL fine-tuning in traffic simulation. We present theoretical variance scaling analysis that justify our proposed MLOO as a reward signal. Moreover, we discuss the issue of controllability in traffic simulation as a part of the model alignment problem, and with the aid of hindsight experience replay (HER) 4 , goal-conditioning, and RLFT, we present an approach to distill controllability. Our contributions are: (i) We introduce realism meta-metric as the reward signal for realism alignment with closed-loop RL-based fine-tuning in multi-agent traffic simulation (§ 4 ). (ii) We provide a theoretical variance analysis for the current WOSAC’s realism meta-metric, and we propose MLOO to have a low-variance and dense reward signal (§ 4.1 ). (iii) We introduce the application of hindsight experience replay with RLFT for the behavior controllability distillation with goal-conditioning in the simulation process (§ 4.2 ). (iv) We conduct comprehensive experiments showcasing the enhanced realism with RLFT to achieve the state-of-the-art performance, and the distilled controllability skills (§ 5 ). 2 Related Works Learning-based Traffic Simulation Approaches. Learning-based models have emerged as a cornerstone for realistic traffic simulation, moving beyond heuristic rule-based simulators that struggled with complex maneuvers and interactions 30 . Early works like 26 modeled multi-agent driving behavior with implicit latent variables and differentiable closed-loop training, significantly improving realism over rule-based baselines. Imitation learning has been widely adopted for traffic simulation 11 ; 42 ; 17 ; 13 . The Waymo Open Motion Dataset (WOMD) 9 , nuPlan 5 , and similar datasets of millions of real trajectories have enabled training transformer-based models that treat multi-agent traffic simulation as a next-token prediction problem 34 ; 12 ; 22 ; 39 ; 44 . SMART predicts the next motion token for each agent conditioned on vectorized state representations and map context 34 . By training on over a billion motion tokens from multiple datasets, SMART achieved state-of-the-art realism in the WOMD Sim Agents Challenge. In this work, we utilize the superior power of SMART in traffic simulation by choosing it as a reference model for alignment fine-tuning. Model Alignment and Fine-Tuning. A key challenge in learned simulators is model alignment : ensuring the distribution of simulated behaviors in closed-loop matches real-world driving data. Pure behavior cloning suffers from the accumulation of errors during closed-loop deployment 24 , where small mistakes compound and drive agents to unseen or unrealistic states. In 36 , Closest Among Top-K (CAT-K) sampling is used to perturb the expert policy. By generating multiple plausible next actions and selecting the one closest to the ground-truth

Read the complete text