Learning Rollout from Sampling: An R1-Style Tokenized Traffic Simulation Model

Ziyan Wang, Peng Chen, Ding Li, Chiwei Li, Qichao Zhang, Zhongpu Xia, Guizhen Yu · 2026

Read Learning Rollout from Sampling: An R1-Style Tokenized Traffic Simulation Model by Ziyan Wang, Peng Chen, Ding Li, Chiwei Li, Qichao Zhang, Zhongpu Xia, Guizhen Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24989.txt.

Introduction

Wang and co-authors present R1Sim, a traffic simulator that learns from human driving data. It writes road scenes as tokens, much like a language model writes words. Older tools train by copying the data, which limits bold exploration of rare moves. R1Sim samples moves with high uncertainty but high promise, guided by token entropy. It then tunes behavior with group-based rewards that favor safety. On the Waymo Sim Agent benchmark it matches the best prior systems while giving varied, realistic scenes.

Catalog record

CREATORZiyan Wang, Peng Chen, Ding Li, Chiwei Li, Qichao Zhang, Zhongpu Xia, Guizhen Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35130
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Rollout from Sampling: An R1-Style Tokenized Traffic Simulation Model Ziyan Wang, Peng Chen, Ding Li, Chiwei Li, Qichao Zhang, Zhongpu Xia, Guizhen Yu Abstract Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration–exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods. I Introduction Smart traffic simulation is essential for autonomous driving validation [ 27 , 7 , 16 ] and the continuous improvement of autonomous driving policies within a safe, scalable environment. Drawing inspiration from the success of large language models (LLMs) in natural language processing [ 1 ] , the smart traffic simulation paradigm can be formulated as a multi-agent policy learning task with autoregressive modeling, with agent trajectories tokenized into discrete motion representations and trained through imitation learning-based next token prediction (NTP) for motion generation. State-of-the-art (SOTA) frameworks like SMART [ 26 ] and CATK [ 29 ] have demonstrated exceptional scalability in real-time multi-agent motion generation, establishing pretrained foundation models for complex traffic simulation scenarios. Despite promising motion generation stability, existing tokenized motion models still fail to handle these two key challenges: 1) how to enable adaptive exploration to sample multiple plausible motion scenarios. SOTA approaches [ 26 , 29 ] implement the Top-K sampling strategy [ 6 ] to select a fixed number of motion tokens for simulation rollout, as shown in Fig. 1 (a). The rigid strategy over-prioritizes high-probability motion tokens from the vocabulary while neglecting potentially valuable “hidden gem” behaviors in the token vocabulary, particularly detrimental in interactive scenarios, where diverse motion outcomes are essential. Once the exploration space is established, 2) how to enable effective exploitation to optimize realism and safety of multi-agent motion behaviors. Existing optimization methods [ 14 , 29 ] such as supervised fine-tuning (SFT), often employ winner-takes-all approaches that force generated states to match expert demonstrations. However, as shown in Fig. 1 (b), over-reliance on potentially suboptimal ground truth may perpetuate unsafe behaviors. Thus, it is crucial to balance well between exploration and exploitation to discover “hidden gem” behaviors that best align with human-preferred motion tokens. Fig. 1 : The motivation of R1Sim. (a) Exploration: Compared with Top-K sampling [ 26 ] , our entropy-guided adaptive sampling explores more high-entropy motion tokens. (b) Exploitation: Compared with SFT [ 29 ] , our refined GRPO estimates group-wise advantages and selects the optimal scenario. To address these challenges, we introduce R1Sim, a novel framework that pioneers the integration of motion token entropy dynamics into reinforcement learning for autonomous traffic simulation. We observe that entropy variations in tokenized motion simulation models effectively capture distributional modeling uncertainty [ 2 , 8 , 17 ] : low entropy typically corresponds to predictable, inertial maneuvers, while high entropy signals complex, multi-modal driving intentions. Motivated by this, R1Sim establishes a balanced exploration–exploitation mechanism tailored for traffic generation. For exploration, we propose an entropy-guided adaptive sampling strategy. Unlike conventional fixed Top-K sampling, our proposed sampling mechanism dynamically relaxes constraints under high uncertainty to uncover physically plausible yet low-probability “hidden gem” behaviors that are often missed by standard greedy decoding. For exploitation, to strictly align these diverse proposals with human safety and realism preferences, we introduce a refined Group Relative Policy Optimization (GRPO) [ 9 , 23 ] . By contrasting token-level advantages within rollout groups and incorporating a fine-grained, safety-aware reward function, our method systematically reinforces optimal decision-making. This approach overcomes the limitations of both traditional RL value estimation and SFT, enabling progressive discrimination between optimal and suboptimal motion patterns as illustrated in Fig. 1 . Our contributions are listed as follows: • We propose a human-preferred motion simulation framework, R1Sim, by balancing exploration-exploitation within a next token prediction pretraining paradigm. • We introduce an entropy-guided adaptive sampling strategy to enhance exploration by dynamically selecting high-uncertainty yet potentially optimal motion tokens. • We develop a GRPO-refined method with safety-aware rewards to effectively exploit high-quality behaviors. II Related Work II-A Tokenized Traffic Simulation Autoregressive modeling has demonstrated strong expressive power and flexibility in modeling complex sequential data. Recent advances in this paradigm have substantially improved the realism of traffic simulation [ 26 , 20 , 22 ] . These methods discretize complex traffic behaviors into sequential motion tokens to enable long-horizon motion generation [ 32 ] . Despite demonstrating promising simulation results, these models suffer from covariate shift due to distributional discrepancies between training and inference states step by step [ 3 ] . To mitigate this limitation, recent works have investigated supervised fine-tuning. For instance, CATK [ 29 ] establishes a closed-loop fine-tuning paradigm, using SMART as its pretrained foundation to generate high-fidelity tokenized rollouts that closely match ground truth trajectories. From another aspect, UniMM [ 14 ] augments closed-loop samples from the policy itself to mitigate covariate shift. However, these methods face two inherent limitations in NTP paradigms: 1) Rigid fixed sampling strategies create a scalability bottleneck for diverse motion generation. 2) Reliance on ground truth constrains exploration in sub-optimal regions, hindering the discovery of tokens that better align with the underlying driving logic. II-B Reinforcement Learning Fine-tuning Great breakthroughs of LLMs in aligning with human preference via reinforcement learning fine-tuning (RLFT) have inspired a parallel evolution in autonomous driving. Initial works such as BC-SAC [ 15 ] utilize simple rewards to enhance imitation learning, while subsequent research [ 11 , 13 ] adopts Reinforcement Learning from Human Feedback (RLHF) to unlock diverse driving behaviors. Notable examples include TrajHF [ 13 ] , which refines multi-modal motion generation results by incorporating multi-conditional denoiser, and Carplanner [ 28 ] , which optimizes planning via Proximal Policy Optimization (PPO) [ 21 ] . Nevertheless, the computational overhead of PPO’s value function estimation poses challenges for real-time control. To address this, DeepSeek-R1 [ 9 ] introduces Group Relative Policy Optimization (GRPO), which optimizes policies through group-wise outcomes without requiring a critic network. This paradigm has validated its efficacy in handling complex generative tasks, ranging from task decomposition [ 19 ] , language reasoning [ 4 ] and text-to-image generation. Although originally developed for large language models, GRPO is well suited for autonomous driving, as both domains adopt an autoregressive formulation and benefit from group-wise comparison over multiple sampled candidates. Recent models such as

Read the complete text