InfGen: Scenario Generation as Next Token Group Prediction
Zhenghao Peng, Yuxin Liu, Bolei Zhou · 2025
Read InfGen: Scenario Generation as Next Token Group Prediction by Zhenghao Peng, Yuxin Liu, Bolei Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23316.txt.
Introduction
Zhenghao Peng and co-authors present SceneStreamer, a system that invents traffic scenes on the fly in this 2025 paper. Self-driving cars need long, varied practice worlds, but most simulators replay fixed logs or start from frozen setups. SceneStreamer instead writes the whole scene as a stream of tokens, covering lights, cars, and motion.
A transformer model then predicts the stream step by step, like a language model writing a sentence. Cars and other agents can enter or leave at any time, so scenes can run without a set end. Driving policies trained inside these worlds proved tougher and adapted better to new tests. The authors offer the system as a practice ground for driving AI.
Catalog record
| CREATOR | Zhenghao Peng, Yuxin Liu, Bolei Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-23316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InfGen: Scenario Generation as Next Token Group Prediction Zhenghao Peng, Yuxin Liu, Bolei Zhou Abstract Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. We propose InfGen, a scenario generation framework that outputs agent states and trajectories in an autoregressive manner. InfGen represents the entire scene as a sequence of tokens—including traffic light signals, agent states, and motion vectors—and uses a transformer model to simulate traffic over time. This design enables InfGen to continuously insert new agents into traffic, supporting infinite scene generation. Experiments demonstrate that InfGen produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in InfGen-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at https://metadriverse.github.io/infgen/ . 1 Introduction Simulating realistic and diverse traffic scenarios is vital for the development and evaluation of autonomous driving systems. Simulation enables safe, cost-effective, and repeatable testing of driving policies without relying on real-world deployment. However, most existing frameworks use static traffic generation methods, such as replaying logged trajectories from real-world datasets li2023scenarionet ; Dosovitskiy17 ; gulino2023waymax . Although faithful to real driving behaviors, they lack interactivity as background agents do not respond to the ego vehicle’s actions, limiting their utility for closed-loop evaluation. Recently, data-driven generative models have emerged to learn to synthesize traffic scenarios from real data, offering a path toward richer and more simulations zhang2025drivegen ; rowe2025scenario . Learning-based traffic simulation is commonly framed as a motion prediction problem: given a history of agent states in a scene, including map, signals, and initial state of agents, a policy generates the future trajectories of all agents. However, most such models are trained as an one-shot prediction model ngiamscene ; shi2022motion ; pronovost2023scenario and do not explicitly model interactions between agents during the prediction horizon, which leads to covariate shift when they are unrolled in the simulation. Small prediction errors can compound, causing the simulator to visit out-of-distribution states and produce unrealistic outcomes. Recently, the autoregressive models have been proposed to better fit into the driving behavior modeling, especially in the context of closed-loop simulation ( suo2021trafficsim ; zhang2023trafficbots ; seff2023motionlm ; kamenev2022predictionnet ) . However, these models still rely on the provided initial states of traffic agents and miss the diversity that emerges from the initial layout of traffic participants. Some other works propose generating the initial conditions and then conducting motion prediction based on these conditions feng2023trafficgen ; bergamini2021simnet ; tan2021scenegen . This separation can be inefficient and inflexible, as it prevents the model from sharing context between the initial and motion phases. It also means the number of agents is fixed at initialization, disallowing new traffic participants to enter the scene over time—but in reality, new traffic participants enter the scene while old ones leave. These limitations hinder the simulator’s ability to model open-world, long-horizon interactions where new agents may appear and leave (e.g., vehicles turning into/from the road from/into a side street) and influence the ego vehicle’s behavior. Figure 1 : InfGen enables unified scenario generation via autoregressive token prediction. We represent a dynamic driving scene using a structured sequence of discrete tokens grouped into traffic light, agent state, and agent motion tokens. InfGen generates these tokens step-by-step on top of static map tokens, allowing flexible and fine-grained simulation. Our unified model supports diverse downstream applications: motion prediction, full-scenario generation from scratch, scenario densification by injecting new agents, and closed-loop simulation for training self-driving planners. To address these gaps, we propose InfGen, a generative traffic simulation framework that models the entire scenario, including agent states and motions, as a single sequence of tokens. InfGen uses a unified autoregressive model to generate both the agent state and agent motion at every step. We tokenize different categories of agents, such as vehicle, pedestrian and cyclist, the same way, with different category embeddings added to their tokens. Notably, InfGen is flexible to adapt to different tasks, including motion prediction, state initialization, scenario generation, and scene editing like adding new agents and desification, by choosing different tokens to be teacher-forced while others to be sampled. We implement a carefully designed agent state tokenization pipeline so that the model can effectively handle heterogeneous agent types and map context when adding new agents. Finally, we demonstrate that using InfGen to generate training scenarios leads to significant improvements in downstream planner performance. Reinforcement Learning-based planners trained on InfGen-generated scenarios exhibit greater robustness and better generalization to novel environments. We summarize our contributions as below: 1) Unified State & Trajectory Tokenization: InfGen is the first framework to employ a single autoregressive model that produces both agents’ initial states and their motion trajectories as part of one continuous token sequence over long horizons. This unified approach ensures consistent conditioning between where an agent starts and how it moves, addressing the inflexibility of prior two-stage models. 2) Agent State Autoregressive Generation: We design a novel generation scheme for agent states by autoregressively rolling out the agent state tokens and generating the map-based relative states of agents, i.e., the agent’s type, its map location, and its detailed kinematic state. This allows the model to accurately place agents on specific map segments (e.g., lanes) and generate realistic state details (position, heading, velocity, etc.) in a compact, learnable representation. 3) Versatile Capabilities : By dynamically teacher-forcing different token groups, InfGen is versatile and applicable to various tasks, including motion prediction, traffic simulation, scenario generation, and scene editing. We demonstrate that training autonomous driving planners in InfGen-generated scenarios yields more robust and generalizable policies, indicating that InfGen can serve as a scenario generator and an effective data augmentation tool for closed-loop simulation. 2 Method Scenario Generation. A driving scenario comprises (1) static map context \mathcal{M} (vectorized lane segments, crosswalks, etc.) and (2) dynamic entities including traffic - lights \{\mathbf{l}^{(k)}_{t}\}_{k=1}^{N_{\mathrm{TL}}} and traffic agents \{\mathbf{a}^{(i)}_{t}\}_{i=1}^{N_{t}} that evolve with time t . Here N_{\mathrm{TL}} denotes the number of traffic lights and N_{\mathrm{t}} denotes the number of agents at step t . Each traffic - light state \mathbf{l}^{(k)}_{t}=(x,y,s) contains 2 - D position and discrete signal s\in\{\mathrm{green},\mathrm{yellow},\mathrm{red},\mathrm{unknown}\} while each agent state \mathbf{a}^{(i)}_{t}=(x,y,v_{x},v_{y},\psi,c,l,w,h) encodes pose, velocity, heading, category c\in\{\text{vehicle},\text{pedestrian},\text{cyclist}\} and length, width and height of the 3D bounding box of the agent. Compared to conventional motion prediction task, which assumes a fixed agent set \mathcal{I}=\{1,\dots,N\} and access to agent history \{\mathbf{a}^{(i)}_{1:t}\}_{i\in\mathcal{I}} , and forecasts future trajectories \{\mathbf{a}^{(i)}_{t+1:T}\}_{i\in\mathcal{I}} , scenario generation task must create the initial agent set and continually inject new agents, traffic - light changes, and motions over a horizon T : p_{\theta}\bigl(\mathcal{S}_{1},\ldots,\mathcal{S}_{T}\bigr|\mathcal{M}\bigr) with \mathcal{S}_{t}=\bigl(\{\mathbf{l}^{(k)}_{t}\},\{\mathbf{a}^{(i)}_{t}\}\bigr) . 2.1 Scenario as a Token Sequence We cast scenario generation as a next - token prediction task: map tokens <MAP> are followed each step by traffic - light tokens <TL> , agent - state tokens <AS> , and agent - motion tokens <MO> and form a single autoregressive token sequence: \mathbf{x}_{1:T}=\;\bigl[\texttt{\textless MAP\textgreater};(\texttt{\textless TL\textgreater},\texttt{\textless AS\textgreater},\texttt{\textless MO\textgreater})_{1};(\texttt{\textless TL\textgreater},\texttt{\textless AS\textgreater},\texttt{\textless MO\textgreater})_{2};\ldots\bigr]. Given all tokens \mathbf{x}_{<t} generated so far, the model predicts the next token or next set of tokens p_{\theta}(x_{t}\mid\mathbf{x}_{<t}) and samples x_{t} . Following this