Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception

Siyuan Meng, Chengbo Ai · 2026

Read Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception by Siyuan Meng, Chengbo Ai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17651.txt.

Introduction

Meng and Ai argue for infrastructure-centric world models, a roadside complement to today's ego-vehicle world models for autonomous driving. Fixed roadside sensors offer temporal depth. They accumulate long-term behavior patterns including rare safety-critical events. Vehicle sensors offer spatial breadth across large road networks. The paper lays out Infrastructure-centric World Models in three phases: generative scene understanding with quality-aware uncertainty propagation, physics-informed predictive dynamics with multi-agent counterfactual reasoning, and a third phase on collaborative world models.

Catalog record

CREATORSiyuan Meng, Chengbo Ai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42851
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17651

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception Siyuan Meng, Chengbo Ai Abstract World models,generative AI systems that learn to simulate how environments evolve,have emerged as a cornerstone of next-generation autonomous driving, exemplified by the Waymo World Model, NVIDIA Cosmos, and a rapidly growing body of academic work (HERMES, ResWorld, DriveVLA-W0). However, all existing driving world models adopt an ego-vehicle perspective, leaving the infrastructure viewpoint entirely unexplored. We argue that infrastructure-centric world models offer a fundamentally different and complementary capability: the bird’s-eye, multi-sensor, persistent viewpoint that roadside systems uniquely possess. Central to our thesis is a fundamental spatio-temporal complementarity : fixed roadside sensors excel at temporal depth ,accumulating long-term behavioral distributions including rare safety-critical events at specific locations,while vehicle-borne sensors excel at spatial breadth ,sampling diverse scenes across large road networks. Neither platform alone can learn a complete model of traffic dynamics. This paper presents a research vision for developing Infrastructure-centric World Models (I-WM) organized in three phases: (I) generative scene understanding with quality-aware uncertainty propagation, (II) physics-informed predictive dynamics with multi-agent counterfactual reasoning, and (III) collaborative world models for vehicle-to-everything (V2X) communication via latent space alignment. We propose a dual-layer architecture,modular, annotation-free perception as a multi-modal data engine feeding end-to-end generative world models,accompanied by a phased multi-modal sensor strategy that progresses from LiDAR-centric core sensing through 4D radar and signal phase data to forward-looking modalities including event cameras and environmental sensors. We establish a taxonomy of driving world model paradigms, position I-WM relative to LeCun’s JEPA framework, Li Fei-Fei’s spatial intelligence vision, and Vision-Language-Action architectures, and introduce the concept of Infrastructure VLA (I-VLA) as a novel unification of roadside perception, natural language commands, and traffic control actions. Our vision builds upon existing multi-LiDAR perception pipelines (FRGB3D, MulDet3D, MulTrack3D) and identifies concrete open-source foundations (DynamicCity, OccWorld, CarDreamer) for each phase, providing an actionable path toward intelligent infrastructure that does not merely observe traffic but understands and anticipates it. Keywords: World Models, Infrastructure-Centric Perception, Roadside LiDAR, Multi-Modal Sensing, V2X Cooperative Perception, Spatial Intelligence, 4D Occupancy Forecasting 1 Introduction Current roadside perception systems primarily focus on instantaneous object detection and tracking. However, they lack a deep understanding of the “rules of the world”,such as physical constraints, causal relationships, and intent prediction. Meanwhile, industry leaders such as Waymo have recently introduced vehicle-centric world models built upon foundation models like Google DeepMind’s Genie 3, demonstrating the power of generative simulation for autonomous driving safety validation [ 38 ] . Concurrently, NVIDIA’s Cosmos platform [ 26 ] has established open-weight world foundation models for physical AI, while a rapidly growing body of work,including HERMES [ 51 ] , ResWorld [ 45 ] , and DriveVLA-W0 [ 22 ] ,continues to push the frontier of vehicle-centric driving world models. This paper proposes a 5-year research trajectory to develop Infrastructure-centric World Models (I-WM) that offer a fundamentally different and complementary perspective: the bird’s-eye, multi-sensor, persistent viewpoint that roadside systems uniquely possess. Unlike ego-vehicle world models that reason about the immediate surroundings of a single agent, I-WM reasons about the entire traffic ecosystem ,enabling predictive analytics, proactive safety interventions, and cognitive V2X collaboration. Central to our thesis is a fundamental spatio-temporal complementarity : roadside systems excel at temporal depth ,accumulating long-term behavioral distributions at fixed locations,while vehicle-based systems excel at spatial breadth ,sampling diverse scenes across large road networks. The I-WM framework is designed to harness this complementarity as a first-class design principle. Architecturally, we advocate a dual-layer design that separates concerns between perception and world modeling. The perception layer employs modular, unsupervised methods,background modeling (FRGB3D), multi-objective detection (MulDet3D), and consensus-based tracking (MulTrack3D),that operate without manual annotation and can be rapidly deployed at new sites. These serve as a structured data engine , providing the world model with high-quality, uncertainty-annotated observations (3D detections, trajectories, static scene priors). The world model layer then builds upon these structured inputs using end-to-end generative architectures,diffusion-based 4D scene generation, autoregressive occupancy forecasting, and self-supervised latent dynamics,to learn scene evolution, counterfactual reasoning, and future prediction. This separation is deliberate: roadside deployment demands annotation-free perception (since each new intersection lacks labeled data), while world models benefit from end-to-end learning over the structured representations that the perception layer produces. In the longer term, we envision a fully end-to-end roadside foundation model that unifies both layers, but the dual-layer approach provides an actionable near-term path that leverages existing validated components. 2 Industry Context: Why Infrastructure-Centric? The recent release of the Waymo World Model (February 2026) marks a milestone for vehicle-centric generative simulation. Built on Genie 3, it generates photorealistic camera and LiDAR data for rare edge-case scenarios, supports counterfactual reasoning via driving action control, and enables language-conditioned scene editing. However, Waymo’s approach is fundamentally ego-centric ,it models the world from a single vehicle’s perspective. Infrastructure-centric and vehicle-centric world models are complementary, not competing. The roadside “God’s-eye view” resolves occlusions invisible to any single vehicle, provides persistent monitoring across all traffic participants, and enables system-level optimization (e.g., signal control) that ego-centric models cannot achieve. Table 1 summarizes these differences. Table 1: Positioning: Vehicle-Centric vs. Infrastructure-Centric World Models. Dimension Waymo World Model (Vehicle-Centric) I-WM (Infrastructure-Centric) Viewpoint Ego-vehicle perspective Bird’s-eye / multi-sensor panoramic Core Strength Fine-grained ego-surroundings modeling Global traffic situational awareness Data Foundation \sim 200M miles of fleet driving data Long-term fixed-site deployment data Occlusion Handling Limited by ego viewpoint Multi-angle occlusion completion Control Dimensions Ego driving actions Traffic signals + multi-agent control Uncertainty Modeling No explicit reliability propagation Quality-aware framework throughout Counterfactual Scope Single ego-vehicle “what-if” Multi-agent, system-level “what-if” Downstream Users Single-vehicle autonomy Traffic management, V2X, urban planning Foundation Model Genie 3 post-training Open foundation models + roadside post-training 2.1 Temporal Depth vs. Spatial Breadth: The Fundamental Complementarity Beyond the functional differences outlined in Table 1 , roadside and vehicle-based sensing exhibit a deeper epistemological complementarity rooted in their deployment geometry,one that determines what each platform can learn about traffic dynamics. Roadside LiDAR: Temporal Deep Wells. A fixed roadside sensor continuously observes the same spatial region over days, weeks, or months. This produces a temporal deep well ,a dense time-series at a single location that captures the complete probability distribution of traffic behaviors, including the critical long tail of rare events (near-misses, wrong-way entries, jaywalking). Such temporal depth enables: (1) scene normality modeling ,learning what “normal” looks like at a specific intersection (cf. FRGB3D background models), so that deviations can be detected and quantified; (2) rare-event statistics ,accumulating sufficient samples of near-crash scenarios that a single passing vehicle would never encounter at the same location; and (3) counterfactual baselines ,when a near-miss occurs, the model can compare against the site’s empirical behavioral distribution to ask “what if the driver had not braked?”,a form of infrastructure-enabled counterfactual reasoning that is infeasible from a transient vehicle perspective. Vehicle-Borne LiDAR: Spatial Wide Nets. A vehicle-mounted sensor traverses many locations but observes each only briefly. This produces a spatial wide net ,a broad but shallow sampling of the road network that captures geometric diversity (road types, lane configurations, urban vs. rural), but offers only a snapshot of behavioral dynamics at any single point. Implications for World Model Design. This complementarity has concrete architectural consequences. Roadside I-WM learns deep temporal priors ,how traffic evolves over time at a specific

Read the complete text