Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
2026
Read Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18137.txt.
Introduction
Lijun Zhou's 2026 report joins two skills for self-driving cars: rebuilding a scene in three dimensions and making new video of it. One part builds compact 3D scenes from sparse queries spread across views and time, keeping frames in line with each other. The other part makes causal video in as few as four clean-up steps. Joined, the pair gives steadier video for closed-loop tests, fresh training data, and driver training.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45334 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving Abstract This report presents a unified technical system addressing the two core capabilities of world models for autonomous driving: world representation and world generation. For world representation, we propose WorldRec , a feed-forward reconstruction architecture driven by sparse scene queries. WorldRec initializes structured queries in 3D space, leveraging them to aggregate cross-view, cross-temporal features, thereby naturally enforcing spatial consistency across frames and yielding compact yet high-fidelity 3D Gaussian scene representations. For world generation, we propose WorldGen , a two-stage training framework of bidirectional pretraining followed by causal fine-tuning through three progressive stages (Teacher Forcing, ODE distillation, and DMD), enabling high-quality online causal video generation in as few as 4 denoising steps. Building on both modules, we further introduce the Joint World Model , which deeply integrates WorldRec and WorldGen to achieve synergistic gains in generation stability, cross-frame consistency, and visual fidelity, providing a solid foundation for closed-loop simulation, data synthesis, and end-to-end training in autonomous driving. \project https://JointWM.github.io 1 Overview Figure 1 : Comparison of reconstruction-only, generation-only, and our joint world model. World models 10 have emerged as a foundational paradigm for autonomous driving, enabling critical capabilities including data synthesis 41 ; 9 that alleviates the scarcity of long-tail driving scenarios, closed-loop training 5 that facilitates end-to-end policy optimization within differentiable environments, and closed-loop simulation 34 that provides high-fidelity virtual evaluation through photorealistic scene rendering. Recent literature has witnessed substantial progress from both the reconstruction perspective 34 ; 4 ; 35 ; 2 ; 24 and the generation side 12 ; 26 ; 7 ; 21 ; 9 , converging toward a hybrid reconstruction-generation paradigm. This paradigm typically proceeds in two stages: first constructing a 3D scene representation 15 from multi-view observations, then leveraging the resulting geometric prior to condition video generation models 11 ; 18 . While such an approach has demonstrated promising properties—notably spatiotemporal consistency and precise viewpoint controllability—current instantiations remain limited in scalability, real-time applicability, and the depth of integration between the two stages. We identify three principal bottlenecks. First , on the representation side, conventional per-scene optimization of 3D Gaussian primitives entails multi-hour training per sequence and exhibits poor generalization to novel viewpoints. Although recent feed-forward methods 35 ; 2 ; 24 mitigate the optimization cost, their reliance on Dense Prediction Transformer (DPT) heads for pixel-aligned Gaussian prediction introduces systematic deficiencies: independently predicted per-frame Gaussians inevitably produce ghosting artifacts and layered surface duplications upon concatenation in 3D space, while the primitive count scales to hundreds of millions per clip, imposing prohibitive rendering overhead. Second , on the generation side, causal diffusion models trained ab initio lack the expressive scene priors necessary for high-fidelity synthesis; the requirement of hundreds of denoising steps at inference precludes real-time deployment; and the well-documented exposure bias 19 inherent to autoregressive generation induces progressive content drift over extended horizons. Third , the coupling between reconstruction and generation in existing systems remains superficial—most approaches treat the two as disjoint modules, rendering it difficult to reconcile the geometric fidelity demanded by precise scene representation with the distributional diversity essential to generative prediction. Moreover, recent hybrid efforts such as NeoVerse 37 , while demonstrating compelling results for general 4D scene understanding from monocular videos, lack the domain-specific architectural considerations required by autonomous driving, including multi-camera geometric consistency, ego-motion conditioning, and structured layout control. This report presents a unified technical framework that systematically addresses the aforementioned challenges through three tightly integrated components. WorldRec formulates scene reconstruction as a sparse-query aggregation problem: a compact set of learnable 3D queries are initialized in world space and progressively enriched via cross-view, cross-temporal feature fusion with visibility-aware weighting, yielding geometrically consistent Gaussian scene representations that reduce reconstruction time from hours to approximately 10 seconds per clip while eliminating the ghosting and redundancy artifacts of prior feed-forward approaches. WorldGen introduces a two-stage training curriculum built upon a Diffusion Transformer (DiT) backbone conditioned on heterogeneous signals including ego trajectory, camera parameters, layout maps, and free-form text. Bidirectional pretraining with unrestricted temporal attention first acquires rich spatiotemporal scene priors; the model is subsequently converted to an autoregressive generator through progressive causal fine-tuning comprising Teacher Forcing for causal adaptation, ODE distillation for step reduction ( 50\to 4 steps, yielding {\sim}12\times inference acceleration), and Distribution Matching Distillation (DMD) for mitigating exposure bias—collectively enabling stable generation of up to one-minute driving videos at 0.19 s/frame. The Joint World Model achieves deep integration of both modules through complementary architectural extensions: an incremental scene fusion mechanism allows WorldRec to maintain and progressively expand a globally consistent 4D Gaussian representation as new observations arrive, while WorldGen is augmented with an ego-projected rendered-prior conditioning pathway that provides a coarse geometric scaffold for generation in unobserved regions while preserving photometric consistency where reconstruction coverage exists. Within this tightly coupled framework, the deterministic geometric constraints from WorldRec suppress error accumulation and content drift during long-horizon autoregressive generation, while the generative capacity of WorldGen compensates for spatially and temporally unreconstructed regions, yielding synergistic improvements along three axes: temporal stability, cross-view consistency, and visual fidelity. As illustrated in Figure 1 , reconstruction-only methods achieve geometrically precise scene recovery yet remain fundamentally confined to observed data, lacking the capacity to synthesize future or unseen viewpoints. Conversely, generation-only approaches model scene dynamics and afford controllable prediction, but the absence of an explicit 3D representation leads to weak geometric consistency and accumulated drift over long horizons. Our joint formulation unifies the complementary strengths of both paradigms: a compact 3D scene state serves as a structured geometric anchor for the generative process, while the generation model extends predictive capability beyond the observation boundary. The resulting closed-loop architecture establishes a principled foundation for closed-loop simulation, synthetic data generation, and end-to-end training in autonomous driving. Table 1 : Comparison of different world modeling paradigms. Recon-only Gen-only Neoverse AlpaDreams Ours Explicit 3D Scene ✓ ✗ ✓ ✗ ✓ Generative Capability ✗ ✓ ✓ ✓ ✓ Novel View Synthesis ✓ ✓ ✓ ✓ ✓ Future Prediction ✗ ✓ ✓ ✓ ✓ Geometry Consistency Strong Weak Medium Weak Strong Long-horizon Stability Static Drift Medium Medium Stable 2 Related Work 2.1 3D Scene Reconstruction. 3D scene reconstruction aims to recover accurate geometric structures and achieve photorealistic appearance rendering from multi-view observations. Early approaches relied on explicit representations such as point clouds and meshes, which struggle to capture fine surface details and view-dependent appearance. 3D Gaussian Splatting (3DGS) 15 has since emerged as the dominant paradigm, representing scenes as collections of explicit 3D Gaussian primitives rendered via efficient differentiable rasterization, offering a compelling combination of geometric expressiveness, rendering speed, and visual fidelity. Early work adapts 3DGS to driving scenes through iterative per-scene fitting. StreetGaussians 34 decomposes the scene into a static background and dynamic foreground vehicles, modeling each dynamic actor as a rigid group of Gaussian primitives with optimizable poses and 4D spherical harmonics to capture appearance variation across frames. DrivingGaussian 44 extends this with composite dynamic Gaussian graphs for multi-object modeling. To address non-rigid actors overlooked by prior work, OmniRe 4 builds dynamic neural scene graphs on 3DGS and constructs multiple canonical-space Gaussian representations covering vehicles, pedestrians, and cyclists, including skinned LBS models for deformable bodies, enabling holistic reconstruction