MiLA: M ulti-view I ntensive-fidelity L ong-term Video Generation World Model for A utonomous Driving

Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, Bing Wang · 2025

Read MiLA: M ulti-view I ntensive-fidelity L ong-term Video Generation World Model for A utonomous Driving by Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, Bing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15875.txt.

Introduction

Haiguang Wang and seven co-authors generate long, multi-view driving videos with a world model. Their 2025 paper presents MiLA for autonomous driving data. Rare scenes are costly to film yet vital for training drivers. Synthetic video could supply them if it stayed consistent.

Videos run up to a minute without falling apart. A coarse-to-fine pass steadies the stream and fixes moving objects. A progressive denoising schedule spreads cleanup over time. A joint module denoises and corrects in one motion. On the nuScenes benchmark, quality topped rival generators. Long, labeled, multi-camera footage can now be made on demand.

Catalog record

CREATORHaiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, Bing Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MiLA: M ulti-view I ntensive-fidelity L ong-term Video Generation World Model for A utonomous Driving Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, Bing Wang Abstract In recent years, data-driven techniques have greatly advanced autonomous driving systems, but the need for rare and diverse training data remains a challenge, requiring significant investment in equipment and labor. World models, which predict and generate future environmental states, offer a promising solution by synthesizing annotated video data for training. However, existing methods struggle to generate long, consistent videos without accumulating errors, especially in dynamic scenes. To address this, we propose MiLA, a novel framework for generating high-fidelity, long-duration videos up to one minute. MiLA utilizes a Coarse-to-Re(fine) approach to both stabilize video generation and correct distortion of dynamic objects. Additionally, we introduce a Temporal Progressive Denoising Scheduler and Joint Denoising and Correcting Flow modules to improve the quality of generated videos. Extensive experiments on the nuScenes dataset show that MiLA achieves state-of-the-art performance in video generation quality. For more information, visit the project website: https://github.com/xiaomi-mlab/mila.github.io . Figure 1 : (a) Comparison between Vista and the proposed MiLA for long video generation. Top: The generation frames of Vista. Bottom: The generation frames of our MiLA. Middle: A timeline in increasing order, where the value in second indicates the capability in generation length of other approaches, and the temporal time for generated frames. As seen in the red boxes, the performance of Vista significantly degrades after 8 seconds. In contrast, the definition of our MiLA remains largely unaffected, as highlighted in the green boxes. Furthermore, the 30-second frame predicted by MiLA maintains acceptable clarity throughout. (b) Visualization of videos generated by MiLA given different waypoints based on identical initial frames. † † footnotetext: ∗ Equal contribution. ◇ Work done while an intern at Xiaomi EV. † Project leader. 🖂 Corresponding author. 1 Introduction Figure 2 : An illustration of the existing Recurrent and the existing Divide-and-Conquer generation frameworks. Left: The generation of newly predicted frames always conditioned the previously generated frames recursively with an LDM [ 12 , 20 ] . Right: Anchor frames are firstly drawn by an LDM, and frames between anchor frames are interpolated in parallel with an Interpolation Model [ 3 , 49 ] . Data-driven approaches have demonstrated remarkable success in full-autonomous driving systems [ 21 , 8 , 25 , 45 ] . However, building robust systems requires the collection of long-tail and rare scenario data, demanding substantial ongoing investments in equipment and human resources. In response, world model systems [ 12 , 29 , 20 , 39 , 52 , 54 , 47 , 30 , 40 ] capable of understanding and predicting future environmental states have emerged as a promising solution. By processing historical observations and human instructions to generate scene representations, which can synthesize consistent sequences, effectively addressing out-of-distribution challenges [ 9 , 38 ] . Recent years have witnessed significant progress in world model frameworks capable of generating high-quality videos with planning signals [ 39 , 52 , 40 , 24 ] . However, two significant challenges persist in generating consistent long-duration data. The first challenge is the accumulated error problem. While generating all frames simultaneously would be ideal, it demands impractical computational resources and memory requirements. Consequently, most approaches adopt a batch-by-batch strategy for long video generation [ 12 , 29 , 33 , 27 ] , predicting new frames recursively based on previously generated ones. This approach, unfortunately, leads to error accumulation throughout the generation process, degrading the quality of both static backgrounds and dynamic objects in long-term videos. The second challenge lies in maintaining scene consistency over extended sequences. When conditioned only on the information with the initial timestamps, including conditional frames and scene descriptions, the control influence of the first frame gradually diminishes over subsequent frames, often resulting in scene degradation and loss of coherence in later portions of the sequence. While existing approaches have attempted to address this by incorporating extra control signals [ 30 , 1 , 11 , 48 ] , such as HD-maps and 3D bounding boxes of the predicted frames. It remains largely unexplored to learn a representation without these signals while still maintaining spatial-temporal consistency. To address these limitations, we present MiLA, a robust video generation framework built upon Latent Diffusion Models (LDM) [ 35 , 2 ] capable of producing high-fidelity long-term videos. Precisely, our approach focuses on maintaining long-term consistency and stability while relying solely on simple scene descriptions or waypoints as control signals. Comparison with approaches that utilized the existing Recurrent framework and demonstration of waypoints controllability are shown in Fig. 1 . We explore a divide-and-conquer video generation framework [ 3 , 49 , 26 , 13 , 22 ] , by first computing low FPS anchor frames in the Coarse process, and completing the video with high FPS video with interpolation models. However, artifacts and uncertainty in the predicted anchor frames that are caused by loss of temporal coherence, remain a hindrance of fidelity. To this end, we introduce a Coarse-to-(Re)fine module, which kills two birds with one stone. This module is capable of rectifying artifacts of the predicted anchor frames predicted in the Coarse process, meanwhile, enhancing the overall smoothness of the long video in the (Re)fine generation process. Without loss of generality, (Re) stands for the correction of anchor frames and fine process for the generation of high FPS frames. Furthermore, we develop a Temporal Progressive Denoising Scheduler module that enriches temporal information throughout the generation pipeline and improves the overall fidelity. In total, our contributions are three-fold: • We propose MiLA, a high-fidelity long-term video generation world model framework, which is tightly controlled by control signals and previous frames. Through our coarse-to-(Re)fine process, the framework successfully produces extended driving scenes lasting up to one minute without degradation of fidelity. • To further explore the potential of our framework of long video generation, we incorporate a Joint Denoising and Correcting Flow ( JDC ) approach that addresses imperfections in predicted anchor frames while optimizing overall video smoothness. Furthermore, we propose a temporal Progressive Denoising Scheduler module ( TPD ) to enhance the fidelity of final frames with the guidance of previously generated frames within the batch. • We validate our approach through comprehensive experiments, demonstrating state-of-the-art performance in both short and long-video generation on the nuScenes. 2 Related Work Figure 3 : The overall pipeline of the proposed MiLA. Top: The Coarse-to-(Re)fine generation framework. The conditions of the current (Re)fine stage involve both the corrected high FPS frames from the previous (Re)fine stage and the predicted low FPS anchor frames from the Coarse stage. The low FPS anchor frames are jointly denoised with high FPS interpolation frames to correct the artifacts. Bottom : The generation model structure of MiLA. The inputs of the model include multi-view video (condition frames, noise, and anchor frames occasionally), FPS, waypoints, camera parameters, and brief textual scene descriptions. After encoding multi-modal inputs with different encoders, a DiT-based framework denoises the noised video tokens for N times and outputs the prediction video. 2.1 World Models World Models refer to generative models that can learn to simulate an environment

Read the complete text