LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

Hao Shao, Letian Wang, Yang Zhou, Yuxuan Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li, [10pt] CUHK MMLab · 2026

Read LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving by Hao Shao, Letian Wang, Yang Zhou, Yuxuan Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li, [10pt] CUHK MMLab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08719.txt.

Introduction

Shao and seven colleagues present LMGenDrive, the first framework joining language model multimodal understanding to generative world models for closed loop driving end to end. From multi view cameras and natural language instructions it generates future driving videos and control signals together, so video prediction sharpens scene modeling while language priors ground instructions. A three stage training path from vision pretraining to long horizon driving yields clear gains on hard closed loop benchmarks.

Catalog record

CREATORHao Shao, Letian Wang, Yang Zhou, Yuxuan Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li, [10pt] CUHK MMLab
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38362
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving Hao Shao, Letian Wang, Yang Zhou, Yuxuan Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li, [10pt] CUHK MMLab Abstract Recent years have witnessed remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains the primary bottleneck for large-scale deployment. To address this, one line of research explores LLMs and VLMs for their vision-language understanding and reasoning capabilities, equipping AVs with the ability to interpret rare and safety-critical situations when generating driving actions. In parallel, another line investigates generative world models to capture the spatio-temporal evolution of driving scenes, enabling agents to imagine and evaluate possible futures before acting. Inspired by human intelligence, which seamlessly unites understanding and imagination as a hallmark of AGI, this work explores a unified model that brings these two capabilities together for autonomous driving. We present LMGenDrive, the first framework that unifies LLM-based multimodal understanding with generative world models for end-to-end closed-loop autonomous driving. Given multi-view camera inputs and natural-language instructions, our model generates both realistic future driving videos and corresponding control signals. By coupling an LLM with generative video capabilities, LMGenDrive gains complementary benefits: future video prediction enhances spatio-temporal scene modeling, while the LLM provides strong semantic priors and instruction grounding learned from large-scale pretraining. A progressive three-stage training strategy—ranging from vision pretraining to multi-step long-horizon driving—is proposed to further improve stability and performance. The resulting model can also operate in two complementary modes: low-latency online planning and autoregressive offline video generation. Experiments show that LMGenDrive significantly outperforms state-of-the-art methods on challenging closed-loop driving benchmarks, improving instruction following, spatio-temporal understanding, and robustness to rare scenarios. Our work not only sets a new state-of-the-art in autonomous driving, but also demonstrates that unifying multimodal understanding and generation provides a promising direction for building more generalizable and robust embodied decision-making systems. 1 Introduction Remarkable progress in autonomous driving has been witnessed in recent years with an increasing number of commercial autonomous vehicles (AVs) deployed on public roads. Amidst this momentum, end-to-end autonomous driving has emerged as a particularly vibrant research direction. Unlike traditional modular pipelines that separately handle perception, prediction, and planning with handcrafted interfaces, end-to-end models provide a holistic paradigm with the potential to remove information bottlenecks among modules, better align model optimization with system-level performance, and scale effectively with large amounts of driving data. Despite this progress, the problem of generalization remains the central bottleneck for the entire autonomous driving community. As we approach the frontier of real-world deployment, the ability to robustly handle long-tail edge cases and operate in open-world settings remains the defining challenge for AV systems. These scenarios can include rare but safety-critical events, distribution shifts across regions, adversarial weather conditions, as well as complex social interactions and ambiguous intent among agents. This challenge manifests across the autonomy stack: perception systems struggle to identify open-set entities, while prediction and planning models falter in extrapolating to nondeterministic and previously unseen behaviors. These generalization failures represent the last barrier between research prototypes and truly scalable, globally deployable autonomous vehicles. Amid this backdrop, large language models (LLMs) have demonstrated strong capabilities in language understanding and structured decision modeling across a wide range of tasks. Recent models such as GPT-5 and DeepSeek-R1 ( 15 ) showcase robust capabilities in commonsense reasoning, abstraction, and decision-making. Meanwhile, vision-language models (VLMs) further extend this capacity to the multimodal domain, enabling unified interpretation of textual and visual inputs ( 51 ; 2 ; 31 ; 27 ; 1 ) . Inspired by these vision-language understanding and reasoning capabilities, a wave of research has begun exploring how to equip AV systems with LLMs and VLMs to address the open-world, long-tail challenges in autonomous driving. As exemplified in works such as LMDrive ( 41 ) and GPT-Driver ( 33 ) , these models act as the cognitive brains to interpret ambiguous scenarios and guide complex decision-making under uncertainty. However, most existing LLM- or VLM-empowered driving methods follow the paradigm that maps inputs directly to actions, falling short in explaining and capturing the temporal evolution of driving scenes—an essential factor for robust and anticipatory planning. Meanwhile, another stream of research, world modeling ( 17 ) , has emerged to simulate the spatio-temporal evolution of the scenes, as exemplified by video-based works such as Genie-3 ( 3 ) and Pandora ( 57 ) . Their potential has also been actively explored in the autonomous driving domain, enabling the agent to “imagine” different futures before committing to a plan. However, existing works either focus on solely generating high-fidelity scenes ( 20 ; 39 ; 12 ; 13 ; 23 ; 53 ; 60 ; 67 ; 42 ) , or utilize world models as a plug-and-play forecasting module to rank multiple possible plans ( 54 ; 52 ) . The integration of joint video generation and motion planning remains underexplored, limiting their ability to address critical challenges such as cumulative control errors, human-robot interaction, and the temporal consistency between generated actions and videos—factors that are essential for long-horizon problem solving in real-world systems. Moreover, these models generally lack the rich knowledge priors and instruction-following capabilities provided by LLMs. Figure 1 : Comparison between existing works and ours. Prior works either leverage LLMs/VLMs for multimodal understanding and reasoning, or adopt world models for video-based scene imagination, but treat these capabilities in isolation. In contrast, our proposed LMGenDrive unifies both within a closed-loop end-to-end framework: the LLM interprets and reasons over multimodal inputs, while the world model simulates future scene evolution, together enabling instruction-guided planning, spatio-temporal understanding, and robust long-horizon driving. In contrast to existing models that specialize in either understanding or generation, human cognition naturally integrates perception and imagination to support long-horizon decision-making. Inspired by this observation, we explore whether combining multimodal understanding with generative world modeling can improve robustness and temporal consistency in autonomous driving systems. While recent studies ( 9 ; 45 ; 6 ; 29 ) have shown encouraging results combining multimodal understanding and generation within a single model, whether this principle extends to embodied agents—and autonomous driving in particular—remains an open challenge. In this work, we propose LMGenDrive, the first framework that unifies LLM-based multimodal understanding with generative world models for closed-loop end-to-end autonomous driving. Our unified model takes multi-view camera data and natural-language driving instructions as inputs, generating both multi-view future driving videos and control signals for the following timesteps. Concretely, we integrate an LLM and a diffusion-based video generation model: the LLM interprets and fuses visual observations with language instructions, producing learnable queries that capture the evolving scene states, which then serve as conditioning signals for the diffusion model to generate realistic multi-view driving futures. Within this unified architecture, video generation enhances the LLM’s spatio-temporal understanding, while the LLM provides instruction-following and reasoning capabilities to the world model, together enabling more robust closed-loop driving. To enable such a unified model, we also propose a three-stage curriculum training strategy for enhanced performance and stability. First, we pretrain a vision encoder for robust driving scene understanding. Next, the frozen encoder is integrated with the LLM and video generator, and fine-tuned on single-step prediction to ground instruction following and immediate action outcomes. Finally, training is extended to multi-step

Read the complete text