LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, Wei Tsang Ooi · 2025
Read LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences by Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, Wei Tsang Ooi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03692.txt.
Introduction
Ao Liang and seven colleagues present LiDARCrafter, a framework for making and editing dynamic LiDAR scenes, in a paper from twenty twenty-five. Generative world models now serve as data engines for self-driving cars. Most work targets video or occupancy grids. It overlooks the distinct traits of LiDAR. Four-dimensional LiDAR modeling that changes over time raises problems of control, coherence, and evaluation.
Given free-form language instructions, the system reads them into scene graphs centered on the ego vehicle. A tri-branch diffusion network then builds object structures, motion paths, and geometry. An autoregressive module yields sequences that hang together over time, with smooth transitions. The team also built a benchmark with metrics at scene, object, and sequence level. On the nuScenes dataset, LiDARCrafter reached top marks for fidelity, controllability, and consistency over time. Code and benchmark were released to the community.
Catalog record
| CREATOR | Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, Wei Tsang Ooi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03692
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, Wei Tsang Ooi Abstract Generative world models have become essential data engines for autonomous driving, yet most existing efforts focus on videos or occupancy grids, overlooking the unique LiDAR properties. Extending LiDAR generation to dynamic 4D world modeling presents challenges in controllability, temporal coherence, and evaluation standardization. To this end, we present LiDARCrafter , a unified framework for 4D LiDAR generation and editing. Given free-form natural language inputs, we parse instructions into ego-centric scene graphs, which condition a tri-branch diffusion network to generate object structures, motion trajectories, and geometry. These structured conditions enable diverse and fine-grained scene editing. Additionally, an autoregressive module generates temporally coherent 4D LiDAR sequences with smooth transitions. To support standardized evaluation, we establish a comprehensive benchmark with diverse metrics spanning scene-, object-, and sequence-level aspects. Experiments on the nuScenes dataset using this benchmark demonstrate that LiDARCrafter achieves state-of-the-art performance in fidelity, controllability, and temporal consistency across all levels, paving the way for data augmentation and simulation. The code and benchmark are released to the community. Figure 1: We propose LiDARCrafter , a 4D LiDAR-based generative world model that supports controllable point cloud layout generation ( left ), dynamic sequential scene generation ( center ), and rich scene editing applications ( right ). Our framework enables intuitive “what you describe is what you get” LiDAR-based 4D world modeling. Project Page — https://lidarcrafter.github.io GitHub — https://github.com/lidarcrafter/toolkit 1 Introduction Generative world models are rapidly advancing the synthesis and interpretation of large-scale sensor data for autonomous driving ( Hu et al. 2023 ; Mei et al. 2024 ) . Recent work has concentrated on structured modalities such as video and occupancy grids, whose dense, regular structure fits image and voxel pipelines ( Gao et al. 2023 ; Wang et al. 2024a ) . LiDAR, though indispensable for metric 3D geometry and all-weather perception, remains underexplored. The point clouds acquired by LiDAR sensors are often sparse, unordered, and irregular ( Kong et al. 2023b ; Liang et al. 2025 ; Liu et al. 2023 ; Xu et al. 2024 ) . Therefore, techniques designed for images or grids do not transfer cleanly. Early efforts, such as LiDARGen, project full 360^{\circ} scans to range images and borrow pixel-based methods ( Zyrianov, Zhu, and Wang 2022 ) . Later approaches improve single-frame fidelity but stop short of dynamic sequences ( Nakashima and Kurazume 2024 ; Ran, Guizilini, and Wang 2024 ) . Multimodal systems like UniScene and GENESIS rely on occupancy or video as intermediaries, limiting LiDAR independence and increasing computation ( Li et al. 2025 ; Guo et al. 2025 ) . A dedicated 4D LiDAR world model is therefore still absent. Bridging this gap requires progress on three fronts. First, user controllability . Text prompts are the most accessible interface, yet they lack the spatial detail to specify LiDAR scenes, whereas precise inputs such as 3D boxes, HD maps, or trajectories demand costly annotation ( Hu et al. 2023 ; Bian et al. 2025 ) . Second, temporal consistency . A single frame cannot reveal occlusion patterns or object kinematics for downstream utilization, necessitating a full 4D generation model. Conventional cross-frame attention strategies neglect the geometric continuity inherent in point clouds. Third, standardized evaluation . Existing video world models benefit from established benchmarks that score visual quality, perception accuracy, and closed-loop applicability ( Huang et al. 2024 ) . LiDAR still lacks standard protocols for evaluating fidelity and consistency across perspectives. To close these gaps, we introduce LiDARCrafter , a unified framework for controllable 4D LiDAR sequence generation. The key is an explicit, object-centric 4D layout that captures geometry and motion while achieving accessible yet precise control . In Text2Layout , a large language model (LLM) lifts the user prompt to an ego-centric scene graph, and a tri-branch diffusion network predicts object boxes, trajectories, and shape priors, yielding the full 4D layout. Layout2Scene feeds the layout to a range-image diffusion network and produces a high-fidelity first scan. The explicit layout modeling also enables fine-grained control like insertion, deletion, and dragging. Scene2Seq then generates the remaining frames autoregressively , warping past points with motion priors to keep temporal coherence. To standardize 4D LiDAR world models assessment, we release an evaluation suite that scores scene, object, and sequence quality. Experiments on the nuScenes dataset ( Caesar et al. 2020 ) demonstrate that our method achieves top performance in both single-frame fidelity and sequence coherence, while offering intuitive control and editing capabilities. This work establishes a new benchmark for LiDAR-based dynamic world modeling in autonomous driving. In summary, the core contributions of this work are: • We present LiDARCrafter , the first 4D generative world model dedicated to LiDAR data, with superior controllability and spatiotemporal consistency. • We introduce a tri-branch 4D layout conditioned pipeline that turns language into an editable 4D layout and uses it to guide temporally stable LiDAR synthesis. • We propose a comprehensive evaluation suite for LiDAR sequence generation, encompassing scene-level, object-level, and sequence-level metrics. • We demonstrate best single-frame and sequence-level LiDAR point cloud generation performance on nuScenes, with improved foreground quality over existing methods. Figure 2: Framework of LiDARCrafter . In the Text2Layout stage ( cf. Sec. 3.1 ), the natural-language instruction is parsed into an ego-centric scene graph, and a tri-branch diffusion network generates 4D conditions for bounding boxes, future trajectories, and object point clouds. In the Layout2Scene stage ( cf. Sec. 3.2 ), a range-image diffusion model uses these conditions to generate a static LiDAR frame. In the Scene2Seq stage ( cf. Sec. 3.3 ), an autoregressive module warps historical points with ego and object motion priors to generate each subsequent frame, producing a temporally coherent LiDAR sequence. 2 Related Work Driving Generative World Models. Generative world models aim to simulate environmental dynamics to aid autonomous driving decisions. Recent works primarily use video and occupancy representations. Video-based methods like GAIA-1 ( Hu et al. 2023 ) , Dreamforge ( Mei et al. 2024 ) , and Epona ( Zhang et al. 2025 ) employ autoregressive modeling and richer conditioning for improved synthesis. MagicDrive ( Gao et al. 2023 ) utilizes a bird’s-eye view (BEV) for temporal consistency ( Wang et al. 2024b ; Zhao et al. 2025 ; Ni et al. 2025a ) . Occupancy-based methods, such as OccWorld ( Zheng et al. 2024a ) , GaussianWorld ( Zuo et al. 2025b ) , QuadricFormer ( Zuo et al. 2025a ) , OccSora ( Wang et al. 2024a ) , and DynamicCity ( Bian et al. 2025 ) , provide structured spatial representations beneficial for downstream tasks. Multimodal models like UniScene ( Li et al. 2025 ) and GENESIS ( Guo et al. 2025 ) align cross-modal features for consistency. Despite its importance, LiDAR remains underexplored, mostly limited to forecasting-oriented methods ( Liu, Zhao, and Rhinehart 2025 ; Zhang et al. 2023a ) . LiDAR Point Cloud Generation. Early LiDAR generative methods, including LiDARGen ( Zyrianov, Zhu, and Wang 2022 ) , projected LiDAR point clouds into range images. Subsequent approaches like RangeLDM ( Hu, Zhang, and Hu 2024 ) , Text2LiDAR ( Wu