DreamWorld: Unified World Modeling in Video Generation
Boming Tan, Xiangdong Zhang, Ning Liao, Yuqing Zhang, Shaofeng Zhang †, Xue Yang, Qi Fan, Yanyong Zhang · 2026
Read DreamWorld: Unified World Modeling in Video Generation by Boming Tan, Xiangdong Zhang, Ning Liao, Yuqing Zhang, Shaofeng Zhang †, Xue Yang, Qi Fan, Yanyong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00466.txt.
Introduction
Tan and nine colleagues give video generators a fuller world model instead of surface plausibility alone. Past systems add only one kind of world knowledge or align it rigidly, missing the joint mix of physical sense, three dimensional shape, and time consistency. DreamWorld predicts video pixels and foundation model features together, then steadies training with gradual constraint tightening and enforces learned priors at inference. Tests show stronger world consistency, beating Wan2.1 by 2.26 points on VBench.
Catalog record
| CREATOR | Boming Tan, Xiangdong Zhang, Ning Liao, Yuqing Zhang, Shaofeng Zhang †, Xue Yang, Qi Fan, Yanyong Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44130 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00466
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DreamWorld: Unified World Modeling in Video Generation Boming Tan, Xiangdong Zhang, Ning Liao, Yuqing Zhang, Shaofeng Zhang †, Xue Yang, Qi Fan, Yanyong Zhang Abstract Despite impressive progress in video generation, existing models remain limited to surface-level plausibility, lacking a coherent and unified understanding of the world. Prior approaches typically incorporate only a single form of world-related knowledge or rely on rigid alignment strategies to introduce additional knowledge. However, aligning the single world knowledge is insufficient to constitute a world model that requires jointly modeling multiple heterogeneous dimensions (e.g., physical commonsense, 3D and temporal consistency). To address this limitation, we introduce DreamWorld , a unified framework that integrates complementary world knowledge into video generators via a Joint World Modeling Paradigm , jointly predicting video pixels and features from foundation models to capture temporal dynamics, spatial geometry, and semantic consistency. However, naively optimizing these heterogeneous objectives can lead to visual instability and temporal flickering. To mitigate this issue, we propose Consistent Constraint Annealing (CCA) to progressively regulate world-level constraints during training, and Multi-Source Inner-Guidance to enforce learned world priors at inference. Extensive evaluations show that DreamWorld improves world consistency, outperforming Wan2.1 by 2.26 points on VBench. Code will be made publicly available at Github . † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Limitations of extending VideoREPA to multi-source knowledge, leading to structural implausibility and unnatural distortion. The physics score(PC) on the Videophy benchmark dropped from 29.7 to 24.1 . The landscape of text-to-video (T2V) generation has been transformed by scalable diffusion transformers ( Peebles & Xie 2023 ; Ma et al. 2025 ) , pushing the field beyond short visual clips toward the ambition of general-purpose world models ( Ha & Schmidhuber 2018 ; Brooks et al. 2024 ) . Although state-of-the-art systems such as Lumiere ( Bar-Tal et al. 2024 ) , OpenSora ( Zheng et al. 2024 ) , and Wan2.1 ( Wan et al. 2025 ) achieve impressive cinematic fidelity through large-scale data and computation, a fundamental gap remains: these models largely function as visual generators rather than world simulators. Optimized primarily for pixel-level distribution matching ( Kang et al. 2025 ; Berg et al. 2025 ) , they fail to internalize structured and comprehensive world knowledge, which is reflected in their limited performance on world-centric benchmarks such as VBench 2.0 ( Zheng et al. 2025b ) . To mitigate the gap between visual realism and world understanding, prior work ( Wu et al. 2025a ; Zhang et al. 2025 ) has explored injecting external world knowledge into video generation models. A representative direction is Representation Alignment (REPA) ( Yu et al. 2025b ) , which aligns generative models with pretrained experts to transfer structured priors. Building on this paradigm, VideoREPA ( Zhang et al. 2025 ) introduces Token Relation Distillation (TRD) as a soft alignment strategy, demonstrating its effectiveness in distilling knowledge from a single expert model. However, constructing a holistic world model inherently requires the simultaneous integration of multiple heterogeneous knowledge sources . Our empirical investigations reveal that naively extending the REPA to concurrently align with semantic, spatial, and dynamic experts leads to a multi-objective optimization dilemma. As illustrated in Figure 1 , conflicting relational gradients from distinct teacher models induce optimization instability, suggesting that relational alignment is insufficient for the multi-source integration required by a unified world model (see Appendix A ). In response to these limitations, we propose DreamWorld , a unified framework designed to internalize comprehensive world knowledge through a Joint World Modeling Paradigm . Inspired by VideoJAM ( Chefer et al. 2025 ) , DreamWorld extends standard video latents into a composite feature space, compelling the model to predict video pixels alongside a set of world features. Specifically, DreamWorld integrates temporal dynamics from Optical Flow, spatial geometry from VGGT ( Wang et al. 2025a ) , and semantic understanding from DINOv2 ( Oquab et al. 2024 ) . However, the direct superposition of such heterogeneous optimization objectives frequently induces optimization instability and temporal flickering. To mitigate this, we propose Consistent Constraint Annealing (CCA) , a decay mechanism that guarantees convergence by progressively modulating the influence of world knowledge, thereby ensuring high-fidelity visual generation while effectively assimilating world priors. Moreover, a Multi-Source Inner-Guidance mechanism is incorporated at inference time, which leverages the model’s own predicted knowledge features to steer the generation process, ensuring trajectories that strictly adhere to real-world laws. Our main contributions are summarized as follows: i) We present DreamWorld , the first unified video generation framework to integrate multi-source world knowledge, including 3D Semantic consistency, Motion Temporal dynamics, and 2D Spatial geometry. ii) We propose a novel training strategy, Consistent Constraint Annealing (CCA) , which harmonizes knowledge injection with visual quality, ensuring coherent and artifact-free generation. iii) Extensive evaluations demonstrate that DreamWorld significantly outperforms baselines and VideoJAM, establishing a new standard for world models. 2 Related Work Video Diffusion Models. The paradigm of video generation has been revolutionized by the adoption of Diffusion Transformer (DiT) architectures ( Peebles & Xie 2023 ; Ma et al. 2025 ) Following the scaling laws originally observed in language modeling ( Kaplan et al. 2020 ) and validated in video ( Tong et al. 2025 ) , recent state-of-the-art systems such as Wan2.1 ( Wan et al. 2025 ) and Hunyuan-Video ( Kong et al. 2025 ) have achieved photorealistic results by training on massive video-text corpora. To enhance training stability and inference efficiency beyond standard diffusion, Flow Matching ( Lipman et al. 2023 ) has emerged as a superior generative framework, as successfully implemented in LTX-Video ( HaCohen et al. 2026 ) and Pyramid Flow ( Jin et al. 2025 ) . Furthermore, the open-source community has democratized access to high-fidelity generation through open-weights models like Mochi 1 ( Team 2024 ) and CogVideoX ( Yang et al. 2025 ) . Despite these achievements, pure diffusion-based approaches often struggle with real-world understanding, lacking the inherent capability for maintaining global consistency ( Qin et al. 2024 ) . Representation Alignment. Representation Alignment (REPA) ( Yu et al. 2025a ) addresses the lack of structural awareness in pixel-space diffusion by injecting high-level semantic priors from pre-trained foundation models. Building on this, recent advancements have introduced more sophisticated alignment mechanisms ( Zheng et al. 2025a ; Jiang et al. 2025 ; Lee et al. 2025 ; Li et al. 2025 ; Zhao et al. 2025 ) . In the video domain, AlignVid ( Liu et al. 2025 ) , VideoREPA ( Zhang et al. 2025 ) and MoAlign ( Bhowmik et al. 2025 ) extend this mechanism to spatio-temporal alignment, enforcing adherence to a coherent semantic layout in generated frames. Although these methods offer a pathway to more robust video generation, they typically focus on appearance consistency rather than dynamic causal logic. World Modeling. The concept of World Models ( Ha & Schmidhuber 2018 ) transcends video generation, aiming for a rigorous understanding of the environment’s underlying laws to predict and simulate the world. Genie ( Bruce et al. 2024 ) and Genie 2 ( Parker-Holder et al. 2024 ) instantiate this as an interactive simulator, learning a latent action space for controlling video rollouts. Alternatively, the Joint-Embedding Predictive Architecture (JEPA), exemplified by V-JEPA