One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control

Zhenxing Mi, Yuxin Wang, Dan Xu · 2025

Read One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control by Zhenxing Mi, Yuxin Wang, Dan Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18922.txt.

Introduction

Zhenxing Mi, Yuxin Wang, and Dan Xu present One4D, a single framework that both generates and reconstructs four-dimensional content, meaning moving three-dimensional scenes. Its output pairs colour frames with point maps that give the scene's geometry over time. A unified masking scheme handles inputs of any sparsity, so the same system can generate a scene from one image. Reconstruct it from a full video, or mix both jobs from a few frames.

The design adapts a strong video generation model to emit colour and geometry jointly. Standard fine-tuning for that joint task quickly degrades the base model, so the team splits control between two adapters, one per output type, in separate branches. The result is a flexible tool for dynamic scene creation from whatever footage is at hand.

Catalog record

CREATORZhenxing Mi, Yuxin Wang, Dan Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18922

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control Zhenxing Mi, Yuxin Wang, Dan Xu Abstract We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D . Figure 1 : One4D supports single-image-to-4D generation, sparse-frame-to-4D generation, and full-video reconstruction in a single model. It outputs synchronized RGB frames and pointmaps, visualized as 4D point clouds with cameras, and RGB-depth sequences. 1 Introduction Simulating the dynamics of the physical world has progressed rapidly with video diffusion and flow-matching models [ 2 , 3 , 63 , 19 , 53 , 36 , 39 , 17 ] . Recent open systems such as Wan [ 39 ] , HunyuanVideo [ 17 ] , and Cosmos [ 23 ] demonstrate remarkable visual fidelity and strong understanding of real-world dynamics. However, these foundation models operate purely in RGB space while lacking explicit geometry. Augmenting them with accurate geometry generation is a key step toward downstream world-simulation tasks such as spatial reasoning [ 50 ] . Concurrently, scalable 3D/4D foundation models have advanced rapidly. Dust3R [ 43 ] introduces a pointmap representation encoding both geometry and camera information, enabling efficient feedforward 3D reconstruction. Monst3R [ 56 ] and VGGT [ 40 ] , etc., show that pointmaps are effective for static 3D and dynamic 4D reconstruction. Meanwhile, several works extend video or multi-view diffusion models with 6D video representations (RGB+XYZ) for 4D reconstruction [ 14 ] , 3D world generation [ 59 ] , 3D generation and reconstruction [ 33 ] , and 4D generation [ 5 ] . However, these methods typically specialize in either reconstruction or generation, or are restricted to static 3D scenes. In this paper, we take a step further and propose One4D, a unified 4D generation and reconstruction framework that significantly enhances the fidelity of joint RGB-geometry modeling through innovative network designs. We represent each 4D scene as RGB frames and pointmaps due to their flexibility and scalability, where pointmaps are 3-channel 2D (XYZ) videos analogous to RGB videos. Prior work [ 14 , 59 , 33 , 5 ] shows that modern video VAEs can effectively encode and decode pointmaps. A central challenge in joint RGB-geometry modeling is to fully exploit the strong priors of a pretrained video diffusion model to generate distinct modalities. Existing diffusion-based geometry methods [ 15 , 14 , 59 ] typically couple RGB and geometry through channel-wise concatenation, which in our setting causes severe cross-modal interference and rapid degradation under low-resource fine-tuning. To address this, we introduce the Decoupled LoRA Control (DLC), an efficient adaptation design guided by three goals: (i) preserve the base model’s strong video priors under low-resource finetuning; (ii) decouple RGB and geometry generation to reduce mutual interference; (iii) enable sufficient cross-modal communication for pixel-level consistency. Concretely, DLC attaches two decoupled and modality-specific LoRA adapters to the video backbone, one for RGB and the other for geometry. The adapters share the frozen base parameters but not the base forward computation, forming two fully decoupled computation branches . The RGB branch maintains pretrained video quality, while the geometry branch adapts to the “geometry video” distribution. To synchronize the two modalities, we add zero-initialized control links [ 57 ] between a few corresponding layers across branches. These links gradually learn to transmit information for precise pixel-wise alignment. In practice, DLC prevents mode collapse and yields accurate geometry without sacrificing RGB fidelity. To unify 4D generation and reconstruction tasks, we further propose Unified Masked Conditioning (UMC). UMC packs different condition types into a single conditioning video that differs in the sparsity of observed frames, filling unobserved frames with zeros. A single-image (with text) corresponds to pure generation. A set of sparse frames corresponds to mixed generation and reconstruction, and a full video corresponds to pure reconstruction. The conditioning video is fed to the diffusion backbone to guide the network to generate missing RGB frames and full pointmaps. With UMC, One4D transitions seamlessly between generation and reconstruction without any architectural changes. We train One4D on a curated mixture of synthetic and real 4D datasets, combining accurate synthetic geometry and in-the-wild appearance. The resulting model achieves generalizable, high-quality 4D generation and reconstruction in a single framework. Our main contributions are: • We introduce One4D, a unified 4D framework that bridges 4D generation and 4D reconstruction within a single video diffusion model, achieving strong performance across diverse dynamic scenarios. • We propose Decoupled LoRA Control (DLC), which uses modality-specific LoRA branches and zero-initialized control links to preserve video priors while enabling accurate, consistent joint RGB-geometry generation. • We design Unified Masked Conditioning (UMC), a single conditioning interface that supports various 4D generation and reconstruction tasks without model changes. 2 Related Work Video generation models. Video generation models [ 24 , 63 , 19 , 39 , 17 , 53 , 10 , 23 ] have advanced rapidly with diffusion [ 31 , 11 , 30 , 28 , 26 ] and flow-matching models [ 20 , 6 ] . Modern systems typically combine 3D causal VAE [ 16 , 2 , 55 , 46 ] with DiTs [ 26 ] to model spatio-temporal dynamics. Recent open-weight models such as Wan [ 39 ] , HunyuanVideo [ 17 ] , and Cosmos [ 23 ] demonstrate strong world modeling abilities, suggesting possibilities for geometry generation. Built upon powerful base text-to-video models, image-to-video systems such as Wan [ 39 ] and LongCat-Video [ 37 ] unify image-to-video, frame interpolation, and video continuation via frame inpainting within a single architecture. One4D follows a similar spirit to unify the 4D generation and reconstruction tasks, with specific designs for joint RGB and geometry generation. Scalable geometry modeling. DUSt3R [ 43 ] introduces a paired pointmap representation that encodes both geometry and camera information in a 2D map, enabling scalable 3D reconstruction with transformers. Subsequent works [ 56 , 40 , 34 , 51 , 42 , 14 ] extend this representation to multiview and dynamic reconstruction. Our method leverages these advances by adopting pointmaps as the geometry representation for joint RGB–geometry generation. Figure 2 : Architecture comparison for joint RGB and geometry modeling. (a) Channel-wise and (b) spatial-wise concatenation feed

Read the complete text