DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation
Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, Xiaoyan Sun · 2025
Read DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation by Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, Xiaoyan Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14530.txt.
Introduction
Xiangchen Yin and seven co-authors propose DeCo-VAE, a video autoencoder that stores video in a more compact form. Standard video autoencoders waste capacity because nearby frames share much content. The new method splits video into a keyframe, motion, and a residual part. Each part gets its own encoder and latent representation.
A shared three dimensional decoder rebuilds the video while keeping space and time coherent. Training freezes some encoders while others learn, in stages. That staged plan avoids interference between static and dynamic features. Broad tests show better reconstruction quality than prior methods. The gain comes from removing redundancy rather than adding raw capacity.
Catalog record
| CREATOR | Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, Xiaoyan Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14530
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, Xiaoyan Sun Abstract Existing video Variational Autoencoders (VAEs) generally overlook the similarity between frame contents, leading to redundant latent modeling. In this paper, we propose decoupled VAE (DeCo-VAE) to achieve compact latent representation. Instead of encoding RGB pixels directly, we decompose video content into distinct components via explicit decoupling: keyframe, motion and residual, and learn dedicated latent representation for each. To avoid cross-component interference, we design dedicated encoders for each decoupled component and adopt a shared 3D decoder to maintain spatiotemporal consistency during reconstruction. We further utilize a decoupled adaptation strategy that freezes partial encoders while training the others sequentially, ensuring stable training and accurate learning of both static and dynamic features. Extensive quantitative and qualitative experiments demonstrate that DeCo-VAE achieves superior video reconstruction performance. 1 Introduction Video Variational Autoencoders (VAEs) transforms video frames into compact latent representation as a critical component of Latent Video Diffusion Models (LVDMs) 31 ; 10 . Several models such as Sora 8 , Open-Sora-Plan 27 , CogVideoX 46 , Stable Video Diffusion 7 have achieved powerful performance, the efficiency and quality of this process directly impact the performance of downstream generation tasks. Early video generation methods directly adopted image VAEs 31 in latent representation to perform video compression through frame-by-frame encoding. These methods fail to capture the temporal correlations between frames, essentially reducing videos to sequences of independent images. To solve this limitation, several methods 46 ; 12 ; 27 have employed dense 3D networks with heavy parameters to enhance spatiotemporal interactions. While improving reconstruction quality, these approaches cause exponential growth in network parameters and computational complexity, significantly compromising video reconstruction efficiency. In contrast, other methods 8 use lightweight 2+1D architectures, reducing computational costs through separating spatial and temporal convolutions. However, such lightweight designs struggle to model complex video dynamics and temporal dependencies. To balance efficiency and quality, recent advances in video VAEs 26 ; 36 ; 45 ; 28 have leveraged lightweight designs such as wavelet transforms, reducing computational overhead while better preserving critical visual information. Additionally, some approaches 40 establish different latent spaces to capture dynamics, but still cannot effectively decouple the motion information of the video. Figure 1 : (a) Visualization of decoupled components in DeCo-VAE, including keyframe, motion and residual components for video frames. (b) Visualization of t-SNE latent distributions in video decoupling, our DeCo-VAE achieves more compact latent space. (c) Performance comparison of video VAEs, our DeCo-VAE achieves superior reconstruction quality with lightweight parameters. Despite some progress made by these methods, they treats videos as monolithic data, without considering the high redundancy between consecutive frames. This creates a paradox: while video data is inherently highly redundant and should be easier to compress, lightweight architectures struggle to effectively leverage this redundancy for simplified modeling. Conversely, heavy networks capable of comprehensive modeling introduce unnecessary computational overhead for handling such redundant content. Video Codec 22 ; 42 ; 5 decomposes videos into keyframe, motion and residual components, as shown in Fig. 1 (a), keyframe contains static texture information and spatial structures, while residual and motion components only represent temporal differences. This line of thinking effectively removes redundancy in videos, offering a new research perspective for current video VAEs methods. By visualizing the latent distributions (both with and without video decoupling, as shown in Fig. 1 (b)), we observe that the latent space learned via video decoupling is significantly more compact, demonstrates the effectiveness of video de-redundancy. This tighter concentration of the latent distribution is validated by the clustered highlight regions in the visualization results. In this paper, we propose decoupled VAE (DeCo-VAE), a video VAE framework utilizing explicit video content decoupling. Instead of directly encoding raw pixels, we construct latent representation for the decoupled motion, residual and keyframe. The motion component focuses on inter-frame dynamic differences and the residual component captures fine-grained details, while the keyframe serve as appearance anchors to preserve basic textures and spatial structures. To avoid cross-interference between these components, we assign dedicated encoders to eliminate feature entanglement and use a shared 3D decoder to restore three latent representation, maintaining spatiotemporal consistency during reconstruction. To enhance training stability under complex constraint conditions, We employ a decoupled adaptation strategy, different encoders are frozen in each phase to train the counterpart sequentially. This staged approach avoids cross-component feature interference, ensuring precise learning of both static and dynamic features. Consequently, Our decoupled design enables strong performance on lightweight VAEs (Fig. 1 (c)), its latent representation serve as an efficient drop-in solution for downstream video generation tasks. Our main contributions are summarized as follows: • We propose DeCo-VAE, a lightweight video VAE framework for explicit decoupled modeling of video content. By decomposing motion dynamics and fine-grained residual details with keyframes as appearance anchors, it avoids feature entanglement in raw pixel encoding, enhancing reconstruction quality and representation interpretability. • We integrate dedicated encoders for each decoupled component and a shared 3D decoder to maintain spatiotemporal consistency. Alongside a decoupled adaptation strategy that freezes partial encoder, which eliminates cross-component interference and ensures stable training with precise learning of static and dynamic features. • Leveraging its decoupled design, DeCo-VAE enables superior performance on video reconstruction, whose latent representation serve as an efficient drop-in solution for downstream video generation tasks. 2 Related Work 2.1 Video Diffusion Models Latent Video Diffusion Models (LVDMs) have emerged as the cornerstone of state-of-the-art video generation, powering flagship frameworks such as Sora 8 , OpenSora 53 , Open Sora Plan 27 , VideoCrafter 9 ; 10 , Latte 29 , CogVideoX 46 , DynamiCrafter 44 , Vidu 4 , and Hunyuan Video 21 . Beyond general video synthesis, LVDMs also enable specialized tasks including controllable video generation 17 and multimodal video generation 16 . The LVDMs pipeline follows a two-stage paradigm: first, a video Variational Autoencoders (VAE) compresses raw video data into a compact latent space, drastically reducing computational costs; second, a noise prediction model operates within this latent domain to learn and perform target transformations. The performance of LVDMs is inherently tied to the quality of the video VAE, as generated video fidelity depends critically on both the representational capacity of the latent space and the VAE’s encoding-decoding efficiency. In image generation, frameworks like the Stable Diffusion series 30 ; 32 ; 47 have achieved remarkable success, largely due to their efficient VAEs that enable high-fidelity reconstruction across diverse image types. By contrast, existing video VAEs have not matched this performance. This gap arises from the unique challenges of compressing video with spatiotemporal correlations while maintaining compactness remains an unresolved hurdle. Consequently, LVDMs are often constrained in scenarios with complex motion, limiting their ability to generate temporally coherent, high-quality videos. 2.2 Video Variational Autoencoder Video Variational Autoencoders (VAEs) are key for latent video compression in Latent Video Diffusion Models (LVDMs), divided into discrete and continuous types. Discrete VAEs like MAGVIT-v2 48 enable high-quality reconstruction but lack backpropagation gradients, making them incompatible with LVDMs. Continuous VAEs (e.g., Stable Video Diffusion 6 ) are widely used in LVDMs, evolving 4\times 8\times 8 compression to reduce temporal redundancy, yet most struggle with large-motion video reconstruction due to weak temporal modeling.