Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video
Xiao Li, Qi Chen, Xiulian Peng, Kai Yu, Xie Chen, Yan Lu · 2025
Read Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video by Xiao Li, Qi Chen, Xiulian Peng, Kai Yu, Xie Chen, Yan Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08376.txt.
Introduction
Video is split into what moves and what stays, by a self-supervised method from Xiao Li and 5 co-authors in a 2025 computer vision paper. Earlier disentangling methods leaned on strong assumptions about the data. Their transformer pipeline instead generates flexible implicit features for frame-by-frame motion and whole-clip content at the same time.
A low-bitrate quantisation bottleneck forces the motion code to stay compact and discrete, which promotes clean separation and a meaningful motion vocabulary. The separated codes then condition a diffusion model that reconstructs the video. Editors and animators want exactly this split: swap the motion, keep the scene, or restyle the content while the action plays on unchanged.
Catalog record
| CREATOR | Xiao Li, Qi Chen, Xiulian Peng, Kai Yu, Xie Chen, Yan Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video Xiao Li, Qi Chen, Xiulian Peng, Kai Yu, Xie Chen, Yan Lu Abstract We propose a novel and general framework to disentangle video data into its dynamic motion and static content components. Our proposed method is a self-supervised pipeline with less assumptions and inductive biases than previous works: it utilizes a transformer-based architecture to jointly generate flexible implicit features for frame-wise motion and clip-wise content, and incorporates a low-bitrate vector quantization as an information bottleneck to promote disentanglement and form a meaningful discrete motion space. The bitrate-controlled latent motion and content are used as conditional inputs to a denoising diffusion model to facilitate self-supervised representation learning. We validate our disentangled representation learning framework on real-world talking head videos with motion transfer and auto-regressive motion generation tasks. Furthermore, we also show that our method can generalize to other types of video data, such as pixel sprites of 2D cartoon characters. Our work presents a new perspective on self-supervised learning of disentangled video representations, contributing to the broader field of video analysis and generation. † † footnotetext: \dagger Joint first authors. The major part of the work was done when Qi Chen was an intern at MSRA. † † footnotetext: 🖂 Corresponding authors. 1 Introduction Video data contains rich temporal information and intricate patterns of movement and change, offering a deeper understanding of real-world environments. Following the intrinsic information structure, a compact and robust representation of video sequences is the disentangled representation in terms of the static part (i.e. content) and the dynamic part (i.e. motion). The content part is highly semantic, encapsulating the unchanging aspects of the scene. The motion component captures the time-varying essentials of the input data, embodying the essence of change and movement within the video. Hence, the disentangled representation offers a structured space for accurately modeling, processing, and understanding the motion and the content, enabling the interpretation, the synthesis and the manipulation of the visual world in its first principles. Figure 1 : Overview of our method. (a) BCD learns compact and disentangled features to represent video data in a self-supervised manner: each video clip is represented by a clip-wise content feature and frame-wise motion feature sequences. (b) The learned disentangled video space natively supports editing tasks, e.g., swapping the content features between two videos for motion transfer. (c) The learned representations form a structured motion space, which further enables training of auto-regressive video generation. Despite the clear advantages of the disentangled representation of video data, the disentanglement learning task itself remains challenging. The decomposition task is intrinsically an underdetermined problem, not to mention the extremely high-dimensional nature of video signals which exacerbates the computational demands as well as introduces complexities in the construction of meaningful disentanglement. Existing works either introduce additional assumptions [ 41 , 5 , 45 , 7 , 54 , 44 ] or impose task-specific prior constraints [ 58 , 53 , 39 , 18 , 66 , 24 , 42 ] . These additional assumptions are often overly idealistic and limit the expressiveness of disentanglement features in real scenarios, while the introduced priors constrain the target framework to specific tasks, restricting its applicability to a narrow domain. In this paper, we propose a general and practical framework for disentangling motion and content representations in real-world video data, minimizing assumptions and task-specific inductive biases. The form of our disentangled representation is uniquely flexible - a purely implicit frame-wise motion feature and clip-wise global content latent features for a given video input sequence. Compared with other explicit or hybrid representations, implicit features have less inductive bias, significantly increasing the expressiveness of our representation. To make our disentanglement assumptions more general, we resort to the first principle and introduce an information bottleneck to regulate the information flow of implicit features during training. Inspired by recent neural codec methods [ 35 ] , we achieve this by leveraging a low-bitrate codebook. Our key insight is that low-bitrate constraint acts as a general prior to enforce meaningful disentanglement, as video reconstruction under an information bottleneck shares the same goal as disentangled representations: providing a compact yet informative representation of the data [ 55 ] . We facilitate the self-supervised video reconstruction under low-bitrate with a latent denoising diffusion model. The usage of the latent diffusion model as a generative decoder for representation learning enhances the fidelity of the learned representation under information bottlenecks [ 32 ] . Putting it all together, we propose B itrate C ontrolled D iffusion model, BCD for short, a new pipeline for learning disentangled motion and content representations from video sequences ( Figure 1 ). We demonstrate the expressiveness and the effectiveness of our proposed BCD framework by showing results on motion transfer and auto-regressive motion generation on a real world talking head video dataset [ 2 ] . Furthermore, we demonstrate the generality of our method by showing additional disentanglement results on other video data such as the LPC Sprites [ 50 ] . Our contributions can be summarized as follows: • A novel self-supervised framework for disentangling motion and content in video with minimal assumptions and greater flexibility than prior methods. • A low-bitrate vector quantization approach that serves as an information bottleneck to enforce meaningful disentanglement for real world videos. • A structured and expressive latent video space enabled by a denoising diffusion model, supporting generative tasks such as motion transfer and auto-regressive video generation. Ethics Statement. BCD is a research project that focuses exclusively on the self-supervised disentanglement of motion and content from video data. Our work strictly follows Microsoft’s AI principles. While we present several examples involving human faces, their sole purpose is to illustrate the effectiveness of disentangling real-world video representations in an intuitive and interpretable way. All human-related datasets used in this work are publicly available benchmarks that are widely adopted across academia and industry for evaluating facial motion transfer. Our use of these datasets follows established community practice to ensure comparability and alignment with prior work. Our proposed framework is general-purpose: we did not develop or employ any human-specific technologies for deepfake generation, nor do we intend to generate content for misleading or deceptive purposes. At present, we have no plans to integrate this technology into any product or to make its implementation publicly available. 2 Related Works Visual Representation Learning Recent years have witnessed significant advancements in image representation learning with self-supervised pre-training. The Vision Transformer (ViT) [ 15 ] paired with pretask of contrastive learning [ 26 , 11 , 22 ] or masked image modeling [ 64 , 47 ] encodes image patches into discrete tokens and leveraging Transformer architectures [ 57 ] typically used in NLP for visual tasks. Concurrently, generative techniques like VQ-VAE [ 56 ] , GANs [ 20 ] , VQ-based diffusion [ 23 , 17 ] or auto-regressive models [ 63 , 65 ] have demonstrated remarkable capabilities in image synthesis and editing. Specifically, recent studies [ 32 , 40 ] have demonstrated the potential of diffusion models in learning robust visual representations. The emergence of video diffusion models such as Sora [ 10 ] also significantly advances the progress in video representation. These techniques learned to