TransVDM: Motion-Constrained Video Diffusion Model for Transparent Video Synthesis

Menghao Li, Zhenghao Zhang, Junchao Liao, Long Qin, Weizhi Wang · 2025

Read TransVDM: Motion-Constrained Video Diffusion Model for Transparent Video Synthesis by Menghao Li, Zhenghao Zhang, Junchao Liao, Long Qin, Weizhi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19454.txt.

Introduction

Menghao Li and co-authors present TransVDM, the first diffusion model built to generate transparent videos, in a 2025 paper. Transparent video, where each frame carries a see-through channel, is prized for effects and design work, but AI video tools ignored it. The team's model joins a special encoder for the transparency channel with a pretrained video generator.

A new module ties the motion of the see-through regions to the motion of the visible subject, which cuts stray artifacts around edges. To train the system, the authors gathered a quarter million transparent frames. Tests across several benchmarks show the approach produces clean transparent clips. The work opens a fresh direction for video generation tools, aimed at creators who need ready-to-composite footage.

Catalog record

CREATORMenghao Li, Zhenghao Zhang, Junchao Liao, Long Qin, Weizhi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19454

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TransVDM: Motion-Constrained Video Diffusion Model for Transparent Video Synthesis Menghao Li, Zhenghao Zhang, Junchao Liao, Long Qin, Weizhi Wang Abstract Recent developments in Video Diffusion Models (VDMs) have demonstrated remarkable capability to generate high-quality video content. Nonetheless, the potential of VDMs for creating transparent videos remains largely uncharted. In this paper, we introduce TransVDM, the first diffusion-based model specifically designed for transparent video generation. TransVDM integrates a Transparent Variational Autoencoder (TVAE) and a pretrained UNet-based VDM, along with a novel Alpha Motion Constraint Module (AMCM). The TVAE captures the alpha channel transparency of video frames and encodes it into the latent space of the VDMs, facilitating a seamless transition to transparent video diffusion models. To improve the detection of transparent areas, the AMCM integrates motion constraints from the foreground within the VDM, helping to reduce undesirable artifacts. Moreover, we curate a dataset containing 250K transparent frames for training. Experimental results demonstrate the effectiveness of our approach across various benchmarks. I Introduction Transparent video generation involves not only creating visually compelling content but also accurately representing the alpha channels for each pixel, which indicates the degree of transparency. This technology has broad applications across various fields, including film production and augmented reality (AR), where transparent elements are crucial for content composition and creation. However, existing video generation methods, such as Video Diffusion Models (VDMs) [ 10 , 6 , 4 , 17 ] , are limited to supporting only RGB channels and cannot generate transparent videos in an end-to-end manner. In this paper, we present TransVDM, the first approach specifically designed to tackle the challenge of transparent video generation using VDMs. Unlike conventional post-processing techniques that depend on video segmentation, matting or color palette modifications, TransVDM emphasizes the direct generation of transparent videos from the outset. Inspired by LayerDiffuse [ 23 ] , we initially utilized a Transparent Variational Autoencoder (TVAE) to separately encode alpha channel information as a minor perturbation within the latent space of the pretrained VDM. This process enables the VDM to incorporate alpha information within its input image, providing it the basic ability to generate transparent videos directly. While this approach successfully facilitates transparent video generation within the existing VDM framework, it also introduces challenges, specifically that VDMs are prone to generating artifacts in transparent areas. We speculate this arises because any region in the image can be affected by the VDM, leading to pixel value changes. Consequently, this sometimes results in undesirable motion within the RGB channels of transparent areas while the corresponding alpha values transform into non-transparent values, which ultimately forms artifacts. To address this issue, we propose a lightweight Alpha Motion Constraint Module (AMCM), which incorporates the coordinates of foreground bounding boxes as motion constraints into the VDM. This integration serves to explicitly reduce the model’s tendency to generate undesirable motion in transparent regions, significantly minimizing artifacts and enhancing the overall quality of the generated transparent videos. To enable transparent video generation, we have developed a dataset consisting of 250k transparent frames for training purposes, which includes 100k high-quality transparent images and 150k transparent video frames. Our training process follows a two-stage approach: initially, we train the TVAE using the transparent images, and subsequently, we utilize the transparent videos to train the AMCM module. Our contributions are summarized as follows: (1) We introduce TransVDM, the first diffusion model designed for transparent video generation. (2) We utilize a Transparent Variational Autoencoder to enhance the existing VDM framework and introduce a motion constraint to reduce artifacts in transparent regions. (3) We have curated a dataset consisting of 250k transparent frames for training, which can serve as a robust baseline for future research endeavors. II Related Work The goal of current video generation is primarily to create RGB videos from a given text prompt or a combination of text and image. Early works, such as those using GANs for video generation [ 14 , 16 , 18 , 9 ] produce low-quality results. Subsequently, diffusion model based methods become popular, such as LVDM [ 4 ] and ModelScopeT2V [ 17 ] , which transform the 2D U-Net architecture [ 13 ] into a 3D framework for joint training on video content. Additionally, Imagen Video [ 5 ] synthesizes high-resolution videos from text prompts by leveraging multiple video diffusion models, while Make-A-Video [ 15 ] integrates text-to-image generation with a decoupled spacetime attention mechanism. Furthermore, methods aimed at achieving fine-grained control have also gained traction, with approaches such as DragNUWA [ 21 ] and Animate-Anything [ 3 ] , which utilize motion trajectories or masks to enhance image animation. However, these methods lack support for transparency, limiting their effectiveness in certain scenarios. We address this by building the TransVDM framework, which incorporates transparency handling to improve the versatility of video generation. Fig. 1: TransVDM Framework. The TVAE is trained during Stage 1, which is utilized in Stage 2 to produce adjusted noisy latents with alpha information. In Stage 1, the RGB channels receive a smoothing operation to mitigate abrupt visual artifacts that may occur when alpha edges are inaccurately predicted, allowing the transparent areas of the original image to blend more seamlessly. The Stage 2 diagram illustrates a U-Net based VDM, the concatenation of the conditioned frame and adjusted noisy latents, along with a CLIP text encoder and a simplified processing flow of the AMCM module. The AMCM module actually comprises fully connected layers and a temporal attention layer. The input features from the temporal transformer are combined with B , where B represents the normalized coordinates of the bounding boxes for each input frame, and processed through the AMCM module before being passed back to the temporal transformer. III Method Starting with a transparent image I_{t}\in\mathbb{R}^{H\times W\times 4} , we denote the first 3 RGB channels as I_{c}\in\mathbb{R}^{H\times W\times 3} and the alpha channel as I_{\alpha}\in\mathbb{R}^{H\times W\times 1} . We first integrate the alpha channel I_{\alpha} into the RGB latent space by leveraging a Transparent Variational Autoencoder (TVAE). Subsequently, we combine a pretrained image-text-to-video model which is based on AnimateAnything [ 3 ] , with our proposed Alpha Motion Constraint Module (AMCM) to finalize the TransVDM framework, as shown in Fig. 1 . In section A, we provide a brief preliminary of video diffusion models, followed by an overview of TVAE in section B, a detailed explanation of AMCM in section C, and a description of our dataset in section D. III-A Preliminary In this section, we introduce the preliminary concepts of latent video diffusion model (LVDM) [ 4 ] . Given an image x_{0}=I_{c}\in\mathbb{R}^{H\times W\times 3} , the process begins by utilizing a pretrained vanilla Variational Autoencoder (VAE) to encode the raw image x_{0} into the latent space z_{0}\in\mathbb{R}^{c\times h\times w} , which could be reconstructed by the VAE decoder. The diffusion model is optimized to predict the added noise to z_{0} via the following objective function: \mathcal{L}_{\epsilon}=\left\|\epsilon-\epsilon_{\theta}(z_{t},t,c)\right\|_{2}^{2}, (1) where c represents the image and text prompt, \epsilon_{\theta} is the noise prediction function and z_{t} is obtained by t-step addition of Gaussian noise to z_{0} : z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I), (2) where t\in[0,T] is the time step, \bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{i}) and \beta_{t} is a coefficient with respect to t. III-B Transparent Variational Autoencoder As shown in Fig. 1 stage 1, our

Read the complete text