AnyPortal : Zero-Shot Consistent Video Background Replacement
Wenshuo Gao Xicheng Lan Shuai Yang ✉ {{\char 12\relax}} Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing,Peking University, Beijing, China {gaowenshuo, lanxicheng}@stu.pku.edu.cn [email protected] · 2025
Read AnyPortal : Zero-Shot Consistent Video Background Replacement by Wenshuo Gao Xicheng Lan Shuai Yang ✉ {{\char 12\relax}} Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing,Peking University, Beijing, China {gaowenshuo, lanxicheng}@stu.pku.edu.cn [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07472.txt.
Introduction
Wenshuo Gao's 2025 paper, written with 2 co-authors, introduces ANYPORTAL, a training-free way to replace the background of a video. Swapping backgrounds while keeping the foreground steady and the lighting believable is hard for current video generators. Zero-shot means the method works without training on the target task, reusing pre-trained diffusion models, the generative models behind much modern image and video AI.
ANYPORTAL joins the temporal prior of a video diffusion model, its sense of how scenes move, with the relighting skill of an image diffusion model. A Refinement Projection Algorithm manipulates details at pixel level to preserve the foreground exactly. The result is temporally coherent relighting with a consistent subject throughout the clip. Experiments show high-quality output on consumer-grade graphics cards, which makes the tool practical for everyday video editing and content creation.
Catalog record
| CREATOR | Wenshuo Gao Xicheng Lan Shuai Yang ✉ {{\char 12\relax}} Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing,Peking University, Beijing, China {gaowenshuo, lanxicheng}@stu.pku.edu.cn [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AnyPortal : Zero-Shot Consistent Video Background Replacement Wenshuo Gao Xicheng Lan Shuai Yang ✉ {{\char 12\relax}} Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing,Peking University, Beijing, China {gaowenshuo, lanxicheng}@stu.pku.edu.cn [email protected] Abstract Despite the rapid advancements in video generation technology, creating high-quality videos that precisely align with user intentions remains a significant challenge. Existing methods often fail to achieve fine-grained control over video details, limiting their practical applicability. We introduce AnyPortal , a novel zero-shot framework for video background replacement that leverages pre-trained diffusion models. Our framework collaboratively integrates the temporal prior of video diffusion models with the relighting capabilities of image diffusion models in a zero-shot setting. To address the critical challenge of foreground consistency, we propose a Refinement Projection Algorithm, which enables pixel-level detail manipulation to ensure precise foreground preservation. AnyPortal is training-free and overcomes the challenges of achieving foreground consistency and temporally coherent relighting. Experimental results demonstrate that AnyPortal achieves high-quality results on consumer-grade GPUs, offering a practical and efficient solution for video content creation and editing. Figure 1 : We propose AnyPortal , a training-free framework for high-consistency video background replacement and foreground relighting. Given an input foreground video and a text or image prompt of the background, our method produces a video with the target background under harmonious illuminations, while maintaining the foreground video details and intrinsic properties. 1 Introduction Teleportation, often regarded as one of the most popular superpowers, offers the fascinating ability to travel anywhere instantly. While true teleportation remains confined to the realm of science fiction, digital technologies have made its virtual counterpart a reality, particularly in the film and entertainment industry. Through the use of green screens and digital techniques, actors can be seamlessly transported from a studio to virtually any location. However, this process is far from trivial. It involves a complex pipeline that includes constructing green screen environments, generating backgrounds that are geometrically consistent with the camera’s perspective, and meticulously replacing the green screen with the synthesized background while ensuring realistic illuminations. Despite its widespread use in professional settings, this workflow remains resource-intensive and labor-expensive beyond normal users. Recent years have witnessed rapid advancement in AIGC, highlighting the potential to make “virtual teleportation” accessible to the general public. The state-of-the-art image diffusion model, IC-Light [ 41 ] , enables users to replace the background of a photo with harmonized illuminations, achieving robust performance through extensive training on paired image datasets. However, collecting large-scale paired video datasets is considerably more difficult compared to paired images, making scaling this approach to video significant challenges. Meanwhile, recent cutting-edge video diffusion models [ 37 , 44 ] demonstrate impressive capabilities in video generation and editing. Despite their potential, these models still fall short for widespread video background replacement tasks. First, existing video diffusion models exhibit limited controllability over generated content. While some approaches [ 31 , 9 , 28 ] introduce coarse controls for edges, poses and motion, they lack pixel-level precision, often resulting in unintended alterations to the foreground appearance. Second, adapting video models to our specialized task typically requires task-specific training or fine-tuning, which is hindered by the scarcity of paired video data and the substantial computational resources needed to train large video models. We believe that pre-trained large diffusion models inherently possess rich prior knowledge for video background replacement: IC-Light provides valuable insight into how lighting should be rendered, while video models capture real-world dynamics. Our key insight is to explore to which extent these pre-trained models can manage tasks that extend beyond their original training task, collaboratively leveraging their inherent priors in a zero-shot setting. To this end, we investigate the zero-shot video background replacement problem. While IC-Light excels at illumination harmonization, and video models provide powerful temporal priors, naively combining them fails to address the critical challenge of foreground consistency, which requires precise pixel-level control over the generation process. While mature training-free control schemes exist for image models – such as inference-time optimization [ 39 , 38 ] or DDIM inversion [ 30 ] with latent manipulations [ 7 , 17 ] – these methods face significant limitations when applied to video models: 1) Optimization on video models incurs prohibitive computational costs; 2) Video models typically operate in a highly compact 3D latent space [ 37 ] , which degrades inversion quality and hinders detailed manipulations. To address these challenges, we propose a novel Refinement Projection Algorithm (RPA) tailored for video models. RPA computes a projection direction in the latent space that simultaneously ensures high consistency with the input foreground details and high-quality background, offering a robust and efficient solution for zero-shot video background replacement. We introduce AnyPortal , a novel training-free framework for video background replacement. AnyPortal first generates a coarse video with illumination harmonized by IC-Light and then enhances its temporal consistency using a pre-trained video diffusion model. To achieve precise control over foreground details, a Refinement Projection Algorithm is proposed to enable pixel-level manipulation. As shown in Fig. 1 , AnyPortal seamlessly transfers foreground subjects ( e.g . , humans or objects) from an input video to a new environment, specified by either a text prompt or a background image, while ensuring natural illuminations, realizing “virtual teleportation” in videos. Remarkably, our framework operates efficiently on a single 24GB GPU. Furthermore, its modular design allows each component to be implemented using the best available pre-trained models, ensuring compatibility with the latest advancements in AIGC. Our contributions are threefold: • We introduce AnyPortal , an efficient and training-free framework for video background replacement. • We design a modular pipeline that integrates the latest pre-trained image and video diffusion models, to combine their strengths for realistic and coherent video generation. • We propose a novel Refinement Projection Algorithm that enables pixel-level detail manipulation in compact latent spaces, ensuring precise foreground preservation. 2 Related Work Image Diffusion Model. Latent Diffusion Model (LDM) has become a strong method for image generation, notably gaining significant popularity with Stable Diffusion [ 27 ] . The main idea is to first compress the image data into latent space using a Variational Autoencoder (VAE), then progressively denoise Gaussian noises with algorithms such as DDPM [ 13 ] and DDIM [ 30 ] , and finally decode the denoised latent back to an image. Traditionally, U-Net architectures were used, followed by DiT [ 23 ] that introduces Transformers to improve generated results as in SD3 [ 5 ] . To meet user demands for generating images with specific conditions, SDEdit [ 18 ] allows for training-free image editing in the LDM framework by denoising the noisy image from an intermediate timestep. Methods like ControlNet [ 40 ] , T2I-Adapter [ 20 ] and ControlNeXt [ 24 ] create a learnable branch of the denoising model to offer additional control conditions such as edges and depth maps. Another approach to image editing is using DDIM inversion [ 30 ] and Null-Text Inversion [ 19 ] , which inverts the denoising process of a given image, then re-denoises it with text guidance and attention manipulations [ 12 , 22 , 32 , 3 ] . Figure 2 : Framework of AnyPortal . (a) Background Generation: A video diffusion model \epsilon_{\theta} is used to generate