MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice
Hongwei Yi Tian Ye Shitong Shao Xuancheng Yang Jiantong Zhao, Hanzhong Guo Terrance Wang Qingyu Yin Zeke Xie Lei Zhu, Wei Li Michael Lingelbach Daquan Zhou · 2025
Read MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice by Hongwei Yi Tian Ye Shitong Shao Xuancheng Yang Jiantong Zhao, Hanzhong Guo Terrance Wang Qingyu Yin Zeke Xie Lei Zhu, Wei Li Michael Lingelbach Daquan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05978.txt.
Introduction
Talking-head videos made from a photo, a script, and a voice usually stall at short clips and stiff faces. Hongwei Yi and co-authors present MagicInfinite, a diffusion transformer built to run without a length limit. It animates realistic people, full-body figures, and anime characters, even from back-facing poses, and can drive several characters at once, with masks naming who speaks.
Three ideas carry the design. Full three-dimensional attention with a sliding-window denoising scheme gives endless, coherent footage. A two-stage curriculum ties lip motion to audio, expression to text, and identity to a reference image. Region masks with adaptive losses balance global text control against local voice cues. Distillation speeds generation twentyfold, producing ten seconds of video in ten seconds at the smaller size. On the team's new benchmark, the system led in lip sync, identity keeping, and natural motion.
Catalog record
| CREATOR | Hongwei Yi Tian Ye Shitong Shao Xuancheng Yang Jiantong Zhao, Hanzhong Guo Terrance Wang Qingyu Yin Zeke Xie Lei Zhu, Wei Li Michael Lingelbach Daquan Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05978
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice Hongwei Yi Tian Ye Shitong Shao Xuancheng Yang Jiantong Zhao, Hanzhong Guo Terrance Wang Qingyu Yin Zeke Xie Lei Zhu, Wei Li Michael Lingelbach Daquan Zhou Abstract We present MagicInfinite, a novel diffusion Transformer (DiT) framework that overcomes traditional portrait animation limitations, delivering high-fidelity results across diverse character types—realistic humans, full-body figures, and stylized anime characters. It supports varied facial poses, including back-facing views, and animates single or multiple characters with input masks for precise speaker designation in multi-character scenes. Our approach tackles key challenges with three innovations: (1) 3D full-attention mechanisms with a sliding window denoising strategy, enabling infinite video generation with temporal coherence and visual quality across diverse character styles; (2) a two-stage curriculum learning scheme, integrating audio for lip sync, text for expressive dynamics, and reference images for identity preservation, enabling flexible multi-modal control over long sequences; and (3) region-specific masks with adaptive loss functions to balance global textual control and local audio guidance, supporting speaker-specific animations. Efficiency is enhanced via our innovative unified step and cfg distillation techniques, achieving a 20× inference speed boost over the basemodel—generating a 10-second 540x540p video in 10 seconds or 720x720p in 30 seconds on 8 H100 GPUs—without quality loss. Evaluations on our new benchmark demonstrate MagicInfinite’s superiority in audio-lip synchronization, identity preservation, and motion naturalness across diverse scenarios. It is publicly available at https://www.hedra.com/ , with examples at https://magicinfinite.github.io/ . Figure 1: Given a portrait image, our model can generate compelling, realistic, and vivid animation videos with control over text and voice, ensuring temporal coherence and perceptual quality even under significant head pose variations and diverse portrait styles. † † footnotetext: ∗ Equal contribution, † Project lead, ‡ Corresponding author. 1 INTRODUCTION Talking avatars—realistic digital characters created from a reference image using audio or text input—are a transformative technology at the intersection of computer vision and human-computer interaction, driving advancements in digital entertainment, education, and AI communication [ 40 , 60 , 25 , 49 , 1 , 24 ] , fundamentally reshaping how humans interact with digital content. Pioneering works [ 52 , 51 , 66 , 5 , 70 , 41 , 62 , 3 , 45 , 9 , 6 , 31 , 75 , 44 , 29 , 37 , 17 , 59 ] use neural networks and rendering like GANs [ 16 ] , NeRF [ 43 ] , and Gaussian Splatting [ 26 ] to fuse motion and identity features for high-fidelity talking heads. Yet, 3DMM [ 55 ] and FLAME [ 33 ] struggle with motion accuracy, and rendering limits resolution and quality. Recent LDMs [ 50 ] boost video synthesis [ 4 , 19 , 7 , 63 , 61 , 65 , 74 , 47 , 28 ] with better diversity and coherence. We apply T2V diffusion to portrait animation. Studies [ 54 , 58 , 8 , 64 , 23 , 72 ] use LDM priors [ 50 ] to model spatial and temporal dimensions separately, but falter in coherence for large movements, non-frontal faces, or high-resolution cases due to weak frame correspondence [ 63 ] . In this work, we propose MagicInfinite , a novel zero-shot framework that utilizes 3D full-attention to effectively model video along both temporal and spatial dimensions. MagicInfinite demonstrates superior perceptual quality and temporal coherence across various situational motions and actions. We explored the simultaneous use of text and voice for synthesizing portrait animation videos, where the voice primarily provides lip movement dynamics, while the text supplies the speaker’s expressions, actions, and background transformations. However, the integration of both audio and textual prompts as motion signals presents certain challenges: The pre-trained T2V model leverages 3D full-attention mechanisms, attending to all tokens from both the video modality and textual prompts, thereby capturing a robust dependency between the entire video content and the associated text. The text plays a significant role in controlling the video content, as each video token is related to the textual tokens. By contrast, the control of character motion by audio primarily focuses on local video regions, such as lip movements, which are only associated with a small subset of video tokens. This creates a conflict between the driving audio and the textual prompt controls. Therefore, when building upon the traditional form of incorporating audio in prior portrait animation works [ 54 ] , which applies cross-attention [ 57 ] between audio features and vision representations within a spatial latent space directly, without further manipulation, the model tends to overlook the driving audio’s influence on lip movements. This limitation becomes even more apparent when the face occupies a smaller portion of the image. To address this challenge, we design a two-stage curriculum learning scheme to effectively enable audio control, guiding the model to progressively learn both textual and audio control. Specifically, in the first training phase, only the portrait image and textual prompt are introduced, allowing the model to learn text-controlled image-to-video (I2V) generation. In the second phase, both text and audio are incorporated. Here, we utilize a face region mask and an adaptive loss function to guide the conditional cross-attention of audio, specifically targeting local facial movements around the mouth. This training scheme effectively integrates the control provided by textual prompts and driving audio for portrait animation. Owing to the substantial increase in the number of function evaluations (NFEs) caused by multi-step sampling and classifier-free guidance (CFG) [ 22 , 10 ] , diffusion models often suffer from sluggish inference speeds—an issue that has been largely neglected in earlier studies. Previous work [ 18 ] obtained an avatar-based few-step generator by employing latent consistency model (LCM) [ 39 ] , which effectively condensed multiple sampling steps into a reduced number, thereby minimizing the sampling process. However, we found that utilizing LCM for MagicInfinite step distillation inevitably led to noticeable degradation in visual quality, often manifesting as significant blurring. To address this pressing concern, we turn to DMD2 [ 67 ] , which has been rigorously validated across numerous image diffusion models [ 46 ] and has consistently demonstrated superior performance compared to LCM, to enable accelerated sampling. Our approach achieves a groundbreaking advancement by simultaneously eliminating MagicInfinite ’s reliance on CFG [ 42 ] during the step distillation process in DMD2, directly reducing NFEs from 50 to just 4. To circumvent the GPU memory constraints posed by importing three models simultaneously in vanilla DMD2, we adopt LoRA to update the parameters of the fake data distribution estimator, thereby ensuring an efficient and resource-aware training process. We conducted a thorough evaluation of MagicInfinite using a rigorous benchmark we developed, called the MagicInfinite -Benchmark. This benchmark consists of 25 driving audio clips spanning diverse speaking scenarios—such as singing, speeches, and rapping—paired with 20 textual prompts describing the speaker’s emotions, actions, and background variations. Additionally, the MagicInfinite -Benchmark includes a varied collection of portrait images featuring different styles, face orientations relative to the camera, and face sizes. Under this benchmark, MagicInfinite exhibits robust performance across a wide range of portrait images and motion scenarios, effectively driven by both audio and textual prompts, while maintaining efficient inference speeds. The key contributions of our work are as