InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming Wei · 2025
Read InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing by Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14033.txt.
Introduction
Shaoshu Yang and ten co-authors present a video dubbing system in this two thousand twenty-five paper. Normal dubbing edits only the mouth, leaving expressions and gestures out of sync with new audio. Their approach, sparse-frame dubbing, keeps chosen keyframes from the original video. Those frames anchor the person's identity, signature gestures, and camera path. The rest of the motion is regenerated to match the new speech.
Their generator, InfiniteTalk, streams audio-driven video of unlimited length. The authors first show why plain image-to-video models fail at this task. Conditioning must adapt as the audio unfolds. The system aims for full-body, lip-synced performances that stay true to the source. Film localization and virtual presenters are natural uses.
Catalog record
| CREATOR | Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming Wei Abstract Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing—a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk: a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization. \captionof figure Compared to the traditional paradigm, sparse-frame video dubbing will not only edit mouth regions. It gives the model freedom to generate audio aligned mouth, facial, and body movements while referencing on sparse keyframes to preserve identity, emotional cadence, and iconic gestures. 1 Introduction Video dubbing is an audio-driven video-to-video generation task that combines an original video with new audio to create localized content Li et al. 2024 ; Zhang et al. 2025 ; Bigata et al. 2025 . This process requires editing facial movements, head rotations, and body gestures to synchronize with the dubbed speech’s timing and emotional tone, while preserving the source video’s visual style and camera motion. These capabilities are essential for global media distribution through streaming platforms. Recent advances in audio-driven generative models have significantly improved lip synchronization capabilities for video dubbing Li et al. 2024 . However, these methods predominantly focus on oral region inpainting, resulting in mismatched head rotations and body gestures that undermine viewer immersion. To address this limitation, we introduce sparse-frame video dubbing, a novel paradigm that preserves only reference keyframes while leveraging modern generative foundation models. As shown in InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing , it references only select keyframes to preserve the original video’s emotional cadence, symbolic gestures, and camera trajectories, while liberating facial expressions, head motions, and body dynamics to synchronize organically with dubbed audio. As a long video generation task, it demands robust temporal continuation capabilities. A practical solution is employing audio-conditioned video generators with initial and terminal frame guidance. Unfortunately, naive application of audio-conditioned image-to-video generators produces unsatisfactory dubbing results, as shown in Fig. 1 . These models fundamentally struggle with identity preservation during extended generation and default to strict motion copying on the conditioning frames. This results in stiff facial expressions, lip and head movements that contradicts speech dynamics. Meanwhile, simply applying initial and terminal frame conditioning creates abrupt inter-chunk transitions. To resolve these challenges, we introduce InfiniteTalk, a audio-driven generator for long sequence sparse-frame video dubbing. It has a streaming video generation base that utilize context frames to inject momentum information that creates smooth inter-chunk transitions. To preserve human identity, background, and camera movements of the source video, it controls the output video by referencing keyframes. To achieve the soft reference mechanism in sparse-frame video dubbing, we investigate how and find the control strength is determined by the similarity between the video context and image condition. Based on our investigation, we propose a sampling strategy that balances control strength and motion alignment by fine-grained reference frame positioning, achieving high quality infinite length long sequence video dubbing with full body audio-aligned motion generation. Meanwhile, we explore methods to achieve accurate subtle camera movement preservation. To comprehensively evaluate our method, we conduct quantitative and qualitative experiments on HDTF Zhang et al. 2021 , CelebV-HQ Zhu et al. 2022 , and EMTD Rang Meng 2025 , including the cases for both face and full body animation. The quantitative results show InfiniteTalk achieves the state-of-the-art performance in audio synchronized motion generation and visual quality. Our human evaluation shows InfiniteTalk successfully produces plausible lip, face, and body movements that align with the speech cadence and the emotional expression. We finish an ablation experiment concerning the sampling strategy and the strength of control, showcasing the effectiveness of our algorithm design. We summarize our contributions as follows: (1) We introduce sparse-frame video dubbing, a novel paradigm for human-centric audio-driven video-to-video generation to produce natural facial expressions, head motions, and body dynamics that synchronize organically with dubbed audio. (2) We analyze the reasons why audio-driven image-to-video generators fail to achieve satisfying performance in this task and how the reference frame positioning during training determines the control strength. With these observations, we propose propose InfiniteTalk, a streaming long video generator with soft conditioning training strategy. (3) Extensive experiments show our method achieves the state-of-the-art performance for video dubbing, especially in lip, head, body motion synchronization. 2 Related works 2.1 Video Generation Recent advances in generative learning methods—including autoregressive models Tian et al. 2024a , diffusion models Wang et al. 2023b ; Song et al. 2020 ; Nichol & Dhariwal 2021 , and flow matching Liu et al. 2023 —have revolutionized video generation. Early efforts like the Video Diffusion Model Ho et al. 2022b pioneered pixel-space denoising, while later works (Make-A-Video Singer et al. 2022 , PYoCo Ge et al. 2023 , Imagen Video Ho et al. 2022a ) integrated large language models for text-to-video synthesis. To tackle video dimensionality, research shifted toward latent space learning: VideoGPT Yan et al. 2021 combined VQ-VAE Esser et al. 2021 with transformers, establishing foundational latent modeling. Subsequent innovations employed diffusion models to approximate latent distributions, spawning latent video diffusion frameworks ( He et al. 2022 ; Zhou et al. 2022 ; Xing et al. 2023 ; Blattmann et al. 2023b ; Blattmann et al. 2023a ; Wang et al. 2023a ; Chen et al. 2023 ; Chen et al. 2024 ). Among these, CogVideoX Yang et al. 2024 introduced diffusion transformers and temporal-compressed VAEs, enhancing motion complexity. Leveraging expanded datasets and compute, modern large-scale generators ( Team 2024 ; Weijie Kong & Jie Jiang 2024 ; Wan et al. 2025 ) now achieve unprecedented quality. 2.2 Audio-driven Human Animation Audio-driven human animation aims to generate dynamic videos from a static reference image, producing synchronized facial expressions and body movements based on audio control signals. In recent years, with the success of diffusion models, end-to-end audio-to-video synthesis methods Tian et al. 2024b ; Wei et al. 2024 ; Xu et al. 2024 ; Chen et al. 2025b ; Cui et al. 2024 ; Ji et al. 2024 ; Li et al. 2024 ; Jiang et al. 2024 have demonstrated considerable potential. These approaches eliminate the need for intermediate representations and exhibit superior performance in portrait animation for talking head generation. Another line of research extends talking head generation to talking body generation. Methods in this category Lin et al. ; Tian et al. 2025 ; Lin et al. 2025 ; Meng et al. 2024 ; Gan et al. 2025 ;