InfinityHuman: Towards Long-Term Audio-Driven Human Animation

Xiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, Zehuan Yuan · 2025

Read InfinityHuman: Towards Long-Term Audio-Driven Human Animation by Xiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, Zehuan Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20210.txt.

Introduction

Xiaodi Li and co-authors build a system that animates a person from sound alone, for long stretches of video. Such tools can dub films or drive virtual hosts, but long clips tend to drift. Faces change, colors shift, and hands warp or fall out of step with the voice. Their 2025 system, InfinityHuman, works in stages from rough to fine.

First it lays down motion that matches the sound. Then a refiner uses body poses and the first frame as anchors to hold the look steady. Poses do not decay over time the way raw frames do, so drift drops. A hand-focused reward, trained on clean hand data, sharpens gestures. Tests on two data sets put the system ahead on quality, lip match, and hand form.

Catalog record

CREATORXiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, Zehuan Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InfinityHuman: Towards Long-Term Audio-Driven Human Animation Xiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, Zehuan Yuan Abstract Audio-driven human animation has attracted wide attention thanks to its practical applications. However, critical challenges remain in generating high-resolution, long-duration videos with consistent appearance and natural hand motions. Existing methods extend videos using overlapping motion frames but suffer from error accumulation, leading to identity drift, color shifts, and scene instability. Additionally, hand movements are poorly modeled, resulting in noticeable distortions and misalignment with the audio. In this work, we propose InfinityHuman, a coarse-to-fine framework that first generates audio-synchronized representations, then progressively refines them into high-resolution, long-duration videos using a pose-guided refiner. Since pose sequences are decoupled from appearance and resist temporal degradation, our pose-guided refiner employs stable poses and the initial frame as a visual anchor to reduce drift and improve lip synchronization. Moreover, to enhance semantic accuracy and gesture realism, we introduce a hand-specific reward mechanism trained with high-quality hand motion data. Experiments on the EMTD and HDTF datasets show that InfinityHuman achieves state-of-the-art performance in video quality, identity preservation, hand accuracy, and lip-sync. Ablation studies further confirm the effectiveness of each module. Code will be made public. Figure 1: InfinityHuman is an audio-driven full-body animation framework that synthesizes long-duration videos with (a) temporally consistent visual appearance, (b) expressive and style-rich hand gestures, (c) dynamic human-object interactions, and (d) emotion-controllable, audio-aligned full-body motions. Figure 2: Progressive Degradation in Long Video Animation by Previous Methods. Existing methods suffer from cumulative errors leading to pronounced identity drift (facial inconsistencies), color shifts (hair, clothing), scene instability (background fluctuations), and hand motion artifacts. These challenges underscore the necessity of InfinityHuman’s pose-guided refiner and hand-specific optimization for producing high-fidelity, temporally coherent animations over extended sequences. 1 Introduction Audio-driven character animation aims to generate realistic human videos from a single image and audio input, transforming static portraits into speaking characters. This technology holds significant potential across various industries, including advertising, vlogging, and film production. With the rapid advancement of video generation models, recent research ( Zhang et al. 2023 ; Wang et al. 2024a ; Lin et al. 2024 ; Hogue et al. 2024 ; Cui et al. 2024b ; Lin et al. 2025 ; Wang et al. 2025 ) has progressed from driving facial and head movements to full-body animation, greatly enhancing the expressiveness and richness of generated content. Despite notable progress in full-body human animation, critical challenges remain in generating high-resolution, long-duration, and naturally coherent videos. These challenges can be grouped into two main areas: i) Long-Term Visual Consistency : Existing methods ( Lin et al. 2025 ; Wang et al. 2025 ; Chen et al. 2025b ; Cui et al. 2024b ; Kong et al. 2025 ; Gan et al. 2025b ) typically extend video sequences using overlapping motion frames. However, as sequence length increases, accumulated errors undermine visual coherence, resulting in progressive degradation. This degradation manifests in three key aspects: inconsistent character identity (e.g., variations in facial proportions or clothing); global color incoherence (e.g., erratic shifts in tone or brightness); and scene instability (e.g., shifting or disappearing background objects). ii) Hand Motion Naturalness : Prior work has predominantly focused on facial naturalness and coarse body movements, neglecting the nuanced handling of hand motions—small-range yet high-speed movements. Consequently, large hand gestures frequently lead to distortions or artifacts, and misalignment between hand movements and audio further diminishes the expressiveness and realism of generated videos. To address the aforementioned limitations, we propose InfinityHuman , a novel coarse-to-fine generation framework. This framework first produces low-resolution motion frames synchronized with audio, and subsequently outputs high-resolution long-form videos via a dedicated Refiner. Our method introduces innovations in two key aspects. First, we design a pose-guided refiner to address visual drift in long-duration sequences. Given that pose sequences are structurally decoupled from visual appearance, they inherently resist temporal degradation in appearance-related features. Consequently, we use them as reliable conditioning signals. In addition, during continuous continuation, we incorporate the initial frame as a visual anchor to further enhance temporal consistency. This combination offers both dynamic guidance for maintaining temporal coherence and a reference for visual fidelity. Furthermore, compared vanilly diffusion-based super-resolution, the pose signal provides strong anatomical structure and preserves fine-grained motion patterns. This enables more accurate lip-syncing while effectively reducing common artifacts such as motion distortions and finger overlap in diffusion-based super-resolution. Secondly, considering that the human visual system is highly sensitive to hand distortions such as incorrect finger count, unnatural joint movements, we adopt a hand-specific reward feedback mechanism and incorporate high-quality hand motion data during training to guide hand generation. The mechanism encourages the model to produce temporally consistent and correct gestures, thereby enhancing character expressiveness and the realism of the video. We evaluate InfinityHuman on the EMTD ( Meng et al. 2024 ) and HDTF ( Zhang et al. 2021 ) datasets, covering long-duration upper-body and talking-head scenarios. Qualitative and quantitative results show it achieves SOTA performance in video quality, id preservation, hand accuracy, and lip-sync. Ablation studies further confirm the effectiveness of our proposed model. Our contributions are summarized as follows: • We propose InfinityHuman, a coarse-to-fine generation framework specifically designed to address the challenges of visual realism and temporal consistency in long-duration audio-driven character animation. • We develop a pose-guided refiner that leverages stable pose sequences and the initial frame as a visual anchor to correct accumulated errors, maintain lip-sync accuracy, and reduce artifacts in extended video sequences. • To improve hand movement realism and expressiveness, we introduce a hand-specific reward feedback mechanism, integrated with high-quality hand motion data. • Comprehensive experiments on EMTD and HDTF datasets demonstrate that InfinityHuman outperforms state-of-the-art methods across multiple metrics. 2 Related work Long Video Generation Existing methods ( Henschel et al. 2024 ; Bao et al. 2024 ; Wang et al. 2023 ; Chen et al. 2024 ) extend video diffusion models to longer durations by modifying objectives or architectures. Autoregressive pipelines and memory modules ( Henschel et al. 2024 ; Bao et al. 2024 ) improve cross-segment consistency but require costly retraining on curated long-video datasets. In contrast, training-free extensions such as Gen-L-Video ( Wang et al. 2023 ) and FreeNoise ( Qiu et al. 2023 ) improve efficiency via sliding-window attention and noise rescheduling. However, they offer limited temporal modeling, often causing temporal drift and less coherent transitions between segments. To balance quality and efficiency, recent works ( Yin et al. 2025 ; Zhang and Agrawala 2025 ; ai et al. 2025 ) fine-tune short-video diffusion models with previous motion frames as conditions for autoregressive continuation. Despite their flexibility, these methods suffer from error accumulation at inference, leading to degraded fidelity and identity shifts. We adopt a similar strategy but address its limitations with a coarse-to-fine two-stage framework. A low-resolution long video is first generated, followed by a pose-guided refiner that corrects artifacts and restores spatial-temporal consistency, yielding high-resolution, identity-consistent long videos. Audio-driven character animation. Recent advancements in audio-driven character animation have significantly improved lip-syncing and facial expression modulation using latent diffusion models. Works such as SadTalker ( Zhang et al. 2023 ) and Hallo ( Xu et al. 2024 ) enhance audio-to-facial synchronization with 3D rendering and diffusion

Read the complete text