SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

Zhengcong Fei, Di Qiu · 2025

Read SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers by Zhengcong Fei, Di Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00830.txt.

Introduction

Zhengcong Fei and a team of co-authors teach a video model to make portraits talk, in a 2025 paper. The catalog names only two authors; the printed byline lists eleven. Talking-head generation from audio, text, and images in one system is still underexplored.

Built on a pretrained video diffusion transformer, SkyReels-Audio can generate or edit portraits of unlimited length. Training aligns voice with face motion in stages. A face-mask loss and audio-guided steering sharpen local coherence. Sliding-window denoising stitches segments smoothly. The team also built a pipeline for clean audio-video-text training triplets. Benchmarks gave it top marks for lip sync, identity consistency, and natural motion, even in hard cases.

Catalog record

CREATORZhengcong Fei, Di Qiu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00830

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers Zhengcong Fei, Di Qiu Abstract The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio , a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions. The model, along with demonstration videos, will soon be made publicly available at the project website: https://www.skyreels.ai . Figure 1: Given a portrait image, text, or video along with audio input, SkyReels-Audio can generate and edit portraits with strong identity consistency, expressive facial and natural body dynamics. In addition, SkyReels-Audio support infinite video generation based on various multi-modal controllable clues. 1 Introduction Recent developments in computer vision and human-computer interaction have highlighted the transformative potential of generating realistic digital avatars responsive to audio stimuli. These avatars, informed by static imagery, video sequences, or textual metadata, are poised to impact a wide range of domains, including digital storytelling, immersive education, and virtual communication platforms [ 43 , 66 , 29 , 54 , 1 , 28 , 26 , 42 ] . A foundational challenge in this domain lies in achieving precise, audio-conditioned modulation of avatar behavior, ensuring seamless integration with multimodal contextual signals. Robust synchronization across modalities is critical to enhancing user immersion and shaping future paradigms of interactive digital media consumption. The field of talking head synthesis has witnessed significant progress through the application of advanced neural rendering techniques, such as GANs [ 21 ] , NeRF [ 46 ] , and Gaussian Splatting methods [ 30 ] . These approaches [ 57 , 56 , 76 , 4 , 78 , 44 , 71 , 2 , 49 , 10 , 5 , 36 , 83 , 48 , 34 , 41 , 22 , 64 ] have enabled the effective fusion of facial motion cues with static identity characteristics, producing highly realistic visual outputs. However, traditional parametric models [ 60 , 37 ] often fall short in capturing the full range of expressive motion, and conventional rendering methods continue to impose constraints on spatial resolution and output quality. The advent of diffusion-based generative models [ 55 ] has markedly improved the realism, diversity, and controllability of video synthesis [ 3 , 23 , 72 , 67 , 75 , 82 , 50 , 32 , 62 , 6 , 20 , 19 ] . Notably, recent approaches [ 59 , 63 , 8 , 73 , 27 , 80 , 52 ] have demonstrated that diffusion priors are well-suited for disentangling spatial appearance from temporal dynamics, resulting in improved motion continuity. Despite these advances, unified frameworks that integrate multimodal controls, particularly audio and text for talking head generation, remain underexplored. In particular, a comprehensive audio-conditioned diffusion model for robust long-range generation and editing across diverse portrait domains has yet to be fully investigated. This paper introduces SkyReels-Audio, a omni audio-conditioned framework designed to generate and edit temporally consistent and visually realistic talking portrait videos using pretrained video diffusion transformers. The proposed system achieves high perceptual fidelity across a broad range of facial expressions and motion patterns, maintaining coherence even under diverse and dynamic conditions. SkyReels-Audio facilitates portrait animation and editing by concurrently leveraging speech signals alongside text, images, and video inputs. In this multimodal configuration, the audio primarily governs articulatory movements, while auxiliary modalities influence expressive behaviors, physical gestures, and environmental transformations. We use Whisper [ 53 ] to encode audio information, which features are further aligned with video representations with audio 1D RoPE. These audio tokens are integrated into the denosing network via a cross-attention mechanism that allows effective fusion with video tokens. To ensure robust audio control and multimodal integration, we employ a hybrid curriculum training strategy that incrementally conditions the model using joint optimization objectives. During inference, a bidirectional latent fusion algorithm is employed, enabling to produce indefinitely long video sequences while preserving temporal continuity and rendering quality across various speaker identities and settings. Additionally, we perform a comprehensive evaluation using both a custom-designed benchmark and established public datasets. Our benchmark comprises over 50 audio-driven scenarios encompassing multiple languages and variable speaking styles and includes both quantitative metrics and human subjective assessments to capture perceptual and technical performance. Under these settings, SkyReels-Audio demonstrates strong generalization capabilities across a diverse set of portrait images and video samples. The model effectively responds to audio-driven cues and textual instructions while preserving high visual quality and natural motion. Furthermore, it maintains consistent performance at varying sequence lengths, enabling efficient and scalable inference for real-world applications. In summary, out contributions are listed as follows: • We present SkyReels-Audio, a unified framework for audio-conditioned talking portrait generation and editing based on pretrained Video Diffusion Transformers. It employs a bidirectional latent fusion strategy to generate temporally coherent and visually consistent long-form videos across diverse speaking styles and contexts. • We integrate a hybrid learning paradigm that combines image and video-based multimodal controls to improve model capacity and generalization. It introduces a facial region-weighted loss to balance local audio-driven articulation with global conditioning, enabling precise synthesis. • We construct our approach on a carefully curated dataset of high-quality, temporally aligned audio-video-text triplets, supported by a comprehensive data pipeline, which facilitates effective multimodal learning. • Experiments demonstrate that SkyReels-Audio achieves robust performance on diverse portrait inputs ranging from static images to dynamic videos under both speech and text conditioning, with efficient inference across variable sequence lengths. 2 Method The goal of this work is to enable realistic portrait animation and editing guided by both speech audio and auxiliary multimodal inputs, including text, static imagery, or video references. The generated output is expected to retain the visual identity of the target subject as defined by the conditioning inputs, while faithfully capturing speech-related articulations such as lip synchronization, facial expressions, and head movements. 2.1 Preliminaries Flow Matching. Flow Matching [ 39 ] provides a principled framework for mapping complex probability distributions to simpler ones by leveraging their probability density functions, thereby enabling sample generation via learned inverse transformations. Several recent works [ 14 , 32 , 6 , 62 ] serve as subclass of Flow Matching models that operate in the latent space, leveraging a pre-trained AutoEncoder [ 31 ] to facilitate this process. In contrast to conventional text-to-video models that rely exclusively on textual conditioning T_{s} , SkyReels-Audio incorporates multimodal conditioning signals, including the driving audio sequence ( A ), a static portrait

Read the complete text