SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion

Junxian Ma, Shiwen Wang, Jian Yang, Junyi Hu, Jian Liang, Guosheng Lin, Jingbo Chen, Kai Li, Yu Meng · 2025

Read SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion by Junxian Ma, Shiwen Wang, Jian Yang, Junyi Hu, Jian Liang, Guosheng Lin, Jingbo Chen, Kai Li, Yu Meng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11515.txt.

Introduction

Junxian Ma and co-authors make video faces speak any audio while keeping the speaker's look in this 2025 paper. Lip sync tools match mouth motion to speech, but many need staged training or fixed alignment hints, which stiffen the motion. Their system, SayAnything, is a video diffusion model that works straight from sound.

Three modules share the work: one guards identity, one steers by audio, and one limits edits to the mouth region. The signals are balanced inside the model's hidden space, with no extra labels needed. Tests show crisp lip and teeth motion and natural results, even for cartoon faces the system never trained on. The authors claim a simpler path to talking-head video.

Catalog record

CREATORJunxian Ma, Shiwen Wang, Jian Yang, Junyi Hu, Jian Liang, Guosheng Lin, Jingbo Chen, Kai Li, Yu Meng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11515

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion Junxian Ma, Shiwen Wang, Jian Yang, Junyi Hu, Jian Liang, Guosheng Lin, Jingbo Chen, Kai Li, Yu Meng Abstract Recent advances in diffusion models have led to significant progress in audio-driven lip synchronization. However, existing methods typically rely on constrained audio-visual alignment priors or multi-stage learning of intermediate representations to force lip motion synthesis. This leads to complex training pipelines and limited motion naturalness. In this paper, we present SayAnything, a conditional video diffusion framework that directly synthesizes lip movements from audio input while preserving speaker identity. Specifically, we propose three specialized modules, including an identity preservation module, an audio guidance module, and an editing control module. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation without requiring additional supervision signals or intermediate representations. Extensive experiments demonstrate that SayAnything generates highly realistic videos with improved lip-teeth coherence, enabling unseen characters to say anything while effectively generalizing to animated characters. † † affiliationnotice: Work done while the first author was an intern at RightBrain.AI, Beijing SayAnything performs audio-driven lip synchronization through video editing, demonstrating zero-shot generalization to in-the-wild and various style domains without fine-tuning. Our fusion scheme eliminates the dependency on additional supervision signals like SyncNet for lip synchronization. 1 Introduction Audio-driven lip synchronization aims to generate synchronized lip movements in videos based on input audio while preserving the speaker’s identity and appearance. This task has significant applications in video dubbing, virtual avatars, and live-streaming platforms. In the field of lip synchronization, GAN-based approaches ( Guan et al., 2023 ; Su et al., 2024 ) remain the dominant paradigm. However, these methods face two major limitations. First, they lack stability and diversity in motion generation, resulting in unnatural visual effects. Second, the inherent training instability and mode collapse issues make them difficult to scale to diverse datasets ( Heusel et al., 2017 ) , limiting their practical applications. With the success of diffusion models ( Ho et al., 2020 ; Song et al., 2020 ) in image and video generation ( Blattmann et al., 2023a ; Rombach et al., 2022 ) , audio-driven facial animation has emerged as a promising research direction. Recent work leveraging diffusion models for lip synchronization encounters a fundamental challenge where models favour visual conditions over audio signals during generation. Existing approaches ( Mukhopadhyay et al., 2024 ; Li et al., 2024 ; Shen et al., 2023 ; Zhong et al., 2024 ) force models to learn audio-visual correspondence through additional supervision from Syncnet ( Chung & Zisserman, 2017 ) as lip experts ( Mukhopadhyay et al., 2024 ; Li et al., 2024 ) or intermediate representations like landmarks ( Shen et al., 2023 ; Zhong et al., 2024 ) . While this mitigates the challenge, the strong dependency on visual priors from lip experts and intermediate representations significantly constrains model generalizability and motion dynamics. To address these limitations, we present SayAnything, an end-to-end framework that directly establishes audio-visual correlations in the latent space without requiring additional supervision signals or intermediate representations. Specifically, we design a multi-modal condition fusion scheme, incorporating three specialized modules: 1) For reference images, we design an identity preservation module with an efficient encoder ID-Guider that extracts identity features while controlling their influence on lip motion styles, 2) For videos, we propose an editing control module with an adaptive masking strategy that regulates the lip editing region and eliminates the dependency of lip movements on surrounding pixel motion patterns, and 3) For audio inputs, we develop an audio guidance module that enhances the control of weakly-correlated audio features over motion generation. As shown in SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion , our method demonstrates effective zero-shot generalization to both real-world and animated characters. To the best of our knowledge, this is the first work to leverage Stable Video Diffusion (SVD) ( Blattmann et al., 2023a ) for lip synchronization. Our main contributions can be summarized as follows: • We propose an end-to-end video diffusion framework for audio-driven lip synchronization that eliminates the need for additional supervision signals or intermediate representations. • We design a novel multi-modal condition fusion scheme that effectively balances audio-driven control and portrait preservation, incorporating identity preservation, audio guidance, and editing control modules. • Both extensive experiments and user studies demonstrate that our approach achieves superior performance in terms of generation quality, temporal consistency, and lip motion flexibility compared to existing methods. Furthermore, our framework shows strong generalization capability to in-the-wild videos and animated characters. Figure 1: (a) Overview of SayAnything architecture for lip synchronization. The denoising UNet takes noisy latents as input, concatenated with video latents obtained from masked video through VAE encoding. The reference image is processed by ID-Guider to produce multi-scale ID features, which are injected as residual signals into the denoising UNet. Audio features from Whisper are fused through cross-attention layers in the denoising process. (b) A typical UNet block, consisting of ResNet block, Self Attention, Audio Cross Attention, and Temporal Attention. 2 Related Work 2.1 Video Diffusion Models Diffusion generative models have recently gained prominence as a leading approach for vision generation tasks, with notable contributions from several studies ( Sohl-Dickstein et al., 2015 ; Song & Ermon, 2019 ; Ho et al., 2020 ; Song et al., 2021 ; Ruiz et al., 2024 ) . In the realm of video generation, diffusion models have also seen significant advancements, with various methods being proposed ( Brooks et al., 2024 ; Polyak et al., 2024 ; Zheng et al., 2024 ; Chen et al., 2024a ; Blattmann et al., 2023a ; Ho et al., 2022c ; Ho et al., 2022a ; Blattmann et al., 2023b ; Guo et al., 2023 ; Ma et al., 2024 ; Girdhar et al., 2023 ; Zhou et al., 2023 ) . For instance, the early diffusion-based video generation model VDM was introduced by ( Ho et al., 2022c ) . ImagenVideo ( Ho et al., 2022a ) employs a cascaded diffusion process ( Ho et al., 2022b ) to generate high-resolution videos. MagicVideo ( Zhou et al., 2023 ) leverages the architecture of Latent Diffusion Models ( Rombach et al., 2022 ) for video generation, utilizing a low-dimensional latent embedding space defined by a pre-trained variational auto-encoder (VAE). The text-to-video model SORA ( Brooks et al., 2024 ) is capable of generating high-quality videos up to a minute long based on user prompts. Movie Gen ( Polyak et al., 2024 ) , developed by Meta, offers a suite of foundation models that can produce high-quality, high-resolution videos with synchronized audio in various aspect ratios. In the open-source domain, recent foundational video generation models include OpenSora ( Zheng et al., 2024 ) , VideoCrafter-2 ( Chen et al., 2024a ) , and Stable Video Diffusion ( Blattmann et al., 2023a ) . 2.2 Video Editing As diffusion-based video generation continues to advance, the ability to edit videos using text inputs has become a focal point of research. One notable contribution is Tune-A-Video ( Wu et al., 2023b ) , which introduces the idea of one-shot video editing tuning. This method converts the spatial self-attention layers of a text-to-image (T2I)

Read the complete text