SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wild
Xindi Zhang, Dechao Meng, Steven Xiao, Qi Wang, Peng Zhang, Bang Zhang · 2025
Read SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wild by Xindi Zhang, Dechao Meng, Steven Xiao, Qi Wang, Peng Zhang, Bang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21736.txt.
Introduction
Xindi Zhang, Bang Zhang, and their co-authors present SyncAnyone, a system for lip-syncing video to audio in uncontrolled footage. Dubbing demands accurate lip motion, sharp image quality, and preservation of identity and background. The standard training trick, masking the mouth and asking the model to repaint it, breaks the surrounding context and causes flicker and drift.
SyncAnyone works in two stages. First, a diffusion-based video transformer learns masked mouth inpainting, using its space-time modeling to generate audio-driven lip movements. Some artifacts remain around the face and background because the input was corrupted. In the second stage, the team builds synthetic training pairs by lip-syncing source videos to sampled audio, then fine-tunes the model without a mask. This self-correction pass teaches precise lip editing with consistent backgrounds. Experiments show state-of-the-art quality, temporal coherence, and identity preservation.
Catalog record
| CREATOR | Xindi Zhang, Dechao Meng, Steven Xiao, Qi Wang, Peng Zhang, Bang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wild Xindi Zhang, Dechao Meng, Steven Xiao, Qi Wang, Peng Zhang, Bang Zhang Abstract High-quality AI-powered video dubbing demands precise audio-lip synchronization, high-fidelity visual generation, and faithful preservation of identity and background. Most existing methods rely on a mask-based training strategy, where the mouth region is masked in talking-head videos, and the model learns to synthesize lip movements from corrupted inputs and target audios. While this facilitates lip-sync accuracy, it disrupts spatiotemporal context, impairing performance on dynamic facial motions and causing instability in facial structure and background consistency. To overcome this limitation, we propose SyncAnyone, a novel two-stage learning framework that achieves accurate motion modeling and high visual fidelity simultaneously. In Stage 1, we train a diffusion-based video transformer for masked mouth inpainting, leveraging its strong spatiotemporal modeling to generate accurate, audio-driven lip movements. However, due to input corruption, minor artifacts may arise in the surrounding facial regions and the background. In Stage 2, we develop a mask-free tuning pipeline to address mask-induced artifacts. Specifically, on the basis of the Stage 1 model, we develop a data generation pipeline that creates pseudo-paired training samples by synthesizing lip-synced videos from the source video and random sampled audio. We further tune the stage 2 model on this synthetic data, achieving precise lip editing and better background consistency. Extensive experiments show that our method achieves state-of-the-art results in visual quality, temporal coherence, and identity preservation under in-the wild lip-syncing scenarios. Project page: https://humanaigc.github.io/sync_anyone_demo_page/ . 1 Introduction Figure 1 : Given any video and audio input, SyncAnyone can modify the mouth movements of characters in the video to synchronize with the audio. Additionally, SyncAnyone is capable of handling challenging scenarios such as large poses, background changes, occlusions, scene cuts, and diverse styles. Audio-driven lip synchronization, or lip-syncing, is the task of algorithmically modifying a speaker’s mouth in a video to achieve precise synchronization with a target audio signal. A key requirement of this task is to preserve the subject’s identity, non-verbal facial expressions, and the integrity of the background scene. This technology has significant application potential in domains such as post-production video dubbing, virtual avatar animation, online education, and cross-lingual content localization. An ideal lip-syncing system is expected to not only synthesize photorealistic and temporally coherent mouth movements but also exhibit robustness against common in-the-wild challenges (Fig. 1 ), including extreme head poses, partial occlusions, scene cuts, and complex background dynamics. Recently, diffusion transformers (DiTs) have emerged as a promising approach to lip-syncing. By leveraging strong spatiotemporal coherence priors from large-scale video data, mainstream methods adopt a mask inpainting paradigm: they sample video sequences with the mouth region masked and train models to reconstruct the lips conditioned on the target audio, enabling accurate audio-driven editing. Furthermore, by feeding sequential speaker frames as context, these methods exploit the in-context learning capability of DiT-based models to generate personalized, identity-consistent talking videos. Despite their effectiveness, mask-inpainting-based methods 33 ; 11 ; 21 are highly sensitive to mask design. When assigned with small mask regions, the model tends to exploit shortcut solutions by inferring lip movements from contextual cues near the mask boundary, such as chin contours and facial dynamics, thereby bypassing audio conditioning. With large mask regions, critical background and identity-related content are masked out, leading to inaccurate background reconstruction, compromised identity consistency, and failure in challenging cases such as cuts or fast motion. To mitigate the mask design trade-off, a feasible solution 20 is to adopt mask-free training on collected paired data 28 with consistent identity, pose, and background but varying lip movements. This help the model to learn both lip editing and visual consistency. However, such data requires highly controlled recording conditions and is difficult to scale, limiting practicality and generalization. In this work, we propose Progressive Self-Correction (PSC), a mask-free lip-syncing framework comprising two stages. In the first stage, we train a multi-reference, mask-inpainting-based audio-to-lip model and distill it into a few-step diffusion model for efficient generation of paired data (with same identity and background but different lip movements). In the second stage, we use the Stage 1 model to generate synthetic pairs on-the-fly and train a mask-free lip-editing model from these pseudo-paired samples. Since the pseudo videos generated by the Stage 1 model still suffer from artifacts near facial boundaries and background distortion due to masking, we introduce a fusion module to correct the training data by replacing the generated background with that of the original video. This ensures the model learns to edit lips accurately while preserving background fidelity and identity consistency. Our main contributions are as follows: • We identify the mask-induced trade-off in lip-syncing and propose Progressive Self-Correction, enabling a transition from mask-based to mask-free editing via self-generated data. • We introduce an efficient online pipeline for generating pseudo-paired data with consistent background and different lip movements. • Building on the above, we introduce SyncAnyone, a novel lip-syncing framework that robustly handles diverse in-the-wild scenarios and achieves state-of-the-art performance in terms of generation quality, temporal consistency, and motion naturalness. 2 Related Works Audio-driven lip synchronization aims to edit an existing video to align the lip movements of a subject with a target audio track, while preserving all other content, such as head pose, identity, and background. This task is framed as a video-to-video editing problem. Following the evolution of their core technical approaches, we classify existing methods into two main categories: GAN-based methods, and emerging diffusion-based methods. Figure 2 : The overall framework of SyncAnyone. The left panel illustrates our two-stage Progressive Self-Correction (PSC) training paradigm for modifying a source video’s lip movements to match a target audio. In Stage 1, a multi-reference mask inpainting model is trained for robust synthesis. In Stage 2, this model is leveraged to create a pseudo-paired dataset, which in turn supervises the training of our final, efficient mask-free model. The right panel details the specific network architecture employed in our framework. 2.1 GAN-based Methods Early and mainstream lip synchronization methods predominantly rely on Generative Adversarial Networks (GANs) 5 ; 3 ; 8 . Wav2Lip 21 stands as a landmark work in this field. It pioneered the use of a pre-trained lip-sync discriminator, SyncNet 2 , to supervise the generator’s training, which significantly enhanced the accuracy of the generated lip movements and established a benchmark for subsequent research. Building on this foundation, subsequent works have introduced improvements from various perspectives. Some methods focus on enhancing generation quality and identity preservation. For instance, DINet 35 deforms the feature maps of a reference image to generate more natural mouth shapes, thereby better preserving high-frequency details. VideoReTalking 1 decomposes the task into three stages: semantic-guided reenactment, lip synchronization, and identity-aware refinement. StyleSync 7 employs StyleGAN2 10 as its generator backbone, leveraging its powerful generative capabilities to improve visual quality. Most of these methods are arbitrary-subject models, designed to build a general model that does not require retraining for different identities. Another line of work introduces intermediate representations to guide the generation process. For example, some works first predict facial landmarks 29 ; 16 or 3D-reconstruction-based landmarks 22 from audio before performing image-to-image translation. IP-LAP 37 adopts such a two-stage approach. Some methods 6 ; 33 further incorporates 3D