OmniEdit: A Training-free framework for Lip Synchronization and Audio-Visual Editing
Lixiang Lin, Siyuan Jin, Jinshan Zhang · 2026
Read OmniEdit: A Training-free framework for Lip Synchronization and Audio-Visual Editing by Lixiang Lin, Siyuan Jin, Jinshan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09084.txt.
Introduction
Lixiang Lin and colleagues rethink lip sync as editing with audio-led diffusion models. Old methods fine-tune pretrained diffusion models on datasets for audio-visual sync, costing much compute and data. Their SyncEdit swaps FlowEdit's edit sequence for the target sequence for unbiased output estimation. Annealed noise matching brings sampled Gaussian noise to model-guessed noise, giving a smooth, stable editing path without training or paired data.
Catalog record
| CREATOR | Lixiang Lin, Siyuan Jin, Jinshan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34779 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09084
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniEdit: A Training-free framework for Lip Synchronization and Audio-Visual Editing Lixiang Lin, Siyuan Jin, Jinshan Zhang Abstract Lip synchronization and audio–visual editing have emerged as fundamental challenges in multimodal learning, underpinning a wide range of applications, including film production, virtual avatars, and telepresence. Despite recent progress, most existing methods for lip synchronization and audio–visual editing depend on supervised fine-tuning of pre-trained models, leading to considerable computational overhead and data requirements. In this paper, we present OmniEdit, a training-free framework designed for both lip synchronization and audio-visual editing. Our approach reformulates the editing paradigm by substituting the edit sequence in FlowEdit with the target sequence, yielding an unbiased estimation of the desired output. Moreover, by removing stochastic elements from the generation process, we establish a smooth and stable editing trajectory. Extensive experimental results validate the effectiveness and robustness of the proposed framework. Code is available at https://github.com/l1346792580123/OmniEdit . 1 Introduction Lip synchronization [ 33 , 18 , 43 , 31 ] , which aims to align mouth movements with corresponding speech audio, is a fundamental yet critical task in computer vision. By enforcing fine-grained temporal and semantic consistency between facial dynamics and acoustic signals, lip synchronization plays an essential role in applications such as film dubbing and digital human generation. As a result, it has become a foundational component in modern video generation systems, underpinning the realism and perceptual quality of AI-generated visual content. Audio-visual editing [ 21 , 4 , 20 ] , whose goal is to simultaneously manipulate visual content and corresponding audio signals according to textual prompts, can be seen as an advanced extension of lip synchronization. Unlike lip synchronization, which focuses primarily on aligning mouth movements with speech, audio–visual editing requires coherent cross-modal modifications, preserving semantic consistency and temporal alignment across both visual and auditory streams, thereby enabling more flexible and expressive multimodal content generation. With the rapid advancement of diffusion-based generative models [ 12 , 22 , 34 , 32 ] , numerous lip synchronization approaches have been developed by performing supervised fine-tuning on text-to-image or text-to-video models. These methods typically require the collection of large-scale paired audio-visual datasets to learn accurate cross-modal alignment. Moreover, they often rely on explicit mouth-region masking or incorporate specialized training strategies to enforce fine-grained correspondence between speech signals and lip movements, thus increasing both data and computational demands. Audio-visual editing similarly relies on paired audio-visual data, which are scarce and difficult to collect, for supervised fine-tuning. Alternatively, it often depends on computationally intensive optimization-based approaches. These requirements pose significant obstacles to progress in the field and limit the feasibility of deploying such techniques in practical applications. Motivated by the limitations discussed above, we propose OmniEdit, a training free framework specifically designed for lip synchronization and audio-visual editing. Leveraging pre-trained audio-to-video diffusion models and audio-visual foundation models, our approach enables precise lip synchronization and supports rich, flexible audio-visual editing without requiring task-specific fine-tuning or large-scale paired datasets. We first replace the iterative refinement of the edit sequence in FlowEdit [ 17 ] with iterations over the target sequence, obtaining an unbiased estimate of the desired output. This reformulation not only reduces the inherent bias introduced by sequential editing but also enables a more direct alignment with the target distribution. Second, we eliminate the stochastic Gaussian sampling in the generation process and replace it with noise estimated by the pre-trained diffusion model, hence constructing a smoother iterative trajectory that enhances both the stability and quality of the generated results. Our contributions can be summarized as follows: • We present OmniEdit , the first training-free framework for lip synchronization and audio-visual editing, eliminating the need for task-specific fine-tuning, and paving the way for plug-and-play multimodal content creation. • We establish a target iterative sequence to enable an unbiased estimation of the desired output, reducing bias from sequential editing and facilitating more direct alignment with the target distribution. • We construct a deterministic and smooth generation trajectory by replacing stochastic Gaussian sampling with semantically consistent noise estimated from a pre-trained diffusion model to enhance output quality. • OmniEdit achieves state-of-the-art performance without additional training, rivaling or surpassing supervised methods in lip synchronization while supporting flexible cross-modal audio–visual editing. 2 Related Works 2.1 Omni-Modal Generation Audio-driven video generation is commonly categorized into lip synchronization and portrait animation, distinguished primarily by their input settings. Lip synchronization operates on a given video to match target audio, while portrait animation generates video content using only audio and a reference image. Early GAN-based [ 7 ] methods [ 33 , 41 , 2 , 26 ] provided fundamental advances that shaped subsequent developments in lip synchronization. The emergence of diffusion models [ 18 , 43 , 25 , 31 ] has markedly advanced the state of the art in audio-driven lip synchronization. Omnisync [ 31 ] adopts a mask-free training strategy with diffusion transformer models, allowing direct frame-level editing in the absence of explicit masking. Despite achieving promising results, the approach still requires additional training or model fine-tuning. Portrait Animation [ 38 , 28 , 3 , 13 , 1 , 5 , 6 ] differs from lip synchronization in that it directly generates videos by driving a reference image with an audio signal. While this approach requires fewer conditional inputs, it typically offers limited control over background, head pose, and detailed facial expressions. We propose a training-free method that leverages a portrait animation model to achieve lip synchronization, hence unifying the two approaches. Audio-Visual Generation refers to the task of jointly synthesizing synchronized video and audio from a single text prompt. Ovi [ 24 ] and BridgeDiT [ 8 ] enable simultaneous audio–visual generation by integrating existing text-to-video (T2V) and text-to-audio (T2A) architectures, which often leads to high computational overhead and limited cross-modal synergy. LTX-2 [ 9 ] employs a decoupled dual-stream architecture with asymmetric design and bidirectional cross-attention, enabling robust audio–visual alignment with substantially lower computational overhead. 2.2 Training Free Editing Training-free editing has emerged as an important direction in diffusion-based generative modeling, as it enables image manipulation without requiring additional model retraining or task-specific fine-tuning. Optimization-based editing [ 15 , 16 , 30 , 10 ] directly modifies the input image by iteratively optimizing it toward consistency with a given textual prompt. Inversion-based editing [ 11 , 35 , 40 , 39 ] involves recovering a noise map via image-to-noise inversion and subsequently applying the conventional diffusion process guided by the target prompts to achieve the intended modifications. FlowEdit [ 17 ] introduces an inversion-free, optimization-free and model agnostic method for text-based image editing. FlowEdit constructs a direct ODE between the source and target distributions, without passing through the standard Gaussian distribution. FlowAlign [ 14 ] extends FlowEdit by promoting a smoother transition from source to target. FlowDirector [ 19 ] further introduces flow correction and applies the framework to text-to-video editing. 3 Preliminaries 3.1 Flow Matching Flow Matching [ 22 ] is a framework for learning continuous-time generative models by directly regressing the vector field of a probability flow. The transportation between the data distribution X_{0} and the Gaussian distribution X_{1}\sim\mathcal{N}(0,I) , is defined by an ordinary differential equation (ODE) dX_{t}=V(X_{t},t)dt, (1) t\in[0,1] , V is the velocity field. Flow matching trains a neural network