JUST-DUB-IT : Video Dubbing via Joint Audio-Visual Diffusion Journal: TOG
Anthony Chen, Naomi Ken Korem, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or · 2026
Read JUST-DUB-IT : Video Dubbing via Joint Audio-Visual Diffusion Journal: TOG by Anthony Chen, Naomi Ken Korem, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22143.txt.
Introduction
Anthony Chen and co-authors dub videos with a joint audio-visual model. Audio-visual foundation models, pretrained to generate sound and pictures together, have shown a striking ability to model generation and editing across both senses. Video dubbing, replacing speech in another language, stands to gain from those priors.
Most dubbing systems still rely on complex, task-specific pipelines stitched together by hand. The team's 2026 preprint, JUST-DUB-IT, performs dubbing through joint audio-visual diffusion, generating the new voice and matching visuals together. Good dubbing must sync lips, timing, and tone at once. A unified model could make foreign films and videos feel native to far more viewers.
Catalog record
| CREATOR | Anthony Chen, Naomi Ken Korem, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22143
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JUST-DUB-IT : Video Dubbing via Joint Audio-Visual Diffusion Journal: TOG Anthony Chen, Naomi Ken Korem, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or Abstract . Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines. Webpage available at https://justdubit.github.io . 1. Introduction Figure 1 . Video dubbing via joint audio–visual generation. Top : an input video with spoken dialogue in the source language. Bottom : the same video dubbed into a target language, generated by our trained model built on top of an audio–visual foundation backbone. Translated speech and lip motion are produced jointly, while the visual context (such as scene dynamics, face expressions, and body movements), speaker identity, and non-speech events (e.g., pauses, background sounds) are preserved. Recent years have seen significant progress in generative models for images ( Labs 2024 ; Esser et al. 2024 ) , video ( Wan et al. 2025 ; HaCohen et al. 2024 ) , and audio ( Liu et al. 2023 ; Liu et al. 2024 ; Qin et al. 2023 ) , enabling high-quality content synthesis and editing. These advances have transformed how visual and audio content is created. In particular, it has become increasingly easy to generate and edit media using intuitive, high-level controls such as natural language. More recently, audio-video generative models have emerged, allowing sound and visual content to be modeled jointly in a single generative process ( HaCohen et al. 2026 ; Low et al. 2025 ) . Video dubbing aims to translate the spoken content of a video into a different language while preserving the speaker’s identity, including both facial appearance and voice, and maintaining accurate lip synchronization. Unlike audio-only translation or voice conversion, dubbing requires modifying facial motion in a highly localized and precise manner while keeping the rest of the video visually consistent. This makes the task particularly challenging, as even small errors in lip motion, voice characteristics, or identity cues are easily perceived, especially in real-world videos with complex motion, pose changes, and varying visual conditions. Most existing video dubbing methods decompose the problem into a sequence of specialized stages, typically combining audio synthesis with audio-driven facial editing ( Li et al. 2024 ; Zhang et al. 2025a ; Yang et al. 2025 ) . While effective in controlled settings, these modular pipelines are often complex and brittle, with each component making strong assumptions about the input. In practice, such assumptions can break down in unconstrained videos, for example when the speaker moves relative to the camera, produces non-speech sounds such as laughter or sighs, or partially occludes the mouth during speech. Furthermore, most modular systems attempt to isolate the speaker’s voice from the background audio to perform translation in a vacuum. This separation-and-remixing strategy fails to account for the temporal and semantic dependencies between the speaker and their environment. If the translated speech differs in duration from the original—a common occurrence in dubbing—simply re-layering the original background audio leads to a loss of synchrony with environmental events, such as a dog barking in response to a speaker or a door slamming at a specific moment in a conversation. These “messy” auditory-visual misalignments break the viewer’s immersion and highlight the limitations of treating speech as a detached signal. In short, separating audio and visual editing prevents the two modalities from informing each other, which limits performance in complex, real-world videos. To address these challenges, we take a different approach and frame video dubbing as a joint audio-video generation problem within a single model. Instead of relying on multiple specialized components, we adapt a foundational audio-visual diffusion model to perform dubbing directly, allowing audio and visual cues to be generated together and to inform each other in cross-modality attention layers. Crucially, this holistic modeling ensures that the interaction between the speaker and the rest of the scene is preserved. By treating the entire audio-visual stream as a single generative task, our model can naturally adjust the timing and placement of environmental sounds to remain coherent with the newly generated speech and facial motion. This joint formulation naturally captures correlations between speech, facial motion, and scene dynamics, while avoiding many of the assumptions and failure modes of modular pipelines. Importantly, the adaptation is lightweight, requiring only a small LoRA ( Hu et al. 2021 ) fine-tuning on top of a strong generative prior, which makes the approach simple, flexible, and robust in practice. A key challenge in adapting a generative audio-video model for dubbing is the lack of paired training data that preserves both the speaker’s identity and the original visual content while allowing the spoken language to change. To address this, we generate training data using the generative model itself. We first generate multilingual videos in which the same speaker switches languages within a single clip, ensuring consistent facial appearance and voice characteristics across languages. We then split the clip into two halves and each time inpaint the face and audio in one half to match the language of the other half, while keeping the underlying semantic content intact. Using audio-video inpainting framework allows the model to leverage both visual and audio context, resulting in aligned bilingual video pairs that provide effective supervision for training. We evaluate our approach on a diverse set of videos and language pairs, including unconstrained real-world scenarios with complex motion and visual variability. Our method consistently produces high-quality dubbed videos that preserve both facial and voice identity while maintaining accurate lip synchronization and the temporal-semantic coherence of interactions within the scene. Compared to existing dubbing pipelines, our approach is more robust to challenging conditions such as non-frontal views, partial occlusions, and expressive facial behaviors, resulting in improved visual fidelity and overall perceptual quality. These results demonstrate the benefit of leveraging a strong joint audio-video generative prior for dubbing. 2. Related Work Audio-Visual Generative Models The field of multimodal synthesis is shifting from cascaded pipelines to unified Audio-Visual foundation models, using Diffusion Transformers (DiTs) ( Low et al. 2025 ; Zhang et al. 2025c ; Peebles and Xie 2022 ; HaCohen et al. 2026 ) . High-fidelity generation