MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation
Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young Yun · 2025
Read MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation by Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young Yun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11026.txt.
Introduction
Sungwoo Cho and colleagues published a paper in 2025 on translating speech and video between languages. Such systems convert a speaker's voice and face together. Keeping the same speaker identity after translation is the hard part. Tone and emotion must survive along with the words.
Their system, MAVFlow, uses a method called conditional flow matching. It renders new audio and video without prior examples of the target speaker. Speaker fingerprints from the voice guide the sound generation. Emotion cues pass along to the face rendering step. Neither guide depends on the actual words spoken. Tests showed better speech quality and better lip and face results. The gains held across different languages.
Catalog record
| CREATOR | Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young Yun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11026
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young Yun Abstract Despite recent advances in text-to-speech (TTS) models, audio-visual to audio-visual (AV2AV) translation still faces a critical challenge: maintaining speaker consistency between the original and translated vocal and facial features. To address this issue, we propose a conditional flow matching (CFM) zero-shot audio-visual renderer that utilizes strong dual guidance from both audio and visual modalities. By leveraging multi-modal guidance with CFM, our model robustly preserves speaker-specific characteristics and significantly enhances zero-shot AV2AV translation abilities. For the audio modality, we enhance the CFM process by integrating robust speaker embeddings with x-vectors, which serve to bolster speaker consistency. Additionally, we convey emotional nuances to the face rendering module. The guidance provided by both audio and visual cues remains independent of semantic or linguistic content, allowing our renderer to effectively handle zero-shot translation tasks for monolingual speakers in different languages. We empirically demonstrate that the inclusion of high-quality mel-spectrograms conditioned on facial information not only enhances the quality of the synthesized speech but also positively influences facial generation, leading to overall performance improvements. 2 2 footnotetext: Corresponding author 1 Introduction With the rapid proliferation of multimedia content and increasing cross-cultural interactions, the expansion from one language to another has become essential to enrich user engagement and comprehension. Traditional approaches in language translation, such as subtitle processing via neural machine translation (NMT) [ 41 ] or single-modality methods like speech-to-speech translation and dubbing [ 19 ] , often fail to deliver a fully immersive experience. For instance, in dubbed films, discrepancies between the original visual content and the dubbed audio can lead to unnatural lip synchronization and a mismatch between the expected and dubbed voices. Such inconsistencies disrupt the viewer’s concentration and diminish the overall experience. The adverse effects of audio-visual incongruence on user perception have been substantiated by the McGurk effect [ 42 ] ; notably, when the dubbed voice deviates from the expected voice of original actors, the naturalness of the content significantly deteriorates [ 31 ] . Figure 1 : Overview of the existing audio-visual translation (AV2AV) framework. Conventional AV2AV methods primarily focus on linguistic content, often neglecting crucial paralinguistic features, such as speaker identity and emotional nuance, which are essential for maintaining consistent speaker characteristics. At a fundamental level, the transition from single-modality to dual-modality translation is achievable via cascaded approaches. A typical pipeline involves using an automatic speech recognition (ASR) model [ 3 , 51 ] to transcribe the source audio into text, subsequently applying NMT [ 18 , 15 ] for language conversion, and finally synthesizing speech via text-to-speech (TTS) systems [ 7 , 63 , 8 ] in conjunction with talking face generation (TFG) models [ 67 , 50 , 48 ] . However, such cascaded methods are complex and often suffer from significant information loss due to repeated modality transformations and intermediate text representations. To overcome these challenges, direct audio-visual-to-audio-visual (AV2AV) translation approaches have been introduced [ 24 , 13 ] . These methods bypass textual representations by leveraging discrete units obtained from self-supervised multimodal models ( e.g . , multilingual AV-HuBERT [ 54 , 13 ] ), enabling a more direct and efficient translation of audio-visual source inputs. Despite these advances, the AV2AV translation still faces a critical challenge: preserving speaker consistency ( e.g . , tone, pitch, or facial expressions) between the original and translated audio-visual data(as shown in Figure 1 ). This limitation is exacerbated by the absence of datasets wherein the same speaker articulates identical content in multiple languages, necessitating zero-shot strategies for speaker preservation. Current AV2AV approaches [ 13 , 12 ] adopt a simple architecture that relies on using speaker embeddings, combining speaker-specific d-vectors [ 60 ] within the speech vocoder [ 29 , 50 ] . This yet underexplored structure of AV2AV, which does not employ advanced conditional generation techniques, still restricts its ability to maintain speaker consistency. Moreover, generating audio and visual components is separated, using a single-modality embedding in each part. This may have limitations from a multimodal perspective since only the reference audio is used, while visual cues could also be considered for speech generation. In this study, to preserve paralinguistic elements of speakers during the linguistic translation , we propose MAVFlow, a conditional flow matching (CFM, [ 43 ] ) based zero-shot audio-visual renderer that leverages dual guidance from audio and visual modalities. Notably, for ideal multilingual translation scenarios, a speaker’s voice characteristics and facial information ( e.g . , appearance or emotion) must remain consistent regardless of language [ 65 , 5 , 1 ] . Based on this hypothesis, we adopt a guidance strategy that utilizes speaker embeddings from audio and emotion embeddings from visual input. This strategy enables the complementary capture of paralinguistic information, which are commonly shared across both audio and visual modalities. Furthermore, MAVFlow leverages the optimal transport (OT) CFM’s structural advantages in integrating a multimodal guidance. OT-CFM [ 43 ] facilitates learning a conditional speech distribution, enhancing zero-shot performances and guidance-based control, making it an ideal approach for preserving paralinguistics in AV2AV translation. MAVFlow incorporates x-vector-based speaker embeddings [ 55 ] of audio and facial emotion embeddings [ 58 ] of visual inputs, directly guiding the flow matching generative module. Additionally, OT-CFM significantly improves speech synthesis performance and enables more efficient sampling with fewer steps. Thus, MAVFlow achieves enhanced speaker consistency while producing seamless audio-visual translation results in cross-lingual scenarios. Our contributions are summarized as follows: \circ We propose MAVFlow, integrating discrete speech units with OT-CFM to efficiently synthesize high-quality mel-spectrograms for advanced audio-visual translation. \circ We transmit paralinguistic speaker characteristics from both audio and visual modalities within the latent space of the OT-CFM model, thereby achieving robust zero-shot capabilities in cross-lingual scenarios. \circ We empirically demonstrate that our dual guidance improves the consistency of synthesized speech by an average of 36% on the MuAViC dataset [ 3 ] , while enhancing face generation with gains in lip-sync accuracy (+0.87) and visual quality score (-0.61) on direct system. \circ We also confirm that MAVFlow effectively represents emotion in both audio and visual generation on the CREMA-D dataset [ 6 ] . 2 Related Works 2.1 Spoken Language Translation Spoken Language Translation (SLT) aims to convert spoken language in one language into another language, promoting natural cross-lingual interaction. Traditional SLT typically adopt cascaded approaches [ 46 , 40 ] for speech-to-speech translation (S2ST), chaining ASR, NMT, and TTS. Although widely used, cascaded methods suffer from cumulative errors, latency, and loss of speaker-specific prosodic and paralinguistic features [ 64 , 26 ] . To reduce these issues, research has shifted toward end-to-end SLT methods that translate speech directly [ 27 , 45 ] , and even textless approaches that eliminate the reliance on textual representations [ 34 , 28 ] . Despite advances, S2ST systems primarily focus on audio signals, often neglecting the alignment between translated speech and visual information, which is crucial for cross-lingual scenarios such as video conferencing or dubbing. This can lead to lip-sync mismatches [ 42 , 31 ] , disrupting realistic multimodal experiences. Speech-driven talking face generation (TFG) has been explored to synchronize video with