MuteSwap: Visual-informed Silent Video Identity Conversion

Yifan Liu, Yu Fang, Zhouhan Lin · 2025

Read MuteSwap: Visual-informed Silent Video Identity Conversion by Yifan Liu, Yu Fang, Zhouhan Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00498.txt.

Introduction

Yifan Liu, Yu Fang, and Zhouhan Lin present a 2025 study on making speech from silent video. Normal voice conversion needs sound from both the source and the target speaker. That fails when the source clip has no sound, as with silent film or a noisy room. Their task takes a still face and a silent clip of lip motion and builds speech from sight alone.

Their system, called MuteSwap, learns to match face and voice traits across the two senses. It also learns to pull apart the traits that face and voice share. In tests, the system made clear speech in the target person's voice and kept the words from the silent clip. It held up in noise, where sound-based methods broke down. The work shows that silent face-based voice conversion can work in practice.

Catalog record

CREATORYifan Liu, Yu Fang, Zhouhan Lin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MuteSwap: Visual-informed Silent Video Identity Conversion Yifan Liu, Yu Fang, Zhouhan Lin Abstract Conventional voice conversion modifies voice characteristics from a source speaker to a target speaker, relying on audio input from both sides. However, this process becomes infeasible when clean audio is unavailable, such as in silent videos or noisy environments. In this work, we focus on the task of Silent Face-based Voice Conversion ( SFVC ), which does voice conversion entirely from visual inputs. i.e., given images of a target speaker and a silent video of a source speaker containing lip motion, SFVC generates speech aligning the identity of the target speaker while preserving the speech content in the source silent video. As this task requires generating intelligible speech and converting identity using only visual cues, it is particularly challenging. To address this, we introduce MuteSwap , a novel framework that employs contrastive learning to align cross-modality identities and minimize mutual information to separate shared visual features. Experimental results show that MuteSwap achieves impressive performance in both speech synthesis and identity conversion, especially under noisy conditions where methods dependent on audio input fail to produce intelligible results, demonstrating both the effectiveness of our training approach and the feasibility of SFVC. Demo — https://pussycat0700.github.io/MuteSwap-Demo Code — https://github.com/PussyCat0700/DiVISe 1 Introduction Voice conversion (VC) ( Wang et al. 2021 ; Li, Tu, and Xiao 2023 ) aims to synthesize speech that preserves the linguistic content of a source utterance and adopting the vocal characteristics of a target speaker. While traditional VC methods rely entirely on speech to extract both content reference and speaker identity, facial features have been shown to correlate closely with vocal characteristics ( Saeed et al. 2021 ; Hannan et al. 2025 ) , making them a plausible alternative for identity representation in speech conversion. To address this, face-based voice conversion (FVC) ( Lu et al. 2021 ; Weng, Shuai, and Cheng 2023 ; Sheng et al. 2023 ; Lee et al. 2024 ; Rong and Liu 2024 ) extends traditional voice conversion by extracting speaker identity from speaker images, instead of speech recordings. While FVC is an exciting advancement, it is not feasible in scenarios where source speech is noisy, corrupted, or entirely missing. Figure 1: FVC (left) and the proposed SFVC task (right). To overcome these limitations, we introduce S ilent F ace-based V oice C onversion (SFVC), a novel extension of FVC that further replaces the content reference from audio to silent lip-region video (shown in Figure 1 ). In SFVC, both content reference and speaker identity are derived from silent videos, enabling speech generation even when no audio is available. The model infers linguistic content directly from the video frames while ensuring that the synthesized speech aligns with the target speaker’s identity, therefore immune to any disruption from the audio source. We demonstrate the robustness of SFVC compared to FVC methods in Figure 2 . As background noise intensifies from babble to siren, the performance of FVC methods declines significantly, with the converted speech exhibiting increasing spectral distortion and a progressive loss of linguistic content until it becomes unintelligible. In contrast, SFVC consistently produces clear and intelligible speech by relying exclusively on lip movements and facial images from silent videos, making it inherently immune to audio corruption. This highlights the fundamental advantage of SFVC in achieving robust speech conversion under real-world conditions where audio is degraded or unavailable. Figure 2: Comparison of log Mel-spectrograms converted from video sources with noisy sound track using FVC and SFVC. Top: source speech with babble noise. Bottom: source speech with siren noise. Compared to FVC, SFVC poses a more challenging problem: while FVC relies on high-quality speech recordings as content references, SFVC must infer linguistic content purely from silent lip-region video. This significantly increases the difficulty of content extraction, as visual cues alone provide limited and ambiguous information compared to audio. To address the challenge of content synthesis, SFVC builds on advances in video-to-speech (V2S) synthesis ( Prajwal et al. 2020a ; Mira et al. 2022 ; Kim, Hong, and Ro 2023 ; Choi, Kim, and Ro 2023 ; Hsu et al. 2023 ; Liu, Fang, and Lin 2025 ) , which serves as a foundational technology for generating intelligible speech from silent videos. However, V2S itself is not designed to handle identity conversion. Existing V2S systems typically synthesize speech in the identity of the input video’s speaker and lack the capacity to manipulate speaker characteristics independently. In this regard, our SFVC task goes beyond basic V2S by introducing speaker identity conversion as an integrated downstream objective. To the best of our knowledge, prior work has rarely explored identity conversion in the context of V2S, making SFVC one of the first to expand V2S into downstream speech manipulation tasks. This not only elevates the complexity of the task by requiring both accurate speech generation and in-place identity transformation, but also broadens the practical potential of V2S-based systems in real-world applications where content and speaker identity must be independently controlled. A key challenge in SFVC is the lack of parallel data: real-world datasets seldom provide silent videos paired with identity-transferred speech from the same content, making fully supervised training for speaker conversion impractical. As a result, SFVC must support zero-shot identity conversion, where the model generalizes to unseen target identities without access to paired training samples. This constraint introduces two unique training challenges specific to SFVC. The first challenge is in-modality disentanglement between the facial speaker identity and the lip-regional content reference. Since the video frames used for content extraction are cropped from facial image frames used for speaker identity, there is a significant overlap in visual information. This overlap makes it harder for the model to disentangle content and identity compared to FVC, where the two are provided through clearly distinct modalities. The second challenge is the cross-modality gap between visual inputs and speech outputs. Since SFVC relies only on facial features to guide speaker identity in the synthesized speech, the model must learn to align visual and audio modalities. Without dedicated training to address this discrepancy, it is difficult to ensure that the facial identity effectively guides speech synthesis in a natural-sounding manner during speech conversion. To address these challenges, we propose MuteSwap , the first identity conversion methodology for SFVC that converts silent facial videos into intelligible speech while simultaneously performing zero-shot speaker identity conversion based solely on reference facial images. To address the in-modality disentanglement challenge, a log-likelihood estimator is integrated in training with an Expectation Maximization (EM) learning process ( Cheng et al. 2020 ) , which minimizes the mutual information upper bound between the output embedding of facial identity encoder and lip-regional content encoder inspired by Wang et al. 2021 . For the cross-modality matching challenge, a speech encoder is trained jointly with our facial encoder using a contrastive learning approach, which ensures consistency between the speaker identity representations in the visual and audio modalities similar to CLIP ( Radford et al. 2021 ) . Both tasks are integrated into a single-stage training framework, allowing them to be learned simultaneously with the V2S task. Experiments show that MuteSwap offers substantial gains over traditional FVC methods in noisy settings, maintaining high speech quality and

Read the complete text