Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation
Fang, Yin, Haoyu · 2025
Read Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation by Fang, Yin, Haoyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19225.txt.
Introduction
Fang Kang, Yin Cao, and Haoyu Chen build a tool that makes a face talk and gives it a voice that fits, in work from 2025. Past tools start from a ready voice track, so face and voice can clash. Their task starts with just a face photo and the words to say.
Their frame, Face2VoiceSync, checks that the made voice matches the face it serves. It also lets users steer voice traits such as tone and pace. A light bridge net links big pre-trained face and sound models, so few parts need fresh training. A new score grades both range and sameness of voice. Tests put the tool at the top on sight and sound, all on one mid-size graphics card.
Catalog record
| CREATOR | Fang, Yin, Haoyu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19225
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation Fang, Yin, Haoyu Abstract Recent studies in speech-driven talking face generation achieve promising results, but their reliance on fixed-driven speech limits further applications (e.g., face-voice mismatch). Thus, we extend the task to a more challenging setting: given a face image and text to speak, generating both talking face animation and its corresponding speeches. Accordingly, we propose a novel framework, Face2VoiceSync, with several novel contributions: 1) Voice-Face Alignment, ensuring generated voices match facial appearance; 2) Diversity & Manipulation, enabling generated voice control over paralinguistic features space; 3) Efficient Training, using a lightweight VAE to bridge visual and audio large-pretrained models, with significantly fewer trainable parameters than existing methods; 4) New Evaluation Metric, fairly assessing the diversity and identity consistency. Experiments show Face2VoiceSync achieves both visual and audio state-of-the-art performances on a single 40GB GPU. Kang Cao Chen University of OuluFinland China keywords talking face generation, face-to-voice mapping, text-to-speech synthesis, variational autoencoder † † email: [email protected], [email protected], [email protected] 1 Introduction Talking Face Generation (TFG) aims to synthesize realistic facial animations synchronized with speech, enabling applications in virtual avatars, digital assistants, and media production. Recent advancements, particularly diffusion-based methods, have significantly improved lip-sync accuracy and facial expressiveness, drawing increasing research interest. However, most existing approaches are based on audio-driven pipelines [ 1 , 2 , 3 , 4 , 5 ] , which generate expressive facial movements and accurate lip synchronization from pre-recorded fixed audio , limiting their flexibility and leading to potential mismatches between voice and facial identity. To address this, we explore a more challenging setting: generating both talking faces and corresponding speech from a single static image and textual input , ensuring voice-identity consistency and enhanced controllability. To our knowledge, the closest setting among existing methods to our proposed task is text-driven TFG that generates speech directly from text, enabling precise control over speech content, and offering greater flexibility [ 6 , 7 ] . Existing methods can be divided into two categories. The first employs a cascaded pipeline [ 6 , 8 , 9 ] , where TFG models are followed by text-to-speech (TTS) systems. The second approach involves latent space fusion [ 7 , 10 , 11 ] , where speech and video are generated simultaneously within a shared latent space. Beyond improving speech controllability, text-driven methods can also enhance face-voice consistency by incorporating facial information into speech generation, ensuring that the synthesized voice aligns with the speaker’s visual appearance. This provides an advantage over audio-driven approaches, which may struggle to fully align visual and auditory features. On the other hand, current methods [ 7 , 8 , 12 ] frequently conceptualize the face-voice relationship as a one-to-one mapping, aiming to establish a direct correlation between facial features and voice attributes to improve the naturalness of generated speech. While text-driven methods offer considerable promise and benefits, they also pose several challenges in real-world implementations. First, generating high-quality audio and video simultaneously requires significant computational resources when training large-scale multimodal models from scratch [ 9 ] . Although adopting pre-trained unimodal models can reduce this burden, maintaining cross-modal alignment remains a challenge. Second, generating corresponding voice characteristics based on facial appearance is barely considered. The existing methods by default assume that voice generation is restricted to a single fixed voice style per face, limiting models’ adaptability to unseen faces (i.e., it’s desirable that models can generate diverse voices with a given face for uses to choose from). Third, as a result of the above assumption (fixed voice style per face), existing methods [ 11 , 13 ] primarily focus on video quality rather than face-voice identity consistency, with few efforts have been made for standardized metrics to evaluate this consistency. Figure 1: A framework comparison between Face2VoiceSync and other existing methods. Towards the challenges discussed above, we propose Face2VoiceSync, a scalable method for the TFG task by improving cross-modal alignment with flexible voice customization. An overview of the Face2VoiceSync framework is shown in Figure 1 . Specifically, we make several unique contributions: 1) Voice-Face Alignment, we use a VoiceAdapter module following the face image encoder to ensure generated voices match facial appearance, 2) we adopt a lightweight VAE, enabling diverse face-based voice generation rather than restricting voice generation to a single fixed voice style per face. 3) Instead of modifying the frozen video generator, we fine-tune the latent space for better cross-modal consistency and zero-shot adaptation. 4) Additionally, we introduce the Diversity-Consistency Tradeoff Score (DCTS) to fairly evaluate the generated voice diversity and identity consistency. By leveraging pre-trained unimodal models and VAE-driven alignment, Face2VoiceSync achieves state-of-the-art performance with reduced training costs. Demos are available 1 1 1 https://face2voicesync.github.io/ . 2 Related works Talking face generation (TFG) has predominantly utilized audio-driven methods to create speaking portraits from a static image and audio input. Early works, such as Chen et al. [ 14 ] and Prajwal et al. [ 15 ] convert speech into mouth movements, laying the foundation for TFG. SadTalker [ 2 ] improved realism using implicit 3D coefficient modulation, while diffusion-based models like GAIA [ 16 ] and DreamTalk [ 3 ] enhanced lip synchronization and speaking style diversity. More recently, Hallo [ 4 ] introduced hierarchical cross-attention to better synchronize audio with non-identity-related facial motions, enabling more natural and expressive talking faces. Beyond audio-driven methods, text-driven talking face generation offers greater flexibility by eliminating reliance on audio input, generating both speech and facial animations directly from text. A common approach integrates TTS and TFG models, though methods such as ADA-TTA [ 9 ] still require reference audio. To improve cross-modal synchronization, Zhang et al. [ 6 ] and Wang et al. [ 8 ] introduce intermediate speech representations, while Jang et al. [ 7 ] and Mitsui et al. [ 10 ] propose direct text-to-animation mappings. However, the absence of audio input poses challenges for natural synchronization, as models lack essential prosodic and temporal cues [ 17 ] . Conditional text-to-speech (TTS) synthesis generates speech from text while conditioning on various inputs, including text, audio, and images. CosyVoice [ 18 ] enables zero-shot voice cloning by extracting speaker characteristics from reference audio but still requires target speaker samples. For TFG, image-conditioned methods provide a more suitable alternative by modeling speaker traits directly from facial images. Lee et al. [ 19 ] train a TTS model using facial features but face challenges in duration prediction and speech quality. Lee et al. [ 20 ] further enhance personalized voice synthesis with a face encoder, achieving better gender differentiation. However, their approach still struggles to capture fine-grained vocal traits and lacks precise modulation of paralinguistic attributes in generated speech. In this work, we propose Face2VoiceSync, which improves cross-modal alignment and enables flexible voice customization using a lightweight VAE. By mapping facial features and speech into a shared latent space, it captures fine-grained vocal traits and refining paralinguistic modulation. 3 Method Figure 2: Framework of the proposed text-driven talking face generation system. The framework takes both text and image inputs, leveraging VoiceAdapter to bridge the audio-to-video generator and text-to-speech generator. Dashed lines represent data flow paths that are only active during the training phase. We propose Face2VoiceSync, a scalable text-driven talking face