Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis

Yejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Geunbae Lee · 2025

Read Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis by Yejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Geunbae Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07376.txt.

Introduction

People who lose speech after a stroke still have a face, and a face carries clues about voice. Yejin Jeon and 4 co-authors work on face-to-speech synthesis in a 2025 research paper. Their system generates a voice from a facial image, aiming to sound like that person. Earlier methods used pre-trained visual encoders that discarded fine details such as gender or ethnicity.

The new method splits the face into small segments and merges them step by step into a rich representation. It learns speaker attributes in both the visual and acoustic domains at once. Training pairs several views of a speaker, under varied angles and lighting, with the same recordings. Subjective and objective tests show better face-voice match and more stable synthesis. Such tools could give non-speaking people a voice that feels like their own.

Catalog record

CREATORYejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Geunbae Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis Yejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Geunbae Lee Abstract For individuals who have experienced traumatic events such as strokes, speech may no longer be a viable means of communication. While text-to-speech (TTS) can be used as a communication aid since it generates synthetic speech, it fails to preserve the user’s own voice. As such, face-to-voice (FTV) synthesis, which derives corresponding voices from facial images, provides a promising alternative. However, existing methods rely on pre-trained visual encoders, and finetune them to align with speech embeddings, which strips fine-grained information from facial inputs such as gender or ethnicity, despite their known correlation with vocal traits. Moreover, these pipelines are multi-stage, which requires separate training of multiple components, thus leading to training inefficiency. To address these limitations, we utilize fine-grained facial attribute modeling by decomposing facial images into non-overlapping segments and progressively integrating them into a multi-granular representation. This representation is further refined through multi-task learning of speaker attributes such as gender and ethnicity at both the visual and acoustic domains. Moreover, to improve alignment robustness, we adopt a multi-view training strategy by pairing various visual perspectives of a speaker in terms of different angles and lighting conditions, with identical speech recordings. Extensive subjective and objective evaluations confirm that our approach substantially enhances face-voice congruence and synthesis stability. 1 Introduction The ability to communicate using one’s own voice is an intrinsic and fundamental aspect of human identity, self-expression, and social interaction. However, a range of neurological and physiological conditions can severely impair speech production mechanisms. For instance, dysarthria is a speech disorder that results from various etiological factors, including cerebrovascular incidents (e.g., strokes) and degenerative neuromuscular disorders such as multiple sclerosis and Parkinson’s disease Darley et al. 1969 ; Mulfari et al. 2021 . The severity of dysarthria varies, but nevertheless manifests as slurred, unintelligible, and phonetically distorted speech. In more extreme cases such as total glossectomy or orofacial myofunctional disorders, individuals may completely lose the ability to generate speech. These impairments significantly hinder verbal communication, often leading to frustration, social isolation, and a reduced quality of life Mertl et al. 2018 . Towards this, text-to-speech (TTS) systems Shen et al. 2018 ; Ren et al. 2021 ; Kim et al. 2021 have been utilized as assistive technologies to convert typed text into artificial speech Mertl et al. 2018 ; Jeon et al. 2025 . However, while they are effective for communication, they fail to preserve the speaker’s unique vocal identity. Due to this, personalized multi-speaker TTS methods Hu et al. 2020 ; Badlani et al. 2023 ; Jeon et al. 2024 could be a potential solution Mertl et al. 2018 as they are able to imitate a target speaker’s vocal characteristics for speech generation. Yet, achieving high-fidelity voice cloning typically requires extensive multi-speaker training or few-shot adaptation with a large amount of speech recordings. Thus, while zero-shot multi-speaker synthesis can ideally enable speaker adaptation, it remains infeasible for those without any accessible prior recordings, such as individuals with complete vocal muscle paralysis. This limitation highlights the need for alternative biometric modalities to infer speaker identity and enable personalized voice synthesis that is independent of speech recordings. Both early and recent psychological research suggests the existence of a strong correlation between facial identity with vocal characteristics. Specifically, studies such as Kamachi et al. 2003 ; von Kriegstein et al. 2005 ; Smith et al. 2016 have indicated that humans can infer aspects of a person’s voice, such as pitch and timbre, based solely on their facial features. As a result, this insight has driven investigations into face-to-voice (FTV) synthesis, where a facial image serves as a reference for generating a corresponding voice, and then producing speech in that inferred voice. Early methods of FTV synthesis relied on statistical techniques that mapped facial images onto facial landmarks, which were then transformed into eigenvoice representations Ohsugi et al. 2018 . These approaches, however, were limited by the need for manual alignment between facial and vocal features. More recent work has adopted multi-stage training pipelines Goto et al. 2020 ; Yang et al. 2023 ; Kang et al. 2025 , wherein a face encoder is first trained to produce embeddings that align with outputs from a separately trained audio-based speech encoder. Afterwards, during inference, the face encoder substitutes the speech encoder. While this strategy improves performance, it introduces training complexity and inefficiency. Moreover, because the facial representations are specifically trained to align with audio features, they fail to directly capture and utilize fine-grained local facial features such as gender or other demographic attributes that are essential in producing identity-consistent speech. Although Yang et al. 2023 attempts to explicitly incorporate such facial attributes, their method requires a combination of demographic metadata, 2D facial features, textures (e.g., skin, muscle), and 3D cranial structures. Not only does this introduce substantial complexity in terms of data acquisition and input modalities, but also necessitates a three-stage learning process to integrate these sources of information, which ultimately results in a complex and resource-intensive training pipeline. To address the limitations of prior work, specifically the reliance of inference-time-only visual encoders, underutilization of fine-grained facial features, and dependence on multi-stage training, we present a novel end-to-end FTV synthesis framework, which eschews external models and instead focuses on effective facial feature learning. Our method starts with a progressive facial encoder that decomposes and hierarchically aggregates local facial regions to form a rich visual identity embedding. Additionally, to ensure alignment of high-level semantic attributes such as gender and ethnicity, we introduce a bilateral attribute enhancement mechanism, which is applied to both the facial representation and to the synthesized audio output. A multi-view data augmentation strategy is further adopted where each speech sample is paired with a diverse set of facial images from the same speaker, which are captured under varying poses, lighting conditions, and facial movements. Through both subjective and objective evaluations, we demonstrate that our method generates speech with higher speaker fidelity and stronger FTV associations. 2 Related Work 2.1 Stylistic Speech Generation Advancements in TTS technology have greatly enhanced the naturalness of synthesized speech, thereby expanding its applications to areas such as voice imitation in the form of multi-speaker TTS. Methodologies of this domain incorporate a speaker encoder that extracts a speaker representation from a reference audio sample, and then conditions it into the backbone TTS model. Existing methodologies typically follow either a few-shot adaptation strategy, where a pre-trained multi-speaker TTS model is fine-tuned with multiple target speaker samples Huang et al. 2022 ; Chen et al. 2019 ; Chen et al. 2021 , or a zero-shot adaptation approach, which generates an embedding from a single target speaker sample to condition the model directly. Regardless of the adaptation objective, research in multi-speaker TTS has either focused on improving speaker encoders Cooper et al. 2020 ; Jeon et al. 2024 or refining speaker conditioning techniques by evolving from simple concatenation to more adaptive methods Min et al. 2021 ; Choi et al. 2022 ; Yoon et al. 2023 . While FTV synthesis is a form of multi-speaker TTS, it takes a fundamentally different approach. Instead of relying on reference audio at

Read the complete text