Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN

Toranosuke Manabe, Yuto Shibata, Shinnosuke Takamichi, Yoshimitsu Aoki · 2026

Read Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN by Toranosuke Manabe, Yuto Shibata, Shinnosuke Takamichi, Yoshimitsu Aoki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10413.txt.

Introduction

Manabe, Shibata, Takamichi, and Aoki carry sign language feeling straight into spoken voice. Turning signs into plain text first drops the non verbal color that signing holds. Their new task, sign to speech prosody transfer, keeps global tone and stress in the spoken result. SignRecGAN trains from single mode sets with no paired sign speech labels, using adversarial training and rebuild losses. Tests show the speech mirrors the signed mood, and the team plans open code.

Catalog record

CREATORToranosuke Manabe, Yuto Shibata, Shinnosuke Takamichi, Yoshimitsu Aoki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32074
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10413

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN Toranosuke Manabe, Yuto Shibata, Shinnosuke Takamichi, Yoshimitsu Aoki Abstract Deep learning models have improved sign language-to-text translation and made it easier for non-signers to understand signed messages. When the goal is spoken communication, a naive approach is to convert signed messages into text and then synthesize speech via Text-to-Speech (TTS). However, this two-stage pipeline inevitably treat text as a bottleneck representation, causing the loss of rich non-verbal information originally conveyed in the signing. To address this limitation, we propose a novel task, Sign-to-Speech Prosody Transfer , which aims to capture the global prosodic nuances expressed in sign language and directly integrate them into synthesized speech. A major challenge is that aligning sign and speech requires expert knowledge, making annotation extremely costly and preventing the construction of large parallel corpora. To overcome this, we introduce SignRecGAN , a scalable training framework that leverages unimodal datasets without cross-modal annotations through adversarial learning and reconstruction losses. Furthermore, we propose S2PFormer , a new model architecture that preserves the expressive power of existing TTS models while enabling the injection of sign-derived prosody into the synthesized speech. Extensive experiments demonstrate that the proposed method can synthesize speech that faithfully reflects the emotional content of sign language, thereby opening new possibilities for more natural sign language communication. Our code will be available upon acceptance. Figure 1: In the reference sign language video (left), the first phrase, “many Italians,” is emphasized through rapid hand movements and facial expressions. The two-stage baseline (middle) fails to reflect this prosody, whereas our approach (right) successfully captures the emphasis on “many.” 1 Introduction Sign language serves as a principal means of communication for individuals with hearing impairments. Translating between sign language and spoken language is, however, a complex task that typically requires specialized knowledge. Recently, sign-to-text translation methods based on deep learning have been developed [ 2 , 32 , 12 , 7 , 33 ] . These methods substantially simplify the process of bridging sign language and spoken language. However, existing studies are all limited to sign-to-text conversion. Therefore, to convert sign language into speech using current methods, one must rely on a decoupled process consisting of sign-to-text and text-to-speech (TTS). In this pipeline, the intermediate text inevitably becomes a bottleneck, causing non-verbal information present in the original sign, such as tension and emphasis, to be lost. Omitting these elements can result in speech that lacks expressiveness. To fill this gap, we propose a novel task, sign-to-speech prosody transfer to incorporate the global prosody embedded in sign language into speech synthesis process. This task can be seen as an extension of existing cross-lingual prosody transfer [ 28 ] . While the existing task aims to transfer prosody from source spoken language to target one, our task transfers from source sign language . Here, translating between sign language and speech requires highly specialized expertise. Consequently, constructing large-scale paired datasets is much more difficult than in other multi-modal transformation problems such as text-to-image [ 21 ] , where massive paired data are readily available. Moreover, both sign language and speech are time-series data, and creating annotations that take their temporal alignment into account is even more challenging. To address these challenges, in this paper we propose SignRecGAN , a method for converting global prosody from sign language into speech using separate unpaired unimodal datasets of sign and speech, together with S2PFormer , a S ign-to- P rosody Trans former designed for high-quality prosody-aware speech synthesis. SignRecGAN combines adversarial learning with reconstruction losses, SignRec loss (Sign Reconstruction loss) and ProMo loss (Prosody-Motion alignment loss), which explicitly force the model to reconstruct motion information of sign language from speech, thereby enabling the motion expressed in signing to be reflected in natural-sounding speech. S2PFormer treats a large-scale pretrained TTS model as the base speech synthesis module and augments it with a carefully designed additional cross-attention branch, achieving speech synthesis that incorporates sign information while preserving the original high speech quality. These training strategy and model design are highly scalable, because SignRecGAN leverages annotation-free unpaired unimodal datasets while S2PFormer reuse a large-scale pretrained TTS model as a base speech synthesizer. We validate the effectiveness of the SignRecGAN through a qualitative evaluation, demonstrating that the resulting speech reflects a richer range of expressions compared to the conventional two-stage approach. CMOS user study also confirms that our method can synthesize emotionally nuanced speech. In ablation studies, we show that the guidance for reconstructing sign language motion contributes to prosodic representation, supporting the soundness of our system design. In summary, our main contributions are as follows: • We propose a novel task sign-to-speech prosody transfer to incorporate the global prosody embedded in sign language into speech synthesis process. For this task, we introduce SignRecGAN , an adversarial and reconstruction-based learning framework using separate unimodal datasets of sign and speech. We also propose S2PFormer , an augmented pretrained TTS model with a carefully designed additional cross-attention branch. • For reconstruction-based learning , we propose SignRec loss and ProMo loss to address the absence of directly aligned sign and speech data. SignRec loss ensures that the synthesized speech retains the nuances of sign language by reconstructing original sign motions, while ProMo loss aligns the distributions of sign language and synthesized speech. These two components ensure proper integration of sign language cues into speech synthesis and enhance overall consistency. • Through user studies and quantitative assessments, we demonstrate that our system captures a broader range of emphatic cues compared to existing pipelines, resulting in more expressive and natural speech. 2 Related Work 2.1 Prosody in Sign Language Just as spoken language prosody conveys emphasis or emotion in a sentence, sign language also exhibits prosodic features. The prosody of sign language is expressed through the speed and amplitude of hand movements, as well as facial expressions. Moreover, there is a correspondence between sign language motion and spoken language [ 1 ] . For instance, the length of a sign is similar to the duration of speech [ 30 ] , the peak velocity of a sign is analogous to pitch [ 11 ] , and the movement displacement is similar to speech intensity [ 30 ] . However, because the correspondence in prosody between sign language and speech is extremely complex and subtle, converting prosody from sign language to speech based on predefined rules is difficult. As a result, no research currently exists that reflects sign language prosody in speech synthesis. 2.2 Sign-to-Speech Sign-to-speech collectively refers to the technologies that convert sign language into speech, and in recent years, many such studies have been conducted. However, these studies mainly focus on a two-stage approach of first converting sign language to text, followed by converting text to speech. Examples include proposals of devices for sign language recognition [ 34 , 23 , 5 ] and image recognition-based deep learning approaches that prioritize higher speeds and better accuracy [ 17 , 18 ] . Consequently, there is no established method that integrates sign language prosody into speech. 2.3 Prosody Transfer Prosody transfer refers to techniques that condition a TTS system on a reference speech signal so that its prosodic characteristic, such as pitch, energy, speaking rate, and pause

Read the complete text