Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer
Soumya Dutta, Avni Jain, Sriram Ganapathy · 2025
Read Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer by Soumya Dutta, Avni Jain, Sriram Ganapathy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17655.txt.
Introduction
Soumya Dutta, Avni Jain, and Sriram Ganapathy build a 2025 system that moves the emotion of one voice recording onto another. Given a source clip and a reference clip, the goal is speech that keeps the source's words and speaker but feels like the reference. Past systems often needed the same sentences spoken in both moods, or text in the middle. Theirs needs neither.
The system first breaks speech into parts: meaning tokens, a speaker profile, and an emotion profile. From those it learns to predict the pitch curve and the timing of the output. A synthesis stage then rebuilds the voice from the parts, wearing the reference emotion. The zero-shot design means it can handle speakers and feelings it never trained on. Such tools serve dubbing, assistants, and expressive reading aids.
Catalog record
| CREATOR | Soumya Dutta, Avni Jain, Sriram Ganapathy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17655
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer Soumya Dutta, Avni Jain, Sriram Ganapathy Abstract Given a pair of source and reference speech recordings, speech-to-speech (S2S) emotion style transfer involves the generation of an output speech that mimics the emotion characteristics of the reference while preserving the content and speaker attributes of the source. In this paper, we propose a speech-to-speech zero-shot emotion style transfer framework, termed as S2S Z ero-shot E motion S tyle T ransfer ( S2S-ZEST ), that enables the transfer of emotional attributes from the reference to the source while retaining the speaker identity and speech content. The S2S-ZEST framework consists of an analysis-synthesis pipeline in which the analysis module extracts semantic tokens, speaker representations, and emotion embeddings from speech. Using these representations, a pitch contour estimator and a duration predictor are learned. Further, a synthesis module is designed to generate speech based on the input representations and the derived factors. The analysis–synthesis pipeline is trained using an auto-encoding objective to enable efficient resynthesis during inference. For S2S emotion style transfer, the emotion embedding extracted from the reference speech along with the rest of the representations from the source speech are used in the synthesis module to generate the style translated speech. In our experiments, we evaluate the converted speech on content/speaker preservation (w.r.t. source) as well as on the effectiveness of the emotion style transfer (w.r.t. reference). The proposed framework demonstrates improved emotion style transfer performance over prior methods in a textless and non-parallel setting. We also illustrate the application of the proposed work for data augmentation in emotion recognition tasks. I Introduction Emotion Style Transfer (EST) aims to modify the emotional expression of a speech signal while preserving key attributes such as speaker identity and linguistic content [ 52 ] . This technology has broad applications, particularly in human-computer interaction, where it enables machines to not only recognize but also respond to human emotions more naturally [ 37 , 6 , 41 ] . Despite recent advancements in speech synthesis, EST remains challenging due to the complexity of modeling human emotions and the difficulty of accurately transferring them in generation tasks [ 38 ] . Early approaches for EST focused on modifying the spectral envelope and pitch contour (F0 contour) of neutral speech to generate emotional speech. Inanoglu et al. [ 17 ] employed a hidden Markov model (HMM) for F0 contour conversion and a Gaussian mixture model (GMM) for spectral transformation. Luo et al. [ 30 ] utilized deep neural networks to convert mel-frequency cepstral coefficients (MFCC) and the F0 contour, while Lorenzo-Trueba et al. [ 28 ] explored deep learning architectures for emotional speech synthesis. A major limitation of these approaches is their reliance on parallel emotional speech datasets, where the same speaker articulates the same content in multiple emotions. Recording such data at scale is both expensive and challenging. To overcome this limitation, a large body of prior work in voice conversion and expressive speech synthesis has explored factorized representations of speech, in which content, speaker identity, and speaking style are modeled separately [ 39 , 21 , 32 , 34 , 49 ] . Inspired by this established paradigm, emotion style transfer can be formulated as a mixing problem, where the content and speaker characteristics of a source utterance are combined with the emotional style of a reference utterance. A model’s ability to perform such mixing without explicit supervised training on parallel data is referred to as zero-shot emotion style transfer (ZEST) . Building on this motivation, we propose a speech-to-speech emotion style transfer (EST) framework that enables zero-shot transfer of emotional speaking style from a reference to a source utterance. Prior speech style transfer and voice conversion methods [ 34 , 49 , 32 ] often exhibit entanglement between speaker identity and expressive style in practice. As a result, these approaches often fail to transfer the emotion style to unseen speakers. A recent effort by Zhang et al. [ 50 ] has attempted to address these challenges through improved disentanglement and control mechanisms. However, this approach relies on large-scale training data. Our early effort attempted emotion style transfer through a factored representation learning setting [ 11 ] . In this paper, we extend our efforts in multiple ways. Our approach leverages two modules - • An analysis module that encodes the input speech into four key components: semantic tokens, speaker representations, pitch features, and emotion embeddings. • A synthesis module, based on BigVGAN [ 25 ] , that reconstructs the speech given the tokens, speaker embeddings, emotion embeddings and pitch features, which is fully trained using the reconstruction (auto-encoding) loss. The key contributions beyond the prior work [ 11 ] are: • We propose the tokenization of soft-HuBERT [ 45 ] features for speech content representation and employ BigV-GAN as a unit-to-speech vocoder, replacing the conventional HuBERT [ 15 ] and HiFi-GAN [ 19 ] models. • We use a duration estimator that allows the emotion and speaker embeddings to modify duration of the semantic tokens during inference. We further develop a pitch reconstruction technique that operates on duration-modified tokens, along with speaker and emotion information, to enhance the naturalness of the synthesized speech. • We conduct extensive subjective and objective evaluations, demonstrating that S2S-ZEST outperforms baseline methods in emotion style transfer. • We showcase the application of the S2S-ZEST as a data augmentation tool for speech emotion recognition (SER). II Related work II-A Emotional Style Transfer for Text-to-Speech Synthesis One of the earliest works, by Wang et al. [ 46 ] , introduced “style tokens” to learn the speaking style of a reference speech signal. Subsequent methods by Min et al. [ 33 ] and Lee et al. [ 24 ] further refined style transfer techniques, though they primarily operated at the sentence level. Generspeech [ 16 ] addressed this limitation by incorporating more localized style cues from reference speech. In contrast to these works, our work on S2S-ZEST operates in a completely textless setting in both training and style transfer. II-B Synthesis of Speech from Discrete Tokens Speech synthesis from discrete tokens was explored with vocoder models, ranging from classical methods (Griffin-Lim [ 14 ] , WaveNet [ 43 ] ) to GAN-based models (HiFi-GAN [ 19 ] , BigVGAN [ 25 ] ) and diffusion-based approaches (DiffWave [ 20 ] ). Polyak et al. [ 39 ] introduced a speech synthesis approach that bypasses mel-spectrogram reconstruction by encoding speech content using HuBERT [ 15 ] . These representations were concatenated with pitch and speaker representations to train a HiFi-GAN vocoder for speech synthesis, a method adopted in multiple studies [ 22 , 23 , 32 , 21 ] . In contrast, our work explores a purely zero-shot setting for the style transfer task using components like duration, pitch, and emotion embeddings. II-C Style Transfer in Speech Synthesis Most prior works in speech style transfer modify both speaker identity and speaking style, and therefore fall under the ambit of voice conversion (VC). Notable examples include DISSC [ 32 ] , PromptVC [ 48 ] , StableVC [ 49 ] , as well as methods that incorporate emotion cues such as Kreuk et al. [ 21 ] and DurFlex-EVC