Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis

Zhu, Yuqing, Xiyuan, Devraj, Nagendra, Shekhar, Matt · 2025

Read Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis by Zhu, Yuqing, Xiyuan, Devraj, Nagendra, Shekhar, Matt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13028.txt.

Introduction

Li and six co-authors teach a speech synthesizer to sound sarcastic, a tone machines usually miss. Sarcasm lives in subtle melody and timing, and labeled sarcastic speech is scarce. Their method borrows feedback from a detector that judges sarcasm from both sound and text.

The detector's score becomes an extra training loss, pushing the synthesizer toward clearer sarcasm. A two-stage tune-up first exposes the model to many speech styles, then narrows to sarcastic speech. Listening tests and objective measures showed gains in quality, naturalness, and sarcasm strength. The authors aim at livelier entertainment and human-computer interaction.

Catalog record

CREATORZhu, Yuqing, Xiyuan, Devraj, Nagendra, Shekhar, Matt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13028

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis Zhu, Yuqing, Xiyuan, Devraj, Nagendra, Shekhar, Matt Abstract Sarcastic speech synthesis, which involves generating speech that effectively conveys sarcasm, is essential for enhancing natural interactions in applications such as entertainment and human-computer interaction. However, synthesizing sarcastic speech remains a challenge due to the nuanced prosody that characterizes sarcasm, as well as the limited availability of annotated sarcastic speech data. To address these challenges, this study introduces a novel approach that integrates feedback loss from a bi-modal sarcasm detection model into the TTS training process, enhancing the model’s ability to capture and convey sarcasm. In addition, by leveraging transfer learning, a speech synthesis model pre-trained on read speech undergoes a two-stage fine-tuning process. First, it is fine-tuned on a diverse dataset encompassing various speech styles, including sarcastic speech. In the second stage, the model is further refined using a dataset focused specifically on sarcastic speech, enhancing its ability to generate sarcasm-aware speech. Objective and subjective evaluations demonstrate that our proposed methods improve the quality, naturalness, and sarcasm-awareness of synthesized speech. Li Zhang Gao Raghuvanshi Kumar Nayak Coler University of GroningenThe Netherlands, Brown UniversityProvidence, R.I., USA Indian Institute of Technology IndoreIndore, India keywords expressive speech synthesis, sarcastic speech, sarcasm detection, feedback loss † † email: {zhu.li, yuqing.zhang, xiyuan.gao, s.nayak, m.coler}@rug.nl, [email protected], [email protected] 1 Introduction Sarcasm is a nuanced form of communication where speakers convey meaning contrary to their literal words, often employing exaggerated intonation, unexpected pauses, and shifts in emphasis. These subtle cues make sarcastic speech challenging to interpret and synthesize [ 1 , 2 ] . As speech technology increasingly integrates into everyday applications, such as virtual assistants, interactive media, and conversational AI, the ability to accurately generate colloquial and expressive speech, including sarcasm, is crucial for enhancing human-computer interaction. Text-to-speech (TTS) systems have advanced significantly with the advent of deep learning, enabling high-quality, natural-sounding speech synthesis from text [ 3 , 4 , 5 ] . However, most existing systems focus on generating neutral or reading-style speech, with limited success in improving speech synthesis in spontaneous or dynamic contexts or capturing expressive variations such as sarcasm [ 6 , 7 , 8 ] . Recent research has sought to address these challenges by introducing methods to capture and control emotional expressiveness and improve the prosody and variability of synthetic speech. For instance, Li et al. [ 9 ] conduct controllable emotional transfer by incorporating an emotion style classifier with a feedback loop, where the classifier encourages the TTS model to generate speech with specific emotions. Similarly, O’Mahony et al. [ 10 ] combine real spontaneous conversations with read speech to augment the training data to improve the prosody of synthetic speech. Li et al. [ 11 ] achieve high-quality expressive speech synthesis by introducing a semi-supervised pre-training approach that leverages a large-scale, low-quality spontaneous speech dataset. This method enriches both the quantity of spontaneous-style speech and the diversity of associated behavioral labels, improving the naturalness and expressiveness of the synthesized speech. Despite significant advancements in enhancing the expressiveness and prosody of TTS systems, synthesizing more complex forms of speech, such as sarcasm, remains a challenging task. Sarcasm often emerges in spontaneous, conversational contexts, where the subtle and dynamic nature of human speech is most pronounced, making its synthesis particularly difficult. Sarcasm, with its distinctive expressive qualities—such as exaggerated intonation, unexpected pitch variations, and altered timing—requires a more nuanced approach to accurately model the dynamic nature of speech. These unique features of sarcastic speech present a challenge for current TTS models, which are primarily trained on neutral or standard emotional speech. Another major obstacle to progress in sarcastic speech synthesis is the limited availability of dedicated datasets [ 12 ] . Sarcasm in speech is less frequently recorded and studied compared to neutral or emotional speech, limiting the training data available for TTS models. The scarcity of training data restricts the ability of models to learn and accurately reproduce the specific features of sarcasm. While techniques such as data augmentation and transfer learning have been explored to mitigate the lack of data [ 13 , 14 , 15 ] , they have yet to fully overcome the inherent difficulties in capturing the essence of sarcasm in synthesized speech. For instance, Huybrechts et al. [ 14 ] introduced a novel 3-step methodology to create expressive style voices with minimal data. Their approach leverages voice conversion to augment data from other speakers, followed by training a TTS model on both the augmented and available recordings, and fine-tuning to further enhance quality. This method demonstrates the potential of data augmentation techniques to build expressive TTS systems with minimal data. However, despite such advancements, synthesizing complex emotional speech like sarcasm remains a significant challenge. In contrast to the relatively limited work on sarcastic speech synthesis, the field of sarcasm detection has received considerable attention, with approaches that utilize both unimodal (e.g., audio) and multimodal (e.g., audio, visual, and textual) data [ 16 , 17 , 18 , 19 , 20 ] . In audio-based approaches, machine learning methods have predominated, with the selection of acoustic features such as prosodic, spectral, and contextual cues playing a crucial role [ 16 , 17 ] . Multimodal approaches leads to the introduction of several multimodal sarcasm datasets that encompass audio, visual, and textual modalities [ 18 , 19 , 20 , 21 , 22 ] , on which detection benchmarks have been established. Subsequent research has concentrated on refining methods for integrating these different modalities and improving fusion techniques to enhance detection accuracy [ 23 , 24 ] . Sarcasm detection models have demonstrated significant potential in identifying the acoustic and contextual markers of sarcasm, and the insights from these models could be valuable for guiding sarcastic speech synthesis. However, to date, there has been limited exploration of how sarcasm detection could inform the generation of sarcastic speech in TTS systems. Building on advances in multi-modal sarcasm detection and drawing on the success of data augmentation techniques in low-resource TTS, we propose three methods to enhance the sarcasm-awareness of the sarcastic speech synthesis model and improve the quality and naturalness of the synthesized speech. The main contributions of this work are summarized as follows: 1. Novel Integration of Sarcasm Detection Feedback: We introduce a novel approach that integrates a bi-modal sarcasm detector into the sarcastic speech synthesis pipeline. Specifically, we incorporate feedback loss derived from sarcasm detection into the TTS model training process. This integration enhances the model’s ability to capture the nuances and expressiveness of sarcastic speech, making it sarcasm-aware. Both objective and subjective evaluation results demonstrate that this incorporation enhances the model’s ability to convey sarcasm, enabling it to better capture the subtle and expressive elements of sarcastic speech. 2. Comprehensive Comparison of Input Modalities: We perform a comprehensive comparison of various input types for sarcasm detection, revealing that combining text and speech inputs leads to a notable improvement in the F1-score. This finding highlights the importance of incorporating textual information to detect sarcastic behaviors that are difficult to identify through speech alone, underscoring the complexity of sarcasm. 3. Two-Stage Fine-tuning Method: We propose a two-stage fine-tuning method for TTS models, which improves the model’s

Read the complete text