EmoNews: A Spoken Dialogue System for Expressive News Conversations

Ryuki Matsuura Shikhar Bharadwaj Jiarui Liu Dhatchi Kunde Govindarajan Carnegie Mellon University {rmatsuur, sbharad2, jiaruil5, dkundego}@andrew.cmu.edu · 2025

Read EmoNews: A Spoken Dialogue System for Expressive News Conversations by Ryuki Matsuura Shikhar Bharadwaj Jiarui Liu Dhatchi Kunde Govindarajan Carnegie Mellon University {rmatsuur, sbharad2, jiaruil5, dkundego}@andrew.cmu.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13894.txt.

Introduction

Talking with a news voice that sounds flat can feel cold, even when the words are right. The authors build EmoNews, a spoken dialogue system that adjusts the emotion of its speech to fit the conversation. A large language model first reads the context and picks a fitting emotion. A prompt-based speech synthesizer then speaks the reply in that emotional tone. The design joins two research lines, task-oriented dialogue and emotional speech synthesis, that have mostly grown apart.

The team also proposes a subjective rating scale for judging such systems, since standard scores for social goals do not exist. In their tests, EmoNews beats a baseline system on emotion regulation and on user engagement. The authors read this as evidence that speech emotion plays a key role in engaging conversation. They release all of their source code openly.

Catalog record

CREATORRyuki Matsuura Shikhar Bharadwaj Jiarui Liu Dhatchi Kunde Govindarajan Carnegie Mellon University {rmatsuur, sbharad2, jiaruil5, dkundego}@andrew.cmu.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13894

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EmoNews: A Spoken Dialogue System for Expressive News Conversations Ryuki Matsuura Shikhar Bharadwaj Jiarui Liu Dhatchi Kunde Govindarajan Carnegie Mellon University {rmatsuur, sbharad2, jiaruil5, dkundego}@andrew.cmu.edu Abstract We develop a task-oriented spoken dialogue system (SDS) that regulates emotional speech based on contextual cues to enable more empathetic news conversations. Despite advancements in emotional text-to-speech (TTS) techniques, task-oriented emotional SDSs remain underexplored due to the compartmentalized nature of SDS and emotional TTS research, as well as the lack of standardized evaluation metrics for social goals. We address these challenges by developing an emotional SDS for news conversations that utilizes a large language model (LLM)-based sentiment analyzer to identify appropriate emotions and PromptTTS to synthesize context-appropriate emotional speech. We also propose subjective evaluation scale for emotional SDSs and judge the emotion regulation performance of the proposed and baseline systems. Experiments showed that our emotional SDS outperformed a baseline system in terms of the emotion regulation and engagement. These results suggest the critical role of speech emotion for more engaging conversations. All our source code is open-sourced. 1 1 1 https://github.com/dhatchi711/espnet-emotional-news/tree/emo-sds/egs2/emo_news_sds/sds1 1 Introduction Figure 1: System architecture. Proposed system uses emoTTS and sentiment analyzer. In this work, we develop a task-oriented SDS that can regulate emotional TTS based on contextual cues (emotional SDS) to enable more empathetic news conversations. Task-oriented SDSs must balance task- and social-goals to create engaging interactions Clavel et al. 2022 , with emotional speech regulation being crucial among social-goals. For instance, synthesizing "sad" speech for tragic earthquake news can foster user empathy and engagement. Appropriately managing emotional tone can thus enhance both user perception and overall experience Kurata et al. 2024 ; Concannon and Tomalin 2024 . To support such needs, the field of affective computing has developed emotional TTS techniques, which generate emotionally expressive speech by adjusting acoustic features like cadence, intensity, and pitch. Recent emotional TTS systems have achieved high-quality oral emotional expressions Cho and Lee 2021 ; Wang et al. 2023 ; Bott et al. 2024 . However, despite these advances, task-oriented emotional SDSs remain underexplored. This is primarily because socio-conversational research has been compartmentalized Clavel et al. 2022 , with SDS and emotional TTS developing separately and lacking an integrated framework. Moreover, evaluating social-goals like emotional speech regulation is difficult Kurata et al. 2024 , as these goals are multidimensional and lack clear definitions, leading to few established evaluation metrics. Thus, the gap between emotional TTS capabilities and their effective integration into SDS highlights an important area for further research. We develop a task-oriented emotional SDS and propose its evaluation method. Specifically, we focus on news summarization and Q&A as a target task due to its extensive prior studies. For emotional speech regulation, we adopt a cascade SDS architecture. We employ a PromptTTS Guo et al. 2022 fine-tuned on the ESD dataset Zhou et al. 2022 as our emotional TTS model. For evaluation, we use an empathy scale originally Concannon and Tomalin 2024 and assess the SDS’s ability to regulate emotional speech. Additionally, we manually evaluate both the system’s emotional speech regulation and task achievement Walker et al. 1997 , comparing it with SDSs that employ non-emotional TTS. Through this study, we contribute to the studies on socio-conversational system by: (i) providing a method for developing emotional SDS; and (ii) proposesing an evaluation method of emotional SDS. 2 System Design and Method As depicted in Figure 1 , we develop an emotional SDS for news conversations using a cascade architecture, building on a strong baseline Arora et al. 2025 by adding emotional awareness via sentiment-guided synthesis. Below, we describe the core components of both systems. The baseline system includes three modules: ASR, LLM, and TTS. The ASR transcribes user speech to text, which is encoded and compared against a News Database for relevant article retrieval. The LLM generates a response based on both the transcript and retrieved news snippets, and the TTS outputs spoken responses in a default tone. We utilize Retrieval Augmented Generation (RAG) in our system. The core ASR and RAG-LLM module are shared: the ASR transcript is passed to a RAG language model that selectively retrieves news to ground its replies, using dynamic in-context prompting for adaptability. Our proposed system enhances the baseline with a Sentiment Analyzer that infers the emotional tone ( neutral , happy , sad , angry , or surprised ) from the LLM’s text response. The emotion tag is fed to PromptTTS, an emotional TTS module that conditions speech synthesis on both text and emotion, producing expressive and empathetic responses. Compared to the emotionally neutral baseline, our system delivers more human-like, engaging interactions through sentiment understanding and emotional prosody. Metric Item RAG Evaluation The system was helpful to understand the retrieved news. Task Achievement 1 (Usefulness) The news that the system retrieved matched the information you wanted to know. Task Achievement 2 (Response Consistency) The system consistently responded according to the retrieved news. Speech Emotion Appropriateness The system seemed to vary its emotional state of speech to demonstrate expressiveness and modify its responses to accommodate the mood of the context. Engagement Did you have favorable feelings toward the one you were talking to? Did you feel a sense of familiarity with the one you were talking to? Did you feel that the system you were talking to understood the mood of contexts? Table 1: Emotional SDS Evaluation Questionnaire. Figure 2: Comparison of Evaluation Metrics by System Type. Metric U p-value Cohen’s d RAG Evaluation 42.5 0.580 0.301 Task Achievement 1 42.0 0.558 0.206 Task achievement 2 51.0 0.968 0.070 Speech Emotion Appropriateness 1.5 <0.001 3.070 Engagement 27.5 0.090 0.824 N Turn 26.5 0.073 0.831 Table 2: Statistical Comparison Between Baseline and Proposed Systems 3 Experiments To evaluate the extent to which the proposed method can control proper speech emotion, we compared it with a baseline system using human subjective judgments. 3.1 Datasets For emotional TTS fine-tuning, we used the English portion of the ESD dataset Zhou et al. 2022 , splitting 17,500 utterances into training, validation, and evaluation subsets across five emotions. For sentiment analyzer fine-tuning, we used GoodNewsEveryone Bostan et al. 2020 and GoEmotions Demszky et al. 2020 , mapping their emotion tags to five target categories following Koufakou et al. 2024 . As the news database for retrieval, we used Free News 2 2 2 https://github.com/Webhose/free-news-datasets , filtering for English articles and embedding news titles with Chroma 3 3 3 https://www.trychroma.com/ and Sentence Transformers Reimers and Gurevych 2019 . 3.2 System Setups Proposed System We used Whisper Large for ASR, LLaMA 3.2 1B for the language model, and a sentence transformer for retrieving the top 1 relevant news. For emotional TTS, we fine-tuned PromptTTS (pre-trained on LJSpeech). Our preliminary analysis showed that its quality was comparable to FastSpeech Ren et al. 2019 and VITS Kim et al. 2021 in terms of UTMOS, DNSMOS, PLCMOS, and WER, and qualitative analysis confirmed clear emotional variation. For the sentiment analyzer, we fine-tuned a distilled RoBERTa model (batch size 8, learning rate 0.00001, 4 epochs) after finding that prompt-based LLM approaches tended to over-predict sadness and surprise, achieving better performance than Koufakou et al. 2024 . Baseline System The baseline system shared the same modules as the proposed system, except the

Read the complete text