Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
Jingran Xie, Shun Lei, Yue Yu, Yang Xiang, Hui Wang, Xixin Wu, Zhiyong Wu · 2025
Read Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data by Jingran Xie, Shun Lei, Yue Yu, Yang Xiang, Hui Wang, Xixin Wu, Zhiyong Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10937.txt.
Introduction
Jingran Xie and six co-authors teach spoken dialogue systems to answer with empathy, without paired speech question-answer data. Their paper from twenty twenty-five notes a data gap. Speech-linked language models learn the words in a question well. Speech style cues, such as tone, rarely come with training pairs. Responses thus miss the speaker's feelings.
Their method, Listen, Perceive, and Express, trains in two stages. First the model learns to hear content and sense emotion in speech. Then chain-of-thought prompts unlock warm, apt replies from those cues. Tests back the design's gains. The authors claim a first use of chain-of-thought for speech-based dialogue of this kind. The work aims at chat that feels more personal and keeps users engaged.
Catalog record
| CREATOR | Jingran Xie, Shun Lei, Yue Yu, Yang Xiang, Hui Wang, Xixin Wu, Zhiyong Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data Thanks: This work is supported by National Key Research and Development Program of China (2021ZD0112900), National Natural Science Foundation of China (62076144) and the Major Key Project of PCL (PCL2023A09). Jingran Xie, Shun Lei, Yue Yu, Yang Xiang, Hui Wang, Xixin Wu, Zhiyong Wu Abstract Empathetic dialogue is crucial for natural human-computer interaction, allowing the dialogue system to respond in a more personalized and emotionally aware manner, improving user satisfaction and engagement. The emergence of large language models (LLMs) has revolutionized dialogue generation by harnessing their powerful capabilities and shown its potential in multimodal domains. Many studies have integrated speech with text-based LLMs to take speech question as input and output text response. However, the lack of spoken question-answering datasets that include speech style information to supervised fine-tuning (SFT) limits the performance of these systems. As a result, while these systems excel at understanding speech content, they often struggle to generate empathetic responses. In response, we propose a novel approach that circumvents the need for question-answering data, called Listen, Perceive, and Express (LPE). Our method employs a two-stage training process, initially guiding the LLM to listen the content and perceive the emotional aspects of speech. Subsequently, we utilize Chain-of-Thought (CoT) prompting to unlock the model’s potential for expressing empathetic responses based on listened spoken content and perceived emotional cues. We employ experiments to prove the effectiveness of proposed method. To our knowledge, this is the first attempt to leverage CoT for speech-based dialogue. I Introduction Empathetic dialogue aims to generate personalized responses that not only address the content of the conversation but also recognize the state of users, such as their emotions. In this process, speech, as a carrier of both content and emotion, is of paramount importance. With the recent developments in Large Language Models (LLMs) such as GPT [ 1 , 2 ] and LLaMA [ 3 , 4 ] , these models have demonstrated significant potential as dialogue systems and have been widely applied in multimodal domains [ 5 , 6 , 7 ] , offering new opportunities for empathetic spoken dialogue. Since text-based LLMs still struggle with controlling speech synthesis, research [ 8 , 9 , 10 , 11 , 12 , 13 ] often focuses on speech-to-text (S2T) dialogue systems first. A common approach is the cascaded pipeline, which combines automatic speech recognition (ASR), speech emotion recognition (SER), and LLMs to generate responses. While this method yields high-quality responses, it suffers from issues like error propagation between modules and high latency (average 5.4s) [ 14 ] , degrading user experience. To address this, researchers have explored end-to-end (E2E) models that integrate speech and text through supervised fine-tuning (SFT) for smoother human-computer interaction. However, these E2E models struggle to generate emotion-based responses due to the lack of spoken empathetic question-answer (QA) datasets. Although using TTS to generate dialogue data is a possible solution [ 15 , 16 ] , the gap between synthesized and natural speech data still limits its real-world applicability. To address the lack of natural speech QA dataset, we draw inspiration from the success of Chain-of-Thought (CoT) reasoning [ 17 ] in both text-based dialogue [ 18 , 19 ] and speech tasks [ 20 , 21 ] . CoT prompting breaks complex tasks into smaller, manageable sub-tasks, allowing the model to solve them step-by-step. Therefore, we propose a novel method called “Listen, Perceive, and Express” (LPE) that aims to eliminate the need for empathetic speech QA data with CoT prompt. Our approach consists of a two-stage training process that trains LLM to listen the content and perceive emotion of speech separately. Then, we employ Chain of Thought (CoT) [ 17 ] prompting to enable step-by-step reasoning for expressing empathetic responses based on speech content and emotional cues. Following this LPE framework, our model acquires the capability to generate empathetic responses without requiring SFT with QA data. Although our method still generates text-based responses rather than spoken ones, it provides a low-cost solution for advancing empathetic spoken dialogue. Our contribution can be summarized as following: • We propose LPE framework that introduces CoT into empathetic spoken dialogue. By decomposing empathetic dialogue into listening, perceiving, and expressing steps, we leverage CoT to guide the model step-by-step from known tasks to unseen targets. • We develop a two-stage approach to adapt LLMs for empathetic dialogue, utilizing existing ASR and SER data to replace the need for spoken empathetic QA data and ensure the model learns from natural speech data distribution. • We conduct quantitative evaluations on our LPE and explore the impact of CoT prompt on model performance. The experimental results demonstrate the overall effectiveness of our approach and confirm that CoT significantly enhances the model’s ability to generate accurate and empathetic responses. Fig. 1: Architecture of the proposed model is shown on left side. On the right is the real sample showing how it listens, perceives, and expresses. () denotes the transcription of the input speech. <> denotes the emotion of the speech. We use <> to highlight each step of LPE, which is not a part of our prompt. II Proposed Method The LPE framework is illustrated in Fig. 1 . Firstly, we leverage a two-stage training process to train LLM to listen content and perceive emotion of speech. Then, we apply CoT prompting to allow the model to further express empathetic responses based on listening and perceiving without the need for SFT on corresponding QA datasets. II-A Model Architecture Our LPE framework consists of three core components: a speech encoder, two feature adapters, and a LLM decoder. We leverage LLaMA’s tokenizer to convert text prompt into text tokens. Speech Encoder We use the WavLM [ 22 ] large model as the speech encoder, which is capable of extracting both content and emotional features from speech. The model consists of 24 Transformer layers, where the upper layers primarily capture content information, and the lower layers focus on paralinguistic features [ 22 ] . We obtain content embedding from the last layer and emotion embedding by applying a weighted sum across output of all layers. Subsampler Adapter Since speech embedding are typically much longer than text embedding, we introduce a subsampler adapter to reduce the length discrepancy between the two modalities. The adapter uses three 1D convolutional layers followed by a bottleneck layer and layer normalization to map the speech features into the textual space. Multiscale Adapter To effectively capture emotional features, multiscale adapter applies weighted sum over all layer outputs of speech encoder. This is followed by two linear layers and mean pooling to generate emotion embedding. A linear classifier is used during training to predict emotion labels, though is not utilized during inference. LLM Decoder To generate text responses, we employ LLaMA2-7B-chat [ 4 ] as the LLM decoder. This model is chosen for its strong conversational abilities. To prevent catastrophic forgetting on internal knowledge, its parameters are frozen throughout the training process. II-B Two-Stage Training LPE employs a two-stage training process to achieve the capability of listening speech content and perceiving emotions. Semantic Alignment The goal of the first stage is to align the speech content embedding with the