The Role of Prosody in Spoken Question Answering

Jie Chi, Maureen de Seyssel, Natalie Schluter · 2025

Read The Role of Prosody in Spoken Question Answering by Jie Chi, Maureen de Seyssel, Natalie Schluter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05389.txt.

Introduction

Jie Chi, Maureen de Seyssel, and Natalie Schluter investigate the role of prosody in spoken question answering. Their 2025 paper notes that spoken language research has taken a heavy text perspective, to the detriment of prosody.

Most datasets start as text and are then synthesized into speech, and most models rely on automatic transcripts. Prosody, the extra information in speech beyond the words, is hard to recover from text. On a dataset of natural speech, the authors separate prosodic and lexical information. Models trained on prosody alone did reasonably well. But when lexical information was present, models leaned on it, so better ways to join the two are needed.

Catalog record

CREATORJie Chi, Maureen de Seyssel, Natalie Schluter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Role of Prosody in Spoken Question Answering Jie Chi, Maureen de Seyssel, Natalie Schluter Abstract Spoken language understanding research to date has generally carried a heavy text perspective. Most datasets are derived from text, which is then subsequently synthesized into speech, and most models typically rely on automatic transcriptions of speech. This is to the detriment of prosody–additional information carried by the speech signal beyond the phonetics of the words themselves and difficult to recover from text alone. In this work, we investigate the role of prosody in Spoken Question Answering. By isolating prosodic and lexical information on the SLUE-SQA-5 dataset, which consists of natural speech, we demonstrate that models trained on prosodic information alone can perform reasonably well by utilizing prosodic cues. However, we find that when lexical information is available, models tend to predominantly rely on it. Our findings suggest that while prosodic cues provide valuable supplementary information, more effective integration methods are required to ensure prosody contributes more significantly alongside lexical features. 1 Introduction Prosody, which is characterized by elements of speech beyond orthographic words, such as pitch, stress and rhythm, plays a critical role in both speech production and perception. It has been shown to impact how people perceive speech, with difficulties often arising when the natural variability in prosodic structure is limited, as is the case with synthetic speech ( Winters and Pisoni, 2004 ; Wester et al., 2016 ) . In human listening comprehension, prosodic cues are essential in guiding listeners through the process of interpreting spoken language Buck (2001) ; Keskin et al. (2019) . The incorrectly-stressed elements in speech can also cause listeners to make incorrect inferences Field (2005) . Motivated by these linguistic findings, researchers have explored how prosody can be leveraged in speech-related tasks computationally. One of the primary tasks in this area is Spoken Language Understanding (SLU), which focuses on extracting meaningful information from spoken language input. Unlike Natural Language Understanding (NLU), which primarily deals with text-based information, SLU incorporates the added complexity of processing signal made of prosodic features such as intonation, stress, and pauses. In this work, we use the term lexical information to refer to information that is also present in the text in its orthographic form. That is, info that encompasses phonetic information from the speech signal. Therefore, in our work, we categorize anything that is not lexical as prosodic information. Figure 1: Illustration of the SQA format Recent advances in NLU research have significantly impacted SLU through the use of a cascade approach. This approach consists of two key components: an Automatic Speech Recognition (ASR) model that transcribes speech into text, followed by an NLU model that is fine-tuned for specific downstream tasks. Spoken Question Answering (SQA) is one of the challenging SLU tasks, and takes the form of listening comprehension. In SQA, the input consists of a spoken passage accompanied by a question about that passage, and the model is required to provide the correct answer. This answer can take the form of timestamps in the passage, as used in this paper and previous works Lee et al. (2018) ; Lin et al. (2022) , or it can be a textual output Shon et al. (2024) . Figure 1 illustrates the typical input-output structure of an SQA task. The evolution of SQA research has gradually shifted from synthetic speech datasets to those based on natural speech. In early studies, researchers utilized Text-To-Speech (TTS) systems to convert existing Textual Question Answering (TQA) datasets into large-scale SQA corpora Lee et al. (2018) ; Lin et al. (2022) ; Ünlü Menevşe et al. (2022) . However, since SQA means prosodic information is available in the input, the prosodic characteristics of synthetic speech may not accurately represent those found in natural speech Wester et al. (2016) ; Clark et al. (2019) ; Chan and Kuang (2024) . This has led to concerns about the effectiveness of using synthetic speech for tasks where prosody plays a crucial role. To address these limitations, recent efforts have focused on integrating more natural speech into SQA datasets. Some studies have developed small test sets read by human speakers Lin et al. (2022) , while others have explored hybrid datasets where the questions are recorded by humans, but the passages are synthetically generated Wu et al. (2024) . Additionally, there has been work on creating training sets by sourcing spoken documents relevant to each question from external natural speech corpora Shon et al. (2023) . Given the availability of the natural SQA training datasets, we aim to explore whether models can utilize prosody when comprehending speech, as humans do. The cascade approach, however, is not well-suited for this task, as prosodic information is typically lost after the transcription stage, and recovering prosody from text has been shown to be difficult Talman et al. (2019) . Although some research has attempted to explicitly incorporate word-level prosodic features into NLU models Tran et al. (2018) ; Tran et al. (2019) , the errors from ASR and alignments tend to propagate, resulting in ill-formed inputs and significantly impacting performance. Recent developments, particularly in Self-Supervised Learning (SSL) representations, have enabled researchers to bypass explicit transcription through end-to-end models Chuang et al. (2020) or by using discrete units as pseudo-text Lin et al. (2022) , which latter is the framework adopted in this paper. In this work we are motivated to gain a comprehensive understanding of how prosodic information, distinct from the lexical information used in NLU tasks, contributes to the SQA task. Specifically, two key research questions are investigated: 1) Is prosodic information sufficient for SQA tasks? as intonation, pitch, and pauses, signal important structural and emphatic aspects of speech , and 2) Do SQA models utilize prosodic information when lexical information is also present? To address these questions, we carefully design two experimental conditions of our dataset: one that approximates prosodic information only, and another that approximates lexical information only. Directly disentangling prosodic and lexical content in speech is a complex challenge that remains unsolved Quamer and Gutierrez-Osuna (2024) ; Skerry-Ryan et al. (2018) . Among the various approaches proposed to achieve delexicalization and isolate the contribution of prosody, applying a low-pass filter has been one of the most widely used techniques, in psycholinguistics but also in modelling Goldman et al. (2014) ; Niebuhr et al. (2020) ; Audibert et al. (2023) ; it can preserve an approximation of the prosodic features while removing most of the discriminating information that phonetically delineates orthographic words Mehler et al. (1988) . Hence, for the prosodic condition, we apply a low-pass filter to remove information above a certain cutoff frequency, ensuring that most lexical information is excluded from the speech signal. In contrast, for the lexical condition, we flatten both pitch and intensity to eliminate most prosodic variation. While prosodic and lexical information are not completely disentangled, our experiments show that the reduction of these elements is sufficient to prevent significant interference of one variable over the other in the results. Through controlled experiments, we demonstrate that prosodic information alone can, to some extent, guide models to answer questions in SQA tasks. However, while prosody offers meaningful complementary cues, we find that models predominantly rely on lexical

Read the complete text