Classification of Spontaneous and Scripted Speech for Multilingual Audio
Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos · 2024
Read Classification of Spontaneous and Scripted Speech for Multilingual Audio by Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11896.txt.
Introduction
Shahar Elisha and three colleagues present a 2024 paper on teaching machines to hear the difference between read-aloud and off-the-cuff speech. The distinction helps researchers study speaking styles and helps media platforms slice huge audio catalogs into searchable pieces. The hard part is a classifier that holds up across formats and languages.
Training on a large podcast collection, the team compared handcrafted sound features against modern audio transformers, checking performance across eleven language groups for bias. The transformers won throughout, setting the best marks on telling the two speech styles apart, and the models carried over reasonably to public non-podcast data. The authors say pretrained audio networks have made this once-fiddly task broadly reliable.
Catalog record
| CREATOR | Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11896
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Classification of Spontaneous and Scripted Speech for Multilingual Audio Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos Abstract Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research. It can also improve recommendation systems and discovery experiences for media users through better segmentation of large recorded speech catalogues. This paper addresses the challenge of building a classifier that generalises well across different formats and languages. We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers, utilising a large, multilingual proprietary podcast dataset for training and validation. We break down the performance of each model across 11 language groups to evaluate cross-lingual biases. Our experimental analysis extends to publicly available datasets to assess the models’ generalisability to non-podcast domains. Our results indicate that transformer-based models consistently outperform traditional feature-based techniques, achieving state-of-the-art performance in distinguishing between scripted and spontaneous speech across various languages. † † address: 1 Spotify Ltd 2 Centre for Digital Music, Queen Mary University of London 1 Introduction Large scale audio-based classification of multilingual speech is a technical challenge that can enable both academic research and industrial applications. For example, distinguishing between speech patterns, styles and tones can underpin contextual recommendations, advanced search and enable assistive technologies, to name a few. In this paper, we develop and evaluate models to distinguish between scripted and spontaneous speech on a large, multilingual podcast dataset. While the differences between read and improvised speech have been investigated in the linguistics literature [ 1 , 2 , 3 , 4 ] , we approach the problem from a computational perspective, with an emphasis on large-scale, diverse data, and modern audio models. The motivation for distinguishing between spontaneous and scripted speech in multiple languages is manifold: increasingly large audio repositories require efficient, performant, and generalisable content classification models. We also want to address the need to understand how speaking styles and languages affect various speech models. In recent years, media services have offered an increasingly large catalogue of recorded speech in a range of formats, such as podcasts, audiobooks, lectures and archival collections of broadcasts. As user preferences range over stylistic variations [ 5 ] , enriching catalogues with style and other labels (e.g., as in video [ 6 , 7 ] and visual arts [ 8 ] applications) results in better recommendation systems and an improved discovery experience for listeners. Building a high-performant and efficient scripted/spontaneous speech classifier is an initial and important step in this direction. In addition, some catalogues, such as those in major streaming platforms, encompass content from around the world, covering a wide range of languages, cultures, and formats. Therefore, we are particularly interested in developing methods that perform well across languages, topics and formats, as well as understanding the biases and shortcomings of the current state-of-the-art. More so, there is an increasing body of work suggesting a direct impact of speech style on speech processing research [ 9 ] , influencing tasks such as speaker identity [ 10 ] , emotion recognition [ 11 ] , disease recognition [ 12 ] , and speech generation [ 13 ] . For example, one study reported that anger detection performs better on acted speech compared to natural conversations [ 11 ] . This raises a question of how we bias speech models with curated datasets, and calls for a deeper understanding of how speech styles interplay with various speech models. We believe that the ability to reliably distinguish between scripted and spontaneous speech is needed to evaluate the effects of speaking style on downstream speech processing models. The implementation can encourage speech researchers to measure and mitigate biases, and thereby build more generalisable models. This is particularly important today given the explosion in popularity, accessibility and performance of transformer models, the increasing availability of large datasets of natural and diverse speech, and the presence and function of speech research in several aspects of day-to-day life [ 4 , 14 ] . Our contributions can be divided into three main parts: first, we train and compare the performance of different audio models for spontaneous vs scripted speech classification on a large, proprietary, multilingual podcast dataset. Second, we break down the performance of the models by language to assess multilinguistic biases. Third, we measure the models’ abilities to generalise to non-podcast domains using publicly available datasets. We provide a comparison of handcrafted acoustic and prosodic features to modern speech representation models, and an evaluation of how our results generalise to a range of data sources and languages. We find that transformer-based models achieve state-of-the-art performance in distinguishing scripted from spontaneous speech across different domains and languages. To our knowledge, this is the first effort to systematically study state-of-the-art models to distinguish spontaneous from scripted speech, with a novel investigation into the nuances of different languages through a large and diverse, multilingual dataset. The rest of the paper is structured in the following way: we review related works in Section 2 , outline details of our experimental setup in Section 3 , provide a breakdown of the experimental results in Section 4 , and conclude with a discussion on our contributions and future works in Section 5 . 2 Related works The relationship between various acoustic and prosodic features and distinct speaking styles, in particular read versus spontaneous speech, has been extensively investigated over the last few decades [ 1 , 2 , 3 , 4 ] . While no single feature holds substantial predictive power, prosodic features such as pitch (F0) contours and articulation rates seem indicative of spontaneous speech [ 1 , 3 , 15 ] . The salience and variability of these features can differ between languages, speaking situations, environments, and even individual speakers [ 4 , 10 ] . More recently, researchers have leveraged automated tools to understand the differences between speaking styles on larger datasets. A couple of studies explored building a multidimensional space of acoustic and prosodic features to characterise a range of speaking styles (e.g. conversations, audiobooks, speeches) [ 16 , 17 ] , while another study trained a binary classifier to distinguish read and spontaneous speech [ 18 ] . They all extracted a range of handcrafted features such as pitch ranges, speaker rates, duration of speech and silence, amongst other temporal, pitch, and conversational dynamics measures. It is generally agreed that the distributions of speech and silence segments are salient features to discriminate spontaneous from narrated speech. Other salient features, according to Christodoulides [ 17 ] , relate to interactivity (speaker turns and overlapping speech), pitch ranges and contours (e.g. rises and falls), and other temporal characteristics (e.g. speaker rate, articulation ratio), however Ryant and Liberman [ 16 ] find that pitch range is less relevant. Spontaneous conversations, compared to other speaking styles (e.g. political and professional public speeches), are found to be harder to classify due to larger speaker-specific variations [ 17 , 10 , 18 ] . More concretely, Veiga et al. [ 18 ] reported accuracies of 93.7% and 69.5% on read and spontaneous speech segments, respectively. The studies drew similar conclusions based on datasets in different languages (Portuguese; French; and English, Chinese, and Spanish). Ryant