OleSpeech-IV: A Large-Scale Multispeaker and Multilingual Conversational Speech Dataset with Diverse Topics

Wei Chu 1 , Yuanzhe Dong 1,2 , Ke Tan 1 , Dong Han 1 , Xavier Menendez-Pidal 1 1 Olewave, San Francisco, USA 2 Stanford University Ruchao Fan 4 4 Microsoft Chenfeng Miao 5 5 PingAn Technology Chanwoo Kim 6 6 Korea University Bhiksha Raj 1,7 , Rita Singh 1,7 7 Carnegie Mellon University · 2025

Read OleSpeech-IV: A Large-Scale Multispeaker and Multilingual Conversational Speech Dataset with Diverse Topics by Wei Chu 1 , Yuanzhe Dong 1,2 , Ke Tan 1 , Dong Han 1 , Xavier Menendez-Pidal 1 1 Olewave, San Francisco, USA 2 Stanford University Ruchao Fan 4 4 Microsoft Chenfeng Miao 5 5 PingAn Technology Chanwoo Kim 6 6 Korea University Bhiksha Raj 1,7 , Rita Singh 1,7 7 Carnegie Mellon University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04702.txt.

Introduction

Conversational speech in several languages fills a large dataset introduced by Wei Chu, lead author with nine co-authors, in a 2025 paper. OleSpeech-IV gathers audio from English-language podcasts, talk shows, teleconferences, and similar public spoken material. It is multispeaker and multilingual, with diverse topics.

Speaker names, turns, and transcripts came from human sources. A proprietary pipeline then refined them. Timestamps and confidence scores come from that pipeline as well. The dataset is Tier IV in the Olewave series. The authors have also released a subset for non-commercial research use, supporting work on speech recognition and speaker analysis.

Catalog record

CREATORWei Chu 1 , Yuanzhe Dong 1,2 , Ke Tan 1 , Dong Han 1 , Xavier Menendez-Pidal 1 1 Olewave, San Francisco, USA 2 Stanford University Ruchao Fan 4 4 Microsoft Chenfeng Miao 5 5 PingAn Technology Chanwoo Kim 6 6 Korea University Bhiksha Raj 1,7 , Rita Singh 1,7 7 Carnegie Mellon University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-04702

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OleSpeech-IV: A Large-Scale Multispeaker and Multilingual Conversational Speech Dataset with Diverse Topics Wei Chu 1 , Yuanzhe Dong 1,2 , Ke Tan 1 , Dong Han 1 , Xavier Menendez-Pidal 1 1 Olewave, San Francisco, USA 2 Stanford University Ruchao Fan 4 4 Microsoft Chenfeng Miao 5 5 PingAn Technology Chanwoo Kim 6 6 Korea University Bhiksha Raj 1,7 , Rita Singh 1,7 7 Carnegie Mellon University Abstract OleSpeech-IV dataset is a large-scale multispeaker and multilingual conversational speech dataset with diverse topics. The audio content comes from publicly-available English podcasts, talk shows, teleconferences, and other conversations. Speaker names, turns, and transcripts are human-sourced and refined by a proprietary pipeline, while additional information such as timestamps and confidence scores is derived from the pipeline. The IV denotes its position as Tier IV in the Olewave dataset series. In addition, we have open-sourced a subset, OleSpeech-IV-2025-EN-AR-100, for non-commercial research use. 1 Introduction The creation of a large-scale dataset typically involves two critical stages: curation and cleaning . Each stage presents unique challenges that must be addressed to ensure both the quality and utility of the resulting dataset. 1.1 Challenges in curating large-scale speech datasets Curating large-scale speech datasets is inherently complex. In this work, we aim to expand the discussion by providing a historical overview of existing large-scale speech resources, while also highlighting the motivation and initiative behind the development of OleSpeech-IV. 1.2 Challenges in cleaning large-scale speech datasets Speech data cleaning is particularly challenging due to the multifaceted nature of speech signals, which introduce complexities beyond those encountered in traditional text-based Natural Language Processing (NLP). While NLP deals with variations in language, vocabulary, and topic, speech processing must additionally contend with channel variations and speaker variations, each of which introduces unique difficulties. Channel variations refer to differences in the acoustic environment and recording conditions, which can significantly degrade the quality of speech data. These variations include: • Background noise: Environmental sounds, such as traffic, wind, or crowd noise, can obscure speech signals. • Reverberation: Echoes caused by recording in large or reflective spaces can distort the clarity of speech. • Microphone quality: Variations in recording devices can lead to differences in audio fidelity, frequency response, and signal-to-noise ratio (SNR). • Transmission artifacts: In telephony or streaming applications, compression, packet loss, or bandwidth limitations can introduce distortions. Speaker variations add another layer of complexity, as speech signals are highly dependent on individual characteristics, such as: • Accent and dialect: Regional or cultural differences in pronunciation and vocabulary can affect recognition accuracy. • Speaking style: Variations in speaking rate, pitch, volume, and articulation can make it difficult to generalize across speakers. • Emotional state: Stress, excitement, or fatigue can alter speech patterns and introduce variability. • Physiological factors: Age, gender, and vocal tract differences can influence the acoustic properties of speech. 2 OleSpeech-IV Datasets OleSpeech-IV dataset is the currently most comprehensive dataset offered by us. The ’IV’ means ’Tier IV’, and it is necessary for us to explain the Tier I, II, and III datasets, since Tier IV dataset is obtained through filtering these lower tier datasets, and applying more advanced data pipelines. Available in multiple languages and covering diverse topics, our datasets are organized into four tiers of pre-labeled offerings, each designed to meet varying levels of annotation depth and application requirements. 2.1 Tier I – Untranscribed Speech Datasets These datasets do not include spoken content transcripts but come with the following labels: Mandatory Metadata • Language labels (see Appendix: Languages for details) • Topic tags (see Appendix: Topics for details) Optional Signal Condition Tags • Signal-to-Noise Ratio (SNR) • Speech Percentage Ratio • … Optional Text-Based Tags • Summaries and keywords • Audio creation metadata • … Take English language for example, we currently offer over 1 million hours of audio at this tier and are ready to collect more on demand. 2.2 Tier II – Pseudo-Transcribed Speech Datasets These datasets include machine-generated transcripts with validation scores: Included Labels • All labels from Tier I Transcript Validation • Accurate onset and offset timings for each utterance • Confidence scores at both the word and utterance levels Take English language for example, we currently offer over 200,000 hours of audio at this tier and can scale further if needed. 2.3 Tier III – Human-in-the-Loop (HITL) Transcribed Speech Datasets These datasets contain transcripts created and validated by humans: Included Labels • All labels from Tier I Human-Generated Transcripts • Validation equivalent to Tier II standards Take English language for example, we currently offer over 40,000 hours of audio at this tier, with the ability to expand upon request. 2.4 Tier IV – Advanced-Labeled Speech Datasets These are the most comprehensive datasets with enriched annotations: Included Labels • All data and labels from Tier III Additional Advanced Labels • Speaker labels and timing information for conversational speech (see Appendix: Conversational Speech Datasets) • Translated transcripts of spoken content (optional) • Optional verbatim (word-for-word) transcripts (optional) • … • Custom labels as specified by clients We currently have over 5,000 hours of audio, and we name it as Olewave-IV-2025, which 2025 denotes the year. 2.4.1 OleSpeech-IV-2025-EN-AR-100 – for Non-Commercial Research Use The OleSpeech-IV-2025-EN-AR-100 dataset is a subset of the Olewave-IV-2025 collection. It contains 100 hours of English (EN) speech with accents from all regions (AR). The data is divided into training, development, and test sets with an 8:1:1 ratio. 3 Methodologies Olewave’s state-of-the-art and highly customizable speech data cleaning pipeline seamlessly integrates with its speech data collection pipeline, together forming the Olewave’s speech data curation pipeline. This end-to-end system is meticulously designed to deliver high-quality, consistent, and cost-effective speech datasets at scale, tailored for a wide range of downstream applications. Additionally, we are offering large-scale, cost-effective speech datasets curated through our pipeline, available in multiple languages and covering diverse topics. Figure 1 illustrates the streamlined workflow of our speech data cleaning pipeline, showcasing its ability to transform raw, unstructured speech data into polished, actionable datasets. At the core of this pipeline is Olewave’s signature speech-to-text alignment module, Olign , which enables accurate and scalable cleaning of speech data. Figure 1: Olewave’s speech data processing pipeline. The central component is the speech-to-text alignment module Olign , which can be accessed through the Olign SDK or deployed on-premises. This innovative pipeline takes raw speech recordings, corresponding transcripts, and optional metadata as inputs, and robustly cleans noisy transcriptions and labels. It delivers validated speaker labels and transcripts, and rich features such as reliable word-level timestamps and confidence scores. By leveraging advanced alignment techniques, our pipeline ensures the highest levels of accuracy and usability, making it an indispensable tool for building high-performance but cost-efficient speech-based systems. Our pipeline offers several advantages: • Effective: It produces validated speaker labels and transcriptions, accurate word timestamps and not-overconfident confidence scores compared to tools like Whisper and other open-source solutions. • Robustness: It handles improvised conversational speech, including scenarios with speaker overlap and transcripts containing ASR errors. • Extensible: It leverages optional metadata and can be upgraded to incorporate additional modality for increased label reliability. The pipeline also supports plug-and-play integration of various Olewave’s or client’s own label tagging models (e.g., emotion, semantics, …). • Efficiency: It runs quickly and is cost-effective, as

Read the complete text