LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization
2025
Read LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23320.txt.
Introduction
Máté Gedeon and Péter Mihajlik built LibriConvo, a simulated speech dataset for systems that tell speakers apart and turn speech into text. The corpus is built from read-aloud book speech, reshaped into dialogues that sound like real talk. Their 2025 pipeline builds on an earlier speaker-aware simulation method and adds careful timing control.
To keep the talk natural, they took timing patterns from real phone calls and squeezed long pauses. They grouped clips by book, so nearby lines stay on topic. They also picked room echoes that fit the scene. The set holds more than 240 hours in 1,496 dialogues with 830 speakers, split so no speaker appears twice. In tests, one modern system cut speaker labeling errors to about eleven percent, far below the older pipeline. A tuned speech model reached word error rates near seven percent, beating a large general model used untuned.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23320
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization Abstract We introduce LibriConvo, a simulated multi-speaker conversational dataset based on speaker-aware conversation simulation (SASC), designed to support training and evaluation of speaker diarization and automatic speech recognition (ASR) systems. Unlike prior resources that mostly rely on semantically disconnected utterances and implausible temporal gaps, LibriConvo ensures semantic coherence and realistic conversational timing. Our pipeline leverages CallHome with external VAD for reliable boundaries, applies compression to reduce unnaturally long silences, and organizes LibriTTS utterances by book to maintain contextual consistency. Acoustic realism is enhanced via a novel room impulse response selection procedure that ranks speaker-microphone configurations by spatial plausibility, balancing realism and diversity. The dataset comprises 240.1 hours across 1,496 dialogues with 830 unique speakers, split in a speaker-disjoint manner for robust evaluation. Baselines show that the sortformer model outperforms the pyannote pipeline in diarization, while a fine-tuned Fast Conformer-CTC XLarge with Serialized Output Training achieves 7.29% WER for ASR, surpassing zero-shot Whisper-large-v3. LibriConvo provides a valuable resource for advancing multi-speaker speech processing research with realistic conversational dynamics and controlled experimental conditions. Keywords: conversation simulation, conversational speech, speech dataset, speech recognition LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization Máté Gedeon ∗,† , Péter Mihajlik ∗ ∗ Dept. of Telecommunications and Artificial Intelligence, Budapest University of Technology and Economics, Hungary † Speechtex Ltd. [email protected], [email protected] Abstract content 1. Introduction Modern speech processing systems, such as end-to-end speaker diarization (EEND) and multi-speaker automatic speech recognition (ASR), require large amounts of annotated conversational data Watanabe et al. 2020 . However, collecting real multi-party conversations with precise speaker turn labels is costly and difficult. Consequently, research has turned to synthetic data generation as an alternative Fujita et al. 2019 . Synthetic mixtures of clean utterances allow precise ground-truth labels ( who speaks when and what ), and have become common in training diarization and ASR models Yu et al. 2016 ; Kanda et al. 2020 . For example, early EEND systems were pre-trained on simulated mixtures of two speakers due to the paucity of real conversational corpora. Such simulated datasets can significantly improve model robustness Landini et al. 2022a , but naive mixing of utterances does not fully capture conversational dynamics. In particular, simple mixtures often lack realistic turn-taking patterns and may not preserve consistent speaker identities throughout a dialogue. Recent work has therefore focused on more naturalistic conversation simulation. Yamashita et al. 2022 proposed a method that explicitly models turn-taking by defining different types of speaker transitions, producing synthetic dialogues whose silence and overlap statistics match real meetings. Landini et al. 2022b likewise used statistics of pause and overlap distributions drawn from actual conversations to generate speech segments that resemble real dialogues. Park et al. 2023 introduced a probabilistic property-aware simulator that dynamically controls silence and overlap amounts to closely match target distributions. Building on this line of work, our previous study introduced the Speaker-Aware Simulated Conversation (SASC) framework Gedeon and Mihajlik 2025 , which unified conversational dynamics into a single gap/overlap distribution, incorporated speaker-specific temporal variation, and modeled turn-taking using a Markov-chain process. The SASC framework achieved closer alignment with real conversational data across multiple intrinsic metrics, including gap statistics, pause-length correlation, and turn-taking entropy. However, that paper focused on theoretical evaluation, with no accompanying dataset released for broader community use. In this paper, we address this gap by introducing LibriConvo, an open-source dataset of speaker-aware simulated conversations built upon the LibriTTS corpus. Following the SASC methodology, LibriConvo transforms independent read-speech utterances into coherent, temporally realistic dialogues that emulate natural human interaction. The corpus comprises 240.1 hours of audio across 1,496 simulated dialogues involving 830 distinct speakers, partitioned into speaker-disjoint training, validation, and test sets. Designed to support research in speaker diarization, ASR, and conversational modeling, LibriConvo enables reproducible experimentation and standardized benchmarking under controlled yet realistic conversational conditions. Our main contributions are as follows: • We introduce LibriConvo, a synthetic conversational speech dataset designed for both ASR and speaker diarization. Each recording consists of multi-turn dialogues with ground-truth transcriptions and speaker turn annotations. • We propose a methodology for constructing conversations that preserves semantic consistency across speaker turns. • We present a strategy for selecting room impulse responses (RIRs) to better approximate realistic acoustic conditions. • We report baseline results for both EEND and ASR on the dataset. The dataset, comprising both audio recordings and complete metadata, is publicly available on Hugging Face 1 1 1 https://huggingface.co/gedeonmate/datasets in two versions. The first version is segmented into clips of up to 30 seconds to facilitate ASR training and evaluation, while the second preserves the full-length conversations without segmentation. The structure of the paper is as follows. Section 2 introduces the theoretical framework of the proposed methodology and its application to dataset generation. Section 3 presents the baseline diarization and ASR results. Finally, Section 4 concludes the paper by summarizing the findings and outlining directions for future work. 2. Methodology 2.1. Speaker-aware conversation simulation The speaker-aware conversation simulation (SASC) method ( Gedeon and Mihajlik 2025 ) generates multi-speaker dialogues with temporal, structural, and acoustic properties that are modeled based on real conversations. Conversational timing is represented by a unified distribution of gaps \delta , where \delta<0 indicates overlap, \delta\geq 0 indicates a pause, and the integral over the negative domain corresponds to the probability of overlap p_{\text{overlap}} . Instead of parametric or histogram-based approaches, kernel density estimation (KDE) is used to obtain smooth, continuous estimates of these gap distributions. For timing consistency, two mean pause distributions are defined: \hat{D}_{=} for same-speaker mean gaps (when no speaker change occurs between utterances) and \hat{D}_{\neq} for different-speaker mean gaps. For each speaker s , an initial base value is sampled from the appropriate distribution, while subsequent gaps are generated by adding a deviation: \delta_{n}=\begin{cases}\mu^{\text{same}}_{s}+v&\text{if }X_{n}=X_{n-1},\;v\sim V_{=},\\[4.0pt] \mu^{\text{diff}}_{s}+v&\text{if }X_{n}\neq X_{n-1},\;v\sim V_{\neq}.\end{cases} Here V_{=} and V_{\neq} are zero-mean speaker deviation distributions that preserve local temporal consistency across turns. Turn-taking is modeled by a first-order (generalizable to n -order) Markov chain with transition matrix P_{\mathrm{turn}} , which defines the probability of selecting the next speaker given the previous one. All speakers are placed within a single acoustic environment by sampling a room from the available RIRs and assigning distinct positions within that room. After all utterances are concatenated with their respective gaps and overlaps, optionally background noise n\sim\mathcal{N} is added and scaled according to a sampled signal-to-noise ratio r\sim\mathcal{R} to produce the final mixture. A compressed version of the procedure is shown in Algorithm 1 . Algorithm 1 Simplified speaker-aware conversation simulation 1: Select N_{\mathrm{spk}} speakers \mathcal{S}^{\prime} ; assign RIRs (same room, distinct positions) 2: Choose initial speaker X_{1} 3: for n=1\dots N_{\mathrm{u}} do 4: if n>1 then sample X_{n}\sim P_{\mathrm{turn}}(X_{n-1}) 5: Sample utterance u_{n}\in U_{X_{n}} , convolve with RIR \to y_{n} 6: if n=1 then set \delta=0 7: else if X_{n}=X_{n-1} then 8: if first gap for X_{n} then \mu^{\text{same}}_{s}\sim\hat{D}_{=} 9: \delta=\mu^{\text{same}}_{s}+v,\;v\sim V_{=} 10: else 11: if first gap for X_{n} then \mu^{\text{diff}}_{s}\sim\hat{D}_{\neq} 12: \delta=\mu^{\text{diff}}_{s}+v,\;v\sim V_{\neq} 13: Mix y_{n} into conversation with gap \delta 2.2. Dataset generation To implement the methodology, we first define the datasets and methods employed. 2.2.1. Statistics In their original work, Gedeon and Mihajlik 2025 noted that the annotations for Switchboard Godfrey