Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization
Alexander, Ivan, Jan, Shinji, Lukáš, Samuele · 2026
Read Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization by Alexander, Ivan, Jan, Shinji, Lukáš, Samuele (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15442.txt.
Introduction
Alexander Polok and five others study how synthetic conversational data shapes multi-talker speech recognition and speaker diarization. Fake data makes up for scarce chat tapes, but simulation choices matter. Their FastMSS simulator lets them vary turn-taking, source domain, acoustic augmentation, and mixing for leading systems. Optimal recipes prove task-linked: more speech overlap helps ID but harms speaker splits, while broad source diversity beats exact domain matching. Fake-only training nears real-data baselines, and joint training a lot beats real-only training.
Catalog record
| CREATOR | Alexander, Ivan, Jan, Shinji, Lukáš, Samuele |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25376 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15442
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization Alexander, Ivan, Jan, Shinji, Lukáš, Samuele Abstract Recent breakthroughs in multi-talker ASR (MT-ASR) and speaker diarization (SD) rely on synthetic data to mitigate the scarcity of large-scale conversational recordings, yet the impact of specific simulation choices remains poorly understood. To mind the gap between simulated mixtures and real-world interactions, we present a study of synthetic data generation for leading MT-ASR (DiCoW) and SD (Sortformer) systems. By introducing FastMSS, a highly efficient open-source simulator, we analyze turn-taking dynamics, source domain, acoustic augmentation, and data mixing strategies. Our findings reveal that optimal simulation recipes are highly task-dependent: increasing speech overlap benefits ASR but degrades diarization. Furthermore, broad source diversity consistently outperforms exact domain matching. Ultimately, synthetic-only training approaches real-data baselines, and combining simulated data with real recordings yields substantial gains over real-only training across both tasks. Polok Medennikov Černocký Watanabe Burget Cornell \bstctlcite IEEEexample:BSTcontrol keywords multi-talker speech processing, speaker diarization, target-speaker speech recognition, synthetic data † † address: 1 Brno University of Technology, Czechia 2 Carnegie Mellon University, USA 3 NVIDIA, USA † † email: [email protected] 1 Introduction Multi-talker conversational speech processing is undergoing a rapid transformation, driven largely by the shift from highly specialized pipelines to less data-hungry methods built on pretrained foundation models [ 48 , 9 , 47 , 63 ] . By leveraging massive amounts of single-speaker or self-supervised data, these foundational backbones can be effectively fine-tuned for complex downstream tasks, including multi-talker automatic speech recognition (MT-ASR) [ 23 , 31 ] , speaker diarization [ 17 , 42 ] , or spoken dialog models [ 54 , 15 ] . Despite this architectural progress, a persistent bottleneck remains: the severe scarcity of real, large-scale conversational training data. Real meeting corpora, such as AMI [ 34 ] , NOTSOFAR-1 [ 55 , 1 ] , MCoREC [ 37 ] , and CHiME-5/6 [ 57 , 3 ] provide spontaneous conversational data but are limited in scale, typically comprising only tens to a few hundreds of hours. Furthermore, annotating real conversational data is prohibitively expensive, time-consuming, and frequently complicated by privacy concerns. To fully unlock the capabilities of foundational models, the field has increasingly relied on synthetic data generation—ranging from heuristic signal-mixing [ 12 , 41 ] to text-to-speech (TTS) pipelines [ 13 , 8 , 30 ] . Despite this widespread adoption, the community’s approach to simulation remains heavily fragmented, leaving three crucial gaps in our understanding: First, existing simulation strategies are almost exclusively designed and evaluated for a single specific task [ 29 , 7 , 59 , 36 , 2 , 18 , 13 ] , leaving it unclear if the same synthetic data properties benefit complementary multi-talker tasks, such as multi-talker ASR (MT-ASR) and diarization. Second, studies often rely on a single source of seed data [ 10 , 14 ] , leaving the impact of source domain mismatch and the generalizability to spontaneous, “in-the-wild” interactions largely unexplored. Finally, it remains unclear to what extent synthetic data can substitute for real conversational recordings, especially for data-hungry tasks such as end-to-end diarization [ 16 ] . Nor is there consensus on the best strategy for combining the two: training jointly, fine-tuning on real data after synthetic pre-training, or relying on synthetic data alone. To address these gaps, we present a comprehensive study systematically varying turn-taking dynamics, source domain, acoustic augmentation, and data mixing strategies across two leading models: Sortformer [ 42 ] for diarization and DiCoW [ 45 ] for MT-ASR. To enable our experiments and ensure full reproducibility, we release FastMSS, 1 1 1 https://github.com/popcornell/FastMSS a fully open-source conversation simulation toolkit that supports flexible turn-taking modeling and scalable generation of multi-talker long-form synthetic conversations. Our main findings are: (i) optimal simulation strategies are task-dependent: boosting overlap improves MT-ASR but degrades diarization; (ii) mixing seed utterances from diverse domains outperforms using a single in-domain source, even when the latter is matched to the test set; (iii) acoustic augmentation is critical for diarization but not for MT-ASR; and (iv) synthetic data provides a highly competitive alternative when in-domain recordings are unavailable: for diarization, synthetic-only training performs close to a strong reference model, while combining synthetic and real data yields our best overall performance. 2 Multi-Talker Speech Processing 2.1 Multi-Talker ASR: DiCoW Multi-talker ASR (MT-ASR) has traditionally been tackled through modular separation-based pipelines [ 38 ] , end-to-end architectures like Serialized Output Training (SOT) [ 25 ] , or target-speaker conditioning [ 26 ] . The latter paradigm has seen rapid advancement with the introduction of diarization conditioning, a data-efficient mechanism for adapting pretrained single-speaker ASR models to overlapping, multi-talker scenarios [ 46 ] . We employ Diarization-Conditioned Whisper (DiCoW) [ 45 , 44 ] , which achieves strong MT-ASR performance on benchmarks like AMI [ 34 ] and LibriMix [ 14 ] by injecting frame-level diarization cues into a Whisper [ 48 ] encoder. 2.2 Speaker Diarization: Sortformer End-to-end neural diarization [ 16 ] is a promising paradigm thanks to its native handling of overlapping speech and architectural simplicity, but it is inherently data-hungry making it an ideal candidate to study the impact of synthetic data properties. In contrast, clustering-based [ 6 ] and hybrid systems [ 27 , 5 , 43 , 17 ] rely on pretrained speaker embeddings and non-differentiable clustering steps, which may obscure the direct effect of synthetic data simulation properties. We therefore employ Sortformer [ 42 , 35 ] , an EEND-based encoder-only model that resolves speaker permutations via Sort Loss in arrival-time order, eliminating attractors [ 19 , 21 ] and clustering entirely. Built on the self-supervised NEST-FastConformer backbone [ 22 , 49 ] , it achieves highly competitive diarization across multiple benchmarks [ 42 ] . 3 Multi-Speaker Conversation Simulation To enable controlled and fast experimentation along the axes described above, we developed FastMSS, an open-source multi-speaker conversation simulator focused on scalable generation with native Lhotse [ 62 ] integration. Given a set of single-speaker utterances from a source dataset, FastMSS generates multi-talker mixtures with configurable turn-taking dynamics and acoustic conditions including room impulse responses via Pyroomacoustics [ 51 ] , support for multiple noise sources and per-utterance gain variation. Word-level alignments, when available, are used to split source utterances at pause boundaries for more realistic turn-taking. Figure 1 demonstrates FastMSS’s scalability against two widely used simulators, MMS-MSG [ 12 ] and the NeMo multi-speaker data simulator [ 41 ] , benchmarked on identical hardware (4 \times AMD EPYC 7742, 256 CPUs, 1 TB RAM) generating 6,000 two-minute meetings from LibriSpeech train-clean-100 without reverberation or noise. FastMSS scales efficiently from single up to 32 processes, generating 1,000 hours of annotated multi-talker audio in under five minutes. Instead, MMS-MSG adopts a producer–consumer design in which examples are generated in parallel but audio is written sequentially, creating an I/O bottleneck that caps throughput beyond eight workers. NeMo is also process-based, with higher per-conversation overhead from PyTorch tensor operations. 3.1 Turn-Taking Model FastMSS Turn-Taking (TT) model simply extends the two-speaker HMM-based approach of Yamashita et al. [ 58 ] to an arbitrary number of speakers. Four utterance transition types are modeled: turn hold (TH, same speaker continues after a