SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development
Minghan Wang, Ye Bai, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, Gholamreza Haffari · 2025
Read SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development by Minghan Wang, Ye Bai, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, Gholamreza Haffari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23848.txt.
Introduction
Minghan Wang and five co-authors present a toolkit for making speech dialogue data in two thousand twenty-five. Good speech data is key for speech language models, but past sources fall short. Human takes cost a lot and raise privacy fears. Fake takes often sound stiff and false. Their system aims to close that gap.
The toolkit builds data through steps like scene setup, script writing, and voice cloning. It adds tone cues so speech sounds more like real talk. A screen tool lets users check samples, and a batch mode makes many at once. Tests show quality near human takes at far less cost. The code and sample sets in English and Chinese are free.
Catalog record
| CREATOR | Minghan Wang, Ye Bai, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, Gholamreza Haffari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development Minghan Wang, Ye Bai, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, Gholamreza Haffari Abstract High-quality speech dialogue datasets are crucial for Speech-LLM development, yet existing acquisition methods face significant limitations. Human recordings incur high costs and privacy concerns, while synthetic approaches often lack conversational authenticity. To address these challenges, we introduce SpeechDialogueFactory , a production-ready framework for generating natural speech dialogues efficiently. Our solution employs a comprehensive pipeline including metadata generation, dialogue scripting, paralinguistic-enriched utterance simulation, and natural speech synthesis with voice cloning. Additionally, the system provides an interactive UI for detailed sample inspection and a high-throughput batch synthesis mode. Evaluations show that dialogues generated by our system achieve a quality comparable to human recordings while significantly reducing production costs. We release our work as an open-source toolkit 1 1 1 http://github.com/yuriak/SpeechDialogueFactory , alongside example datasets available in English 2 2 2 https://huggingface.co/datasets/minghanw/sdf_dataset_en and Chinese 3 3 3 https://huggingface.co/datasets/minghanw/sdf_dataset_zh , empowering researchers and developers in Speech-LLM research and development. Dialogue Generation Framework No Tuning Required Text Generation Speech Generation Emotion Awareness Multilingual Support Interactive UI PLACES Chen et al. (2023) ✓ ✓ ✗ ✗ ✗ ✗ CHATS Mitsui et al. (2023) ✓ ✗ ✓ ✗ ✗ ✗ Style-Talker Li et al. (2024) ✗ ✗ ✓ ✓ ✗ ✗ CSR-data Cornell et al. (2024) ✗ ✓ ✓ ✗ ✗ ✗ PSYDIAL Han et al. (2024) ✓ ✓ ✗ ✓ ✗ ✗ RefGPT Yang et al. (2023) ✓ ✓ ✗ ✗ ✗ ✗ SpeechDialogueFactory (ours) ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of representative dialogue generation frameworks from 6 perspectives. 1 Introduction The emergence of voice-based AI interfaces has accelerated with recent breakthroughs in large language models (LLMs), establishing Speech-LLMs ( Défossez et al., 2024 ; Chu et al., 2023 ; OpenAI et al., 2024 ) as essential components in next-generation human-computer interaction systems. Developing effective Speech-LLMs involves two crucial phases: cross-modal alignment through joint pretraining of text and speech units ( Zhang et al., 2024 ; Xie and Wu, 2024 ; Zeng et al., 2024 ) , followed by specialized training on speech dialogue datasets to develop conversational capabilities ( Zeng et al., 2024 ; Wang et al., 2024 ) . High-quality speech dialogue datasets are fundamental to this second phase, requiring both linguistically authentic dialogue content ( Zhan et al., 2023 ; Zhong et al., 2022 ) and naturalistic prosodic features ( Chen et al., 2024b ; Maiti et al., 2022 ) . However, current approaches to acquiring such datasets face significant limitations that hinder progress in the field ( Défossez et al., 2024 ; Wang et al., 2024 ) . Human-recorded dialogues present substantial practical challenges for dataset creation. While professional voice actor recordings can achieve high quality, they involve prohibitive costs that severely limit dataset size and diversity ( Lee et al., 2023 ; Zandie et al., 2021 ) . Conversely, spontaneous dialogues often contain elements unsuitable for training, including excessive disfluencies and unstructured responses that can negatively impact model performance ( Yang et al., 2022 ; Barker et al., 2018 ; Saito et al., 2022 ; Li et al., 2022 ; Liu et al., 2024 ) . Figure 1: Example dialogue created by SpeechDialogueFactory with comprehensive character metadata and paralinguistic annotations. Therefore, to effectively address the aforementioned challenges in data collection, researchers have increasingly turned to synthetic methods for generating dialogue data. However, current synthetic data generation frameworks still face two critical issues: ( i ) Insufficient naturalness of generated dialogues. There are distinct differences between conversational and written language styles ( Zhang et al., 2024 ; Wang et al., 2024 ) . Dialogues generated by LLMs, which typically excel at producing written text, often lack the natural rhythm, emotional expressiveness, and turn-taking dynamics characteristic of genuine human interactions. ( ii ) Lack of a unified, end-to-end generation framework. As shown in Table 1 , existing frameworks typically focus separately on either text or speech dialogue generation ( Mitsui et al., 2023 ; Han et al., 2024 ) . These methods often require extensive model fine-tuning ( Li et al., 2024 ) and usually do not incorporate essential features such as emotional expressiveness Chen et al. (2023) , multilingual support, or user-friendly interfaces. Consequently, these limitations make it impractical to build large-scale and comprehensive dialogue datasets. To overcome these limitations, we present SpeechDialogueFactory , a systematic framework to generate high-quality speech dialogues with flexible customization and control. Our work makes four key contributions: • Quality-Assured Generation Pipeline : An integrated end-to-end workflow combining structured content creation (metadata, scripting, simulation) with expressive speech synthesis, ensuring both linguistic authenticity and natural prosody through comprehensive paralinguistic annotations. • Production-Optimized Implementation : An efficient system with interactive UI and parallel batch processing capabilities, designed for both exploratory development and large-scale dataset creation. • Multilingual Dataset : Publicly released dialogue datasets in both English and Chinese, along with intermediate records and quality evaluation results to support further research. Our experimental evaluation also confirms that SDF generates dialogues matching the quality of professional recordings, while offering researchers and developers unprecedented flexibility and usability for creating custom speech dialogue datasets tailored to their specific needs. 2 Related Work Currently, numerous frameworks have been proposed in the field of conversational data generation. Among these, some approaches focus primarily on utterance-level generation Sahu et al. (2022) ; Chen et al. (2022) ; Rosenbaum et al. (2022) ; Aher et al. (2023) , while others aim to generate entire conversations. However, there is presently no unified framework capable of comprehensively generating conversations, from textual content to corresponding speech, while simultaneously supporting emotional expressiveness and multilingual capabilities. As summarized in Table 1 , frameworks such as Places , RefGPT , and psydial Chen et al. (2023) ; Yang et al. (2023) ; Han et al. (2024) predominantly concentrate on text-based conversation generation, whereas CHATS and Style-Talker Mitsui et al. (2023) ; Li et al. (2024) specialize in speech-based conversation generation. Although CSR-data Cornell et al. (2024) is capable of generating both textual and speech conversations, it lacks emotional expressiveness and multilingual support. Additionally, none of the aforementioned frameworks provide user interfaces designed for easy interaction by general users. Furthermore, each of these existing frameworks demonstrates unique strengths. For instance, the Places Chen et al. (2023) framework does not require external seed data; instead, it utilizes high-quality expert-written dialogues as prompts, structured according to the desired conversational output, allowing LLMs to directly generate multi-party conversations. Style-Talker Li et al. (2024) analyzes the acoustic features of the user’s input speech to synthesize responses that align closely with the user’s speaking style. Additionally, the CSR-data Cornell et al. (2024) framework uniquely integrates both textual and speech conversation generation, which is relatively rare among current approaches. Researchers leveraging this framework can produce high-quality synthetic dialogue data suitable for fine-tuning automatic speech recognition (ASR) models. 3 SpeechDialogueFactory Figure 2: The SpeechDialogueFactory pipeline with integrated quality control. The framework processes user-specified settings through three main stages: content generation (including metadata generation, dialogue scripting, and simulation), speech generation (speaker retrieval and TTS synthesis), and quality evaluation at both text and speech levels to filter low-quality outputs before proceeding to subsequent processing steps. 3.1