MoonCast: High-Quality Zero-Shot Podcast Generation

Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li · 2025

Read MoonCast: High-Quality Zero-Shot Podcast Generation by Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14345.txt.

Introduction

Zeqian Ju and co-authors introduce MoonCast, a system that turns plain text into full podcast-style audio with voices it has never heard before. Speech tools handle short, single-speaker clips well, but podcasts run for minutes, involve several speakers, and sound loose and spoken rather than read aloud. Those traits make podcasts a much harder target.

MoonCast pairs a long-context speech model, trained on long recordings, with a script writer that adds the small spontaneous touches of real talk. In tests it beat rival systems, with the clearest gains in how natural and flowing the speech felt. Input can be a story, a report, or a web page. The work points toward audio shows made on demand from written sources.

Catalog record

CREATORZeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoonCast: High-Quality Zero-Shot Podcast Generation Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li Abstract Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize natural podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To generate long audio, we adopt a long-context language model-based audio modeling approach utilizing large-scale long-context speech data. To enhance spontaneity, we utilize a podcast generation module to generate scripts with spontaneous details, which have been empirically shown to be as crucial as the text-to-speech modeling itself. Experiments demonstrate that MoonCast outperforms baselines, with particularly notable improvements in spontaneity and coherence. https://mooncastdemo.github.io 1 Introduction In recent years, significant advancements in large language models (LLMs) and speech codec technologies have substantially enhanced the performance of text-to-speech (TTS) synthesis, improving its naturalness, expressiveness, and tonal richness. These advancements have led to widespread adoption in industries such as customer service and short video production. As TTS technology continues to evolve, there is a growing demand for generating long-duration podcast content from text-only sources, such as news, technical reports, and stories. Podcast speech requires not only extended audio lengths but also highly spontaneous expressions, often involving multiple speakers and dynamic interaction. The limitations of previous efforts in generating high-quality podcasts stem from two key challenges. First, long-context audio modeling presents significant challenges. Podcasts typically span over several minutes, featuring numerous utterances from multiple speakers. This requires the system to generate not only realistic individual speech but also seamless transitions between utterances. Furthermore, high-quality podcast generation must account for the contextual coherence of each speaker, encompassing aspects such as prosody and timbre. Second, podcasts are highly spontaneous, typified by the fluid and casual flow of human conversation. They often contain human-like details, including filler words such as ”um,” occasional hesitations, and minor mistakes. In the multiple-speaker scenario, the system must also account for the interactions between speakers. However, the TTS community has largely focused on improving short individual utterance generation, with limited efforts exploring long-context, spontaneous scenarios. Specifically, academic research has primarily focused on short conversational speech ( Nguyen et al., 2023 ; Mitsui et al., 2023 ) , but these efforts often face difficulties when applied to longer, more complex podcast scenarios, particularly in capturing spontaneity and naturalness within inter-sentence interactions. Recently, industrial solutions like NotebookLM 1 1 1 https://notebooklm.google.com/ have emerged to facilitate podcast creation from various knowledge sources. However, these solutions often lack transparency in their technical details, limiting their adaptability. To overcome these limitations, we propose a high-quality podcast generation system MoonCast. On one hand, inspired by the remarkable success of LLMs, we adopt a language model-based approach for speech modeling, leveraging its powerful in-context learning capabilities, scalability, and, notably, its ability to effectively manage long contexts. We gather large-scale long-context speech data from diverse sources and scale both model parameters and context length to enhance long-context capability. We employ a curriculum learning approach to progressively equip the model with zero-shot, long-context, and spontaneous speech generation capabilities in three stages. Furthermore, we employ a chunk-wise autoregressive speech detokenizer to ensure effective inference in the long-context scenario. On the other hand, to further enhance the spontaneity of the generated podcast, we use an LLM-powered podcast script generation module to transform complex information from input text-only sources into a conversational podcast script format. Building on our empirical finding that the spontaneity of the generated audio is significantly influenced not only by text-to-speech modeling but also by the podcast script text itself, we provide specific examples and guidance to facilitate the LLM in generating spontaneous details in the script. This helps bridge the gap between the input text used for audio modeling during inference and training, thereby activating the model’s spontaneous generation capabilities developed through large-scale multi-domain speech training. With this design, we can generate a spontaneous podcast of up to ten minutes from text-only input sources in a comprehensive manner. The experimental results show that the proposed system consistently outperforms the concatenate baselines in terms of intelligibility, coherence, and spontaneity for multi-lingual podcast generation. Specifically, MoonCast achieves subjective evaluation improvements of 0.40 in spontaneity, 0.33 in coherence, 0.05 in intelligibility, 0.10 in speech quality and 0.20 in speaker similarity for Chinese, and 0.85 in spontaneity, 0.70 in coherence, and 0.08 in intelligibility for English podcast generation. We invite readers to listen to audio samples at https://mooncastdemo.github.io for a more intuitive experience. We open-source MoonCast, including the prompts 2 2 2 Refer to Appendix C for script generation and the audio modeling module 3 3 3 https://github.com/jzq2000/MoonCast for speech generation, to support future research. 2 Background 2.1 Zero-Shot TTS Zero-shot text-to-Speech synthesis aims to synthesize speech that mimics the characteristics of a target speaker using only a brief prompt speech, without requiring additional fine-tuning ( Shen et al., 2023 ; Ju et al., 2024 ; Chen et al., 2024 ) . Recent advancements in zero-shot TTS can be broadly categorized into two types based on the representation: discrete code or continuous latent. For the code-based method, VALL-E ( Wang et al., 2023 ) utilizes neural codec language models and achieves high fidelity in zero-shot TTS. Seed-TTS ( Anastassiou et al., 2024 ) and CosyVoice ( Du et al., 2024a ; Du et al., 2024b ) leverage a single semantic codebook to reduce the difficulty on discrete code generation. Also, discrete diffusion can be leveraged to enable code generation in a non-autoregressive manner ( Borsos et al., 2023 ; Ju et al., 2024 ) . For the latent-based method, Naturalspeech 2 ( Shen et al., 2023 ) leverages latent diffusion to predict the latent of a speech codec conditioned on a short prompt speech. VoiceBox ( Le et al., 2024 ) utilizes flow matching to model Mel-spectrogram of a speech. In this paper, we adopt a pipeline that combines both the code-based and latent-based methods. Rather than focusing on speech that contains only a single speaker, we consider zero-shot two-speaker podcast speech generation. 2.2 Dialogue and Conversation Generation Most works in zero-shot TTS focus on the speech synthesis of one speaker. However, many scenarios such as dialogues, conversations, and podcasts require the TTS model to be able to synthesize speech with multi-speaker at the same time. Generating spoken dialogues ( Schuller et al., 2013 ) that include natural turn-taking, laughter, and other paralinguistic cues ( Zhang et al., 2020 ; Adiwardana et al., 2020 ; Lewis et al., 2020 ; Xu, 2021 ) is non-trivial. Recent work has explored various approaches to address this challenge. DGSLM ( Nguyen et al.,

Read the complete text