Fish Audio S2 Technical Report

Fish Audio Team · 2026

Read Fish Audio S2 Technical Report by Fish Audio Team (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08823.txt.

Introduction

Shijia Liao and colleagues present Fish Audio S2, an open text-to-speech system that follows written instructions. It handles many speakers and many turns, trained through a multi-stage recipe over captioning, voice-quality checks, and reward modeling. The streaming engine answers fast, with a real-time factor well under one and first audio in under a tenth of a second.

Catalog record

CREATORFish Audio Team
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55262
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fish Audio S2 Technical Report Fish Audio Team Abstract We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ​ ms 100\,\text{ms} . Our code and weights are available on GitHub and Hugging Face . We highly encourage readers to visit https://fish.audio to try custom voices. https://github.com/fishaudio/fish-speech https://huggingface.co/fishaudio/s2-pro https://fish.audio 1 Introduction Figure 1: Fish Audio S2 is a multilingual, controllable, and expressive TTS system supporting long-form, multi-speaker, multi-turn generation with ultra-low TTFA and RTF. High-quality, controllable text-to-speech (TTS) has become increasingly important in modern AI systems, enabling scalable audio content creation and natural conversational experiences across applications such as audiobook narration, video dubbing, and personalized chatbots. Recent progress in TTS has been driven by large-scale models ( Zhang et al. 2025 ; Du et al. 2025 ; Li et al. 2026 ; Hu et al. 2026 ) . Many of these works follow a two-stage paradigm: conditioned on text, the model first produces high-level discrete speech tokens, which are then decoded into the full waveform by a separate acoustic decoder ( Wang et al. 2023 ; Défossez et al. 2022 ; Kong et al. 2020 ; Anastassiou et al. 2024 ) . Alongside these architectural innovations, the success of large-scale TTS relies heavily on robust data curation. Recent efforts have introduced sophisticated pipelines for cleaning speech corpora and annotating paralinguistic features ( Cheng et al. 2025 ; Yang et al. 2025 ) . However, generating fine-grained natural-language instructions for vocal features at scale remains a major bottleneck. From a training perspective, although reinforcement learning (RL) methods such as Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) , Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) and Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) have become standard for improving model behavior in the large language model (LLM) domain ( Guo et al. 2025 ; Agarwal et al. 2025 ) , their adoption in TTS remains limited. In this report, we present Fish Audio S2, which retains the decoder-only Transformer backbone and RVQ-based audio codec of Fish Audio S1 ( Liao et al. 2024b ) . We extends it with a unified data curation and RL alignment framework to improve controllability, naturalness, and robustness in speech generation. Specifically, we introduce two key technical innovations: • Multi-Purpose Data Pipeline. We build a data pipeline with a speech quality assessment model and a rich-transcription ASR model to filter and annotate large-scale audio data for TTS pre-training. The same models are then directly reused as reward signals for RL alignment, eliminating distribution mismatch between the two stages. • Multi-Reward RL Alignment. We implement a variant of GRPO that jointly optimizes semantic accuracy, acoustic quality, and speaker similarity, ensuring a balance between expressiveness and robustness. These innovations directly enable three major functional breakthroughs: • Enhanced Instruction Following. Fish Audio S2 exhibits superior adherence to natural language instructions. It enables broad and fine-grained control over speech generation through free-form textual descriptions. • Native Multi-Speaker and Multi-Turn Generation. The model can natively generate complex, interleaved dialogues involving multiple distinct speakers in a single pass, capturing the dynamics of natural conversation. • Stable Long-Form Synthesis. The system supports the generation of coherent and continuous audio, maintaining stability and consistency over extended durations. To evaluate our model, we conduct extensive experiments along two complementary tracks: (i) objective evaluation and (ii) LLM-as-a-Judge assessments of higher-level capabilities. For intelligibility, content accuracy, long-form, and multilingual performance, we report Word Error Rate (WER), Character Error Rate (CER), and speaker similarity on widely used benchmarks, including Seed-TTS-Eval ( Anastassiou et al. 2024 ) , MiniMax Multilingual Testset ( Zhang et al. 2025 ) , CosyVoice3-Eval ( Du et al. 2025 ) , Long-TTS-Eval ( Wang et al. 2025a ) . Across public benchmarks, Fish Audio S2 shows consistently strong objective performance, achieving leading results on Seed-TTS benchmark while maintaining robust multilingual intelligibility and speaker similarity on both the MiniMax Multilingual Testset and CV3-Eval. To assess higher-level capabilities such as instruction following and human-likeness, we further employ the Audio Turing Test ( Wang et al. 2025b ) and Emergent TTS Eval ( Manku et al. 2025 ) . On the Audio Turing Test, Fish Audio S2 achieves a posterior mean of 0.483, which further improves to 0.515 with instruction rewriting. On Emergent TTS Eval, it reaches an overall win rate of 81.88% against the baseline, further supporting its strong instruction-following capability. Furthermore, to address the lack of dedicated benchmarks for fine-grained control, we introduce a novel evaluation benchmark, the Fish Audio Instruction Benchmark, which systematically evaluates models’ inline tag-following performance across English and Chinese. On the Fish Audio Instruction Benchmark, Fish Audio S2 achieves an overall tag-activation rate of 93.3% and an overall quality score of 4.51/5.0 across English and Chinese, as evaluated by Gemini 3 Pro. To accelerate research and lower the barrier to high-quality TTS development, we publicly release our model weights, fine-tuning code, and the SGLang-based inference engine on GitHub and Hugging Face . We also highly encourage readers to explore interactive demos at our official site https://fish.audio/ . The remainder of this paper is organized as follows: Section 2 details the model architecture; Section 3 describes the data curation pipeline; Section 4 presents the pre-training and RL-based post-training; Section 5 instrodces our inference engine and its performance, Section 6 presents the experimental setup and comprehensive evaluation results; and finally, Section 7 concludes with a discussion on limitations and future directions. 2 Architecture Figure 2: Fish Audio S2 architecture. 2.1 Audio Tokenizer Our audio tokenizer is built upon the architecture of the Descript Audio Codec (DAC) ( Kumar et al. 2023 ) , optimized for high-fidelity, real-time streaming at a 44.1\,\text{kHz} sampling rate. The model employs a hierarchical Residual Vector Quantization (RVQ) strategy utilizing N codebooks ( N{=}10 in our model): the primary codebook serves as the semantic codebook, while the remaining nine capture progressively finer-grained acoustic details. Streaming Architecture. To adapt the vanilla DAC for low-latency TTS tasks, we introduce several key modifications to the encoder and decoder structures: • Causal Convolutions. We refactor the model to be strictly causal by replacing standard convolutions with masked causal convolutions. This ensures the generation process depends solely on past context, enabling low-latency streaming capabilities. • Transformer Bottleneck. Following the design of Mimi ( Défossez et al. 2024 ) , we integrate causal sliding-window Transformer blocks both before and after the RVQ layers. By restricting attention to a fixed-size window, this mechanism models long-range dependencies with bounded memory usage, preventing out-of-memory issues during long-form inference. • Extended Downsampling. . The encoder extends the standard DAC encoder ( 512\times ) with additional ConvNeXt V2 ( Woo et al. 2023 ) layers ( 4\times ), achieving a total downsampling ratio of 2048 and a compact frame

Read the complete text