FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System
Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu Feng-Long Xie, Xu Tang, Yao Hu Super Intelligence Team, Xiaohongshu Inc. · 2026
Read FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System by Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu Feng-Long Xie, Xu Tang, Yao Hu Super Intelligence Team, Xiaohongshu Inc. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10420.txt.
Introduction
Kaituo Xu and co-authors present FireRedASR2S, an industrial all-in-one speech recognition system with four modules in one pipeline. Alongside recognition itself, it handles voice activity detection, spoken language identification, and punctuation prediction, each at leading benchmark levels. The recognition module comes in large and smaller variants and covers English, code-switching, Mandarin, and many Chinese dialects and accents, for speech and singing alike. The other modules are light yet strong, with language identification spanning over a hundred languages. Model weights and code are released.
Catalog record
| CREATOR | Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu Feng-Long Xie, Xu Tang, Yao Hu Super Intelligence Team, Xiaohongshu Inc. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44627 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu Feng-Long Xie, Xu Tang, Yao Hu Super Intelligence Team, Xiaohongshu Inc. Abstract We present FireRedASR2S, a state-of-the-art industrial-grade all-in-one automatic speech recognition (ASR) system. It integrates four modules in a unified pipeline: ASR, Voice Activity Detection (VAD), Spoken Language Identification (LID), and Punctuation Prediction (Punc). All modules achieve SOTA performance on the evaluated benchmarks: FireRedASR2 : An ASR module with two variants, FireRedASR2-LLM (8B+ parameters) and FireRedASR2-AED (1B+ parameters), supporting speech and singing transcription for Mandarin, Chinese dialects and accents, English, and code-switching. Compared to FireRedASR, FireRedASR2 delivers improved recognition accuracy and broader dialect and accent coverage. FireRedASR2-LLM achieves 2.89% average CER on 4 public Mandarin benchmarks and 11.55% on 19 public Chinese dialects and accents benchmarks, outperforming competitive baselines including Doubao-ASR, Qwen3-ASR, and Fun-ASR. FireRedVAD : An ultra-lightweight module (0.6M parameters) based on the Deep Feedforward Sequential Memory Network (DFSMN), supporting streaming VAD, non-streaming VAD, and multi-label VAD (mVAD). On the FLEURS-VAD-102 benchmark, it achieves 97.57% frame-level F1 and 99.60% AUC-ROC, outperforming Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD. FireRedLID : An Encoder-Decoder LID module supporting 100+ languages and 20+ Chinese dialects and accents. On FLEURS (82 languages), it achieves 97.18% utterance-level accuracy, outperforming Whisper and SpeechBrain. FireRedPunc : A BERT-style punctuation prediction module for Chinese and English. On multi-domain benchmarks, it achieves 78.90% average F1, outperforming FunASR-Punc (62.77%). To advance research in speech processing, we release model weights and code at https://github.com/FireRedTeam/FireRedASR2S . 1 Introduction Automatic speech recognition (ASR) has advanced rapidly with end-to-end modeling, large-scale training, and the integration of large language models (LLMs) [ 49 , 36 , 4 , 1 , 42 , 29 , 3 , 10 , 11 , 48 , 34 , 26 , 46 , 31 , 50 , 9 , 24 , 20 , 27 ] . However, practical deployment in real-world applications typically requires more than a standalone ASR model. Real-world audio often contains long-form recordings, silence and non-speech regions, background music, singing, multilingual speech, and Chinese dialects and accents (hereafter referred to as dialects for brevity). To deliver reliable transcription in such conditions, a complete pipeline is needed, including voice activity detection (VAD) for segmentation, spoken language identification (LID) for routing and tagging, and punctuation prediction (Punc) for readable outputs. In practice, many systems are built by assembling modules from heterogeneous sources (e.g., separate VAD/LID/ASR/Punc toolkits or cloud services). Such pipelines frequently suffer from inconsistent interfaces, limited reproducibility, and complex error propagation. Moreover, some components rely on weak or indirect supervision (e.g., VAD trained from ASR forced alignment), which may degrade robustness under challenging acoustic conditions. These limitations motivate an open-source, industrial-grade all-in-one ASR system with strong performance, clear modularization, and comprehensive evaluation. In this technical report, we present FireRedASR2S , a state-of-the-art (SOTA), all-in-one ASR system integrating four modules: FireRedASR2 for ASR, FireRedVAD for VAD and multi-label VAD (mVAD), FireRedLID for multilingual and dialect LID, and FireRedPunc for punctuation prediction. The suffix 2S denotes the 2 nd-generation FireRedASR, expanded into an all-in-one ASR S ystem. FireRedASR2S provides a unified pipeline from waveform to structured transcription outputs, while allowing each module to be deployed independently. FireRedASR2 builds upon our previous FireRedASR [ 49 ] models with minimal architectural changes. Compared to FireRedASR, FireRedASR2 improves recognition accuracy and expands coverage to a broader range of Chinese dialects, primarily by scaling supervised training data to approximately 200k hours with broader domain, language, and dialect diversity. FireRedVAD is trained on thousands of hours of high-quality human-annotated acoustic event data, providing reliable segmentation under diverse acoustic conditions. FireRedLID is implemented as an Encoder-Decoder-based [ 6 , 8 ] model initialized from the FireRedASR2-AED encoder and performs hierarchical language and dialect prediction. FireRedPunc adopts a BERT-style encoder [ 17 ] initialized from LERT [ 14 ] and is trained on large-scale multi-domain Chinese and English corpora. Our main contributions are: • All-in-one open-source system : We release an integrated ASR pipeline with unified interfaces and modular deployment. • Improved ASR accuracy and broader dialect coverage : Building upon FireRedASR, FireRedASR2 improves recognition accuracy and expands support for Chinese dialects, achieving strong results on 24 public test sets. • Robust segmentation from human-labeled events : FireRedVAD provides strong multilingual VAD performance and is trained using high-quality human-annotated event data rather than forced-alignment-derived supervision. • Hierarchical multilingual and dialect LID : FireRedLID supports 100+ languages and 20+ Chinese dialects with a compact two-token decoding formulation. • Effective punctuation prediction : FireRedPunc achieves strong results on multi-domain Chinese and English punctuation benchmarks. The remainder of this report is organized as follows. Section 2 presents the system overview. Sections 3 , 4 , 5 and 6 describe each module. Section 7 reports evaluation results. Section 8 discusses key design choices and limitations, and Section 9 concludes the report. 2 FireRedASR2S: System Overview FireRedASR2S is an industrial-grade, all-in-one ASR system that integrates four modules—FireRedVAD ( Section 4 ), FireRedLID ( Section 5 ), FireRedASR2 ( Section 3 ), and FireRedPunc ( Section 6 )—into a unified pipeline. The system is designed in a modular manner: each component can be used independently, while the default configuration forms an end-to-end transcription pipeline that handles diverse acoustic conditions (speech, singing, music, and non-speech) as well as multilingual and Chinese dialect scenarios, and produces structured outputs including punctuated text, timestamps, confidence scores, and language labels. Figure 1: Overview of FireRedASR2S. The input waveform is processed sequentially by FireRedVAD (VAD), FireRedLID (LID), FireRedASR2 (ASR), and FireRedPunc (Punc) to produce structured transcription outputs, including punctuated text, timestamps, confidence scores, and language labels. Pipeline : As illustrated in Figure 1 , FireRedASR2S processes an input waveform through four stages. First, FireRedVAD detects voice segments on the original audio timeline and filters out non-voice regions to improve robustness on long-form audio. Second, FireRedLID predicts an utterance-level language label for each detected segment and, when applicable, a Chinese dialect label. Third, FireRedASR2 transcribes each segment into text and returns an ASR confidence score; when using FireRedASR2-AED, it additionally provides token- and word-level timestamps. Finally, FireRedPunc restores punctuation for the ASR output to improve readability and downstream usability. Structured outputs : FireRedASR2S returns structured outputs containing (1) the final transcription text, (2) a list of sentence-level segments with start/end timestamps, recognized text, ASR confidence, and optional language labels with confidence, and (3) VAD segmentation results. When ASR timestamping is enabled, the system can further derive sentence-level timestamps by leveraging punctuation prediction. All timestamps are reported on the original waveform timeline. Modularity : Although FireRedASR2S is designed as an end-to-end pipeline, each module can be deployed as a standalone component (e.g., VAD-only segmentation, LID-only routing, ASR on pre-segmented audio, or punctuation on plain text). This modular design enables flexible deployment and independent iteration of each component. 3 FireRedASR2: Automatic Speech Recognition We summarize the key components of FireRedASR2 here and highlight the incremental updates. For detailed specifications of the Conformer and Transformer blocks, the Encoder-Adapter-LLM training procedure, and the optimization strategies, we refer readers to the FireRedASR technical report [ 49 ] . FireRedASR2 comprises two variants: FireRedASR2-AED and FireRedASR2-LLM. FireRedASR2-AED follows the conventional Attention-based Encoder-Decoder architecture [ 6 ,