Mega-ASR: Towards In-the-wild 2 Speech Recognition via Scaling Up Real-world Acoustic Simulation

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao · 2026

Read Mega-ASR: Towards In-the-wild 2 Speech Recognition via Scaling Up Real-world Acoustic Simulation by Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19833.txt.

Introduction

Xie and colleagues present Mega-ASR, a framework for speech recognition in real world noise built on large scale acoustic simulation. Models often lose sound grounding under severe compound distortions, dropping words or hallucinating. Their Voices-in-the-Wild-2M data covers seven classic sound effects and fifty four compound cases, trained with sound to meaning progressive fine tuning and dual granularity policy optimisation. Tests beat state of the art systems on adverse condition benchmarks, and on compound scenarios word error rates fall by over thirty percent relative to baselines.

Catalog record

CREATORZhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101473
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mega-ASR: Towards In-the-wild 2 Speech Recognition via Scaling Up Real-world Acoustic Simulation Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao Abstract Despite rapid advances in automatic speech recognition (ASR) and large audio-language models, robust recognition in real-world environments remains limited by an “acoustic robustness bottleneck”: models often lose acoustic grounding and produce omissions or hallucinations under severe, compositional distortions. We propose MEGA-ASR , a unified ASR-in-the-wild framework that combines scalable compound-data construction with progressive acoustic-to-semantic optimization. We introduce VOICES-IN-THE-WILD-2M , covering 7 classic acoustic phenomena and 54 physically plausible compound scenarios, and train MEGA-ASR with Acoustic-to-Semantic Progressive Supervised Fine-Tuning and Dual-Granularity WER-Gated Policy Optimization . Extensive experiments demonstrate that MEGA-ASR achieves significant advantages over prior state-of-the-art systems on adverse-condition ASR benchmarks (45.69% vs. 54.01% on VOiCES R4-B-F, and 21.49% vs. 29.34% on NOIZEUS Sta-0). On complex compositional acoustic scenarios, MEGA-ASR further delivers over 30% relative WER reduction against strong open- and closed-source baselines, establishing a scalable paradigm for robust ASR in-the-wild. Project page: https://xzf-thu.github.io/Mega-ASR/ Data: huggingface.co/datasets/zhifeixie/Voices-in-the-Wild-2M Bench: github.com/xzf-thu/Voices-in-the-Wild-Bench Figure 1 : Radar comparison of Qwen3-ASR-1.7B and Mega-ASR across selected ASR evaluation subsets, covering both clean and robustness benchmarks. 1 Introduction Automatic speech recognition (ASR) is one of the most fundamental tasks in the speech domain, and has evolved rapidly in recent years. State-of-the-art ASR models ( 38 ; 49 ; 8 ) achieve excellent accuracy on widely used benchmarks ( 25 ) , with word error rates approaching 1%. Beyond this, large audio-language models (LALMs) ( 48 ; 6 ) scale to billion-parameter architectures that integrate pretrained linguistic knowledge and even support reasoning-based error correction ( 22 ) , improving contextual consistency and achieving human-level performance on canonical benchmarks. However, performance drops sharply under real-world acoustic conditions: WER typically rises to 10%–30%, and in harder cases can be as high as 70%, often accompanied by dropped utterances or severe hallucination . Recent work on ASR-in-the-wild ( 50 ; 12 ) seeks to bridge this gap through improved data and post-training strategies. Nevertheless, three limitations persist. (D1) Limited scenario coverage. Prior work typically targets one or two isolated conditions (e.g., noise or far-field), requiring different specialized models for different environments. (D2) Lack of compositional robustness. Robustness factors are studied independently, while real-world conditions are inherently compositional (e.g., simultaneous reverberation, echo, and frequency dropout), and large-scale data for such mixtures remains scarce. (D3) Mismatch between training data and real-world conditions. The data that existing models are trained on emphasize relatively mild WER ranges ( 4\% – 10\% ), which do not reflect challenging settings where WER exceeds 30% and demands stronger semantic reasoning over degraded signals. These gaps motivate a shift toward ASR-in-the-wild 2 , pushing ASR models to handle acoustic conditions that are not just singly complex, and to recognize speech under much harder settings. In this work, we propose Mega-ASR , a framework specifically designed to strengthen ASR capability under in-the-wild complex acoustic environments. Mega-ASR is able to (1) achieve state-of-the-art accuracy on individual environmental conditions within a single model, (2) deliver superior performance on real-world recordings exhibiting compound environmental effects, and (3) recover semantic information under highly challenging conditions, which requires a dataset that is both close to the real-world distribution and scalable. To this end, we introduce Voices-in-the-wild-2M , a large-scale ASR dataset comprising 7 canonical meta-scenarios and 54 newly constructed compound scenarios, generated by a spectral-manipulation-based simulation method. We first (i) simulate 7 atomic acoustic effects in isolation as the foundation, then (ii) scale to 54 compound scenarios with an agentic check that verifies physical plausibility (e.g., a church corresponds to far-field plus echo). To obtain data that is both challenging and suitable for training, we (iii) calibrate the difficulty distribution through controlled experiments, and finally (iv) filter out samples with WER above 70% to ensure training stability. We then develop Acoustic-to-Semantic Progressive Supervised Fine-Tuning (A2S-SFT) , addressing two coupled bottlenecks at medium-to-high WER: extracting semantic information from acoustic signals under heavy perturbation, and recovering the intended semantics. Through this progressive capability building, we obtainx Mega-ASR-Base , whose foundational capabilities for the reward signal that subsequent reinforcement learning depends on. Finally, during RL training, recognition errors at medium difficulty are mostly word-level mistakes, but once WER exceeds 30%, the dominant failure mode changes sharply into severely incorrect semantics, hallucinated guesses, and large portions of dropped sentences. As a result, WER-based rewards cannot provide an effective learning signal in this situation. We therefore propose Dual-Granularity WER-Gated Policy Optimization (DG-WGPO) , a dynamic reward scheme with two parts. We also adopt a classic static rule-based reward consisting of WER and a repetition penalty as the basic learning signal. As the core of DG-WGPO, we introduce a Dual-Granularity Dynamic Reward designed specifically for ASR under complex acoustic environments, which combines a token-level refinement reward for local information recovery and a sentence-level reconstruction reward for overall semantic preservation on hard samples , with a WER-gated mirrored fusion strategy that dynamically allocates the weights between them. Extensive experiments show that MEGA-ASR substantially outperforms prior state-of-the-art systems on adverse-condition and compositional real-world benchmarks. 2 Related Work ASR Foundation Models and Robust Speech Recognition. Recent ASR foundation models, spanning encoder-decoder systems, large-scale self-supervised models, and audio-language models, have achieved strong results on standard benchmarks ( 30 ; 49 ; 38 ; 7 ; 47 ; 48 ; 6 ; 43 ) . However, strong performance under clean or mildly noisy conditions does not imply robustness in deployment, where speech is often corrupted by simultaneous degradations such as noise, far-field propagation, reverberation, obstructed, device distortion, and transmission dropout. Existing robust ASR studies typically address only one or two such factors, leaving severe and compositional conditions underexplored. Datasets and Simulation for In-the-wild ASR. A long line of robust ASR benchmarks studies recognition under adverse conditions, including additive noise, distant microphones, reverberation, replayed speech, and device effects ( 15 ; 42 ; 32 ; 24 ; 33 ; 1 ; 28 ) , but most emphasize isolated factors or mild degradation regimes. In practice, environments such as classrooms, corridors, or vehicles routinely combine background noise, far-field attenuation, echo, occlusion, and device-induced distortion. Augmentation methods like noise mixing, RIR convolution, spectral masking, clipping, and codec simulation partially address this ( 39 ; 31 ; 18 ; 19 ; 26 ) , but typically serve as local training perturbations rather than a systematic model of real acoustic worlds. 3 Voices-in-the-wild-2M 3.1 Overview Existing datasets for robust ASR mostly cover only a narrow set of isolated acoustic conditions, with mild WER typically between 4%–10% as shown in Table 1 , whereas real-world environments mix multiple environmental effects (e.g., far-field with echo&reverb in a church interior) and routinely push WER beyond 30%. To facilitate research in this regime, we introduce Voices-in-the-wild-2M , a large-scale dataset built through spectrogram-level code-based simulation, the design choice that makes its scale tractable. To faithfully simulate the complex acoustic conditions encountered in-the-wild, we first identify, as shown in Figure 2 , seven classic in-the-field acoustic effects \left\{\textit{noise},\ \textit{far-field},\ \textit{obstructed},\ \textit{echo\&reverb},\ \textit{recording},\ \textit{electronic distortion},\ \textit{transmission dropout}\right\} , which we term atomic acoustic effects . Each atomic effect is implemented as a dedicated spectral processing pipeline and iteratively calibrated against real recordings,

Read the complete text