A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic

Jing, Shuqing, Yongyi, Pan, Ting, Gongping, Jingdong, Jacob · 2026

Read A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic by Jing, Shuqing, Yongyi, Pan, Ting, Gongping, Jingdong, Jacob (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24086.txt.

Introduction

Yang and colleagues build a pronunciation coach for learners of Modern Standard Arabic, a language whose emphatic and throat consonants are hard for software to judge. Their system must spot and diagnose mispronounced phonemes, the smallest sound units, yet real learner recordings are scarce and synthetic speech differs from real voices. The 2026 framework therefore trains in two stages, first on native and synthetic material, then adapting gently to the small set of genuine learner data.

A pretrained encoder paired with temporal convolutional networks preserves fine sound detail that ordinary recognition often smooths away. Several saved model versions vote together at test time, with an N-gram rescoring step steadying the final answer. On the QuranMB.v2 test set the system posts a large relative gain over its baseline and tops the IqraEval challenge. Yang's team presents the result as a new high mark for low-resource Arabic pronunciation diagnosis.

Catalog record

CREATORJing, Shuqing, Yongyi, Pan, Ting, Gongping, Jingdong, Jacob
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24086

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic Jing, Shuqing, Yongyi, Pan, Ting, Gongping, Jingdong, Jacob Abstract Accurate phoneme recognition is pivotal for mispronunciation detection and diagnosis (MDD) in modern standard Arabic (MSA), yet remains constrained by data scarcity and the synthetic-real domain gap. This work proposes a two-stage end-to-end framework. It integrates a pre-trained encoder with causal dilated temporal convolutional networks to preserve fine-grained phonetic variations. A hierarchical two-stage strategy first learns general mappings from native/synthetic corpora, then adapts to scarce real learner data to mitigate domain shift without over-correction. Prediction stability is further enhanced via multi-checkpoint ensemble inference with N-gram rescoring. Evaluated on the QuranMB.v2 test set, our system achieves an F1-score of 0.7201 0.7201 , a 63.1 63.1 % relative improvement over baseline ( 0.4414 0.4414 ). This performance ranks at the top of the IqraEval.2 Challenge, establishing a new state-of-the-art for low-resource MSA in MDD. Yang Zhang Deng Li Dang Huang Chen Benesty keywords Modern standard Arabic, phoneme recognition, mispronunciation detection and diagnosis, two-stage training, causal temporal convolutional networks. † † address: 1 School of Electronic Information, Wuhan University, Wuhan Hubei, 430072, China 2 School of Computing and Information Systems, University of Melbourne, Melbourne, Australia 3 Northwestern Polytechnical University 4 University of Quebec † † email: [email protected], [email protected] 1 Introduction Computer-aided pronunciation training (CAPT) has become an essential tool for second language acquisition. Mispronunciation detection and diagnosis (MDD) [ 12 , 13 , 10 , 2 , 15 , 14 ] serves as a critical component of CAPT systems, and its effectiveness hinges on a specific capability: for fine-grained diagnosis, the ability to distinguish canonical from non-canonical pronunciation units is paramount. Unlike standard automatic speech recognition (ASR), which prioritizes semantic intelligibility and often tends to ”correct” learner errors, effective MDD requires the system to faithfully capture these non-canonical variations to enable precise error localization and diagnosis. This distinction is particularly critical for modern standard Arabic (MSA), a language with a complex phonological inventory (e.g., pharyngeal and emphatic consonants) that poses significant challenges for automated detection [ 8 ] . Recent studies have explored deep learning architectures for CAPT, specifically integrating connectionist temporal classification (CTC) [ 20 , 11 , 5 , 3 , 18 ] with self-supervised acoustic encoders [ 7 , 4 , 17 ] . While these end-to-end approaches [ 10 , 23 , 1 , 22 , 16 ] have demonstrated notable improvements in languages like English, their application to MSA remains constrained. Existing models struggle to capture MSA’s subtle articulatory variations, while the scarcity of standardized resources and benchmarks further stifles progress in this domain. Although recent initiatives, such as the Qur’anic mispronunciation benchmark [ 8 ] , have begun to bridge this resource gap, critical obstacles persist, such as bridging the distributional gap between synthetic and real learner speech and enabling fine-grained error diagnosis. To address these challenges, we propose a robust end-to-end MSA phoneme recognition framework specifically tailored for MDD. Our approach substantially outperforms existing methods on the Qur’anic mispronunciation benchmark, driven by three key innovations. First, we design a hybrid architecture that integrates a large-scale multilingual pre-trained encoder with causal dilated temporal convolutional networks (TCNs) [ 19 , 6 ] in a sequential, end-to-end manner. Unlike transformer-based models [ 1 ] that inherently prioritize global semantic coherence, our TCN-based module explicitly leverages a strong local inductive bias to preserve fine-grained phonetic variations essential for distinguishing canonical from non-canonical pronunciations, all while maintaining computational efficiency. Second, to effectively bridge the distribution gap between controlled synthetic errors and noisy real-world learner speech, we introduce a hierarchical two-stage training paradigm. In the first stage, the model learns general acoustic-phonetic mappings from large-scale native and synthetic data; in the second stage, it adapts specifically to the unique distribution of scarce real learner recordings, thereby mitigating domain shift without overfitting to synthetic artifacts. Third, we employ a multi-checkpoint ensemble inference method combined with modified Kneser-Ney (MKN) N-gram language model rescoring [ 9 ] , leveraging complementary features from different training phases to enhance prediction stability on unseen data. The effectiveness of our proposed framework is decisively validated on the blind QuranMB.v2 test set. Our system achieves a phoneme-level F1-score of 0.7201, surpassing the official baseline ( 0.4414 ) by a remarkable 63.1 % relative improvement. This result ranks at the top of the IqraEval.2 Challenge at Interspeech 2026 1 1 1 hf.co/spaces/IqraEval . It establishes a new state-of-the-art for low-resource Arabic MDD, demonstrating exceptional generalization capability and resilience against data scarcity. Figure 1: Overview of the proposed framework. The system consists of a wav2vec2-XLS-R encoder coupled with causal dilated TCNs. Training follows a two-stage hierarchy: (1) general feature learning on native and synthetic data, and (2) pronunciation pattern adaptation on real learner data. Inference employs a multi-checkpoint ensemble fused via CN and rescored with N-gram language model. Our main contributions are summarized as follows: • A Robust End-to-End Architecture Tailored for MSA: We propose a hybrid framework that integrates a large-scale multilingual pre-trained encoder with causal dilated TCNs. The design is optimized to preserve fine-grained phonetic details that are critical for the complex phoneme inventory of MSA. In doing so, it reduces the tendency to over-smooth subtle acoustic anomalies that is often observed in recurrent architectures such as LSTMs, and it outperforms LSTM-based baselines in low-resource settings. • Hierarchical Two-Stage Training Strategy: We propose a novel training paradigm that sequentially optimizes for general feature learning (on native+synthetic data) and pronunciation pattern adaptation (on real learner data). This strategy effectively resolves the synthetic-real domain gap identified in the IqraEval benchmark. • Diversity-Aware Ensemble Inference: We introduce a multi-checkpoint ensemble inference framework that fuses predictions from multiple model states via confusion networks (CN) and enhances stability through MKN N-gram language model rescoring. This approach leverages complementary strengths across training phases, achieving a 5.5 % relative improvement over the single-checkpoint baseline. The remainder of this paper is organized as follows: Section 2 details the proposed methodology. Section 3 presents the experimental setup and results. Finally, Section 4 concludes the paper. 2 Proposed Method As illustrated in Figure 1 , our approach comprises three core components: (1) a hybrid model architecture integrating a large-scale multilingual pre-trained encoder with causal dilated TCNs; (2) a hierarchical two-stage training strategy that first learns general acoustic-phonetic representations from native and synthetic data, then adapts the model to the distribution of real learner speech to capture non-canonical pronunciation patterns; and (3) a diversity-aware ensemble inference mechanism leveraging CN and linguistic constraints. 2.1 Hybrid Model Architecture Our model uses a ”Pre-trained Encoder + causal dilated TCNs + CTC Decoding” framework. The encoder incorporates cross-lingual and language-specific acoustic priors. The TCNs model local temporal patterns via causal dilated convolutions. The CTC loss enables end-to-end training without explicit frame-level alignment. 2.1.1 Multilingual Pre-trained Encoder To overcome the limitations of low-resource MSA data, we utilize wav2vec2-XLS-R [ 4 ] as the upstream acoustic encoder. Pre-trained on 126 languages with billions of audio samples, this model possesses rich cross-lingual acoustic priors that provide a robust foundation for representing diverse phonetic units. Crucially, its exposure to Arabic during pre-training endows it with the latent capacity to distinguish fine-grained phonetic features inherent to MSA,

Read the complete text