Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition Wonjun Lee, Hyounghun Kim, Gary Geunbae Lee Abstract Accented speech remains a persistent challenge for automatic speech recognition (ASR), as most models are trained on data dominated by a few high-resource English varieties, leading to substantial performance degradation for other accents. Accent-agnostic approaches improve robustness yet struggle with heavily accented or unseen varieties, while accent-specific methods rely on limited and often noisy labels. We introduce Moe-Ctc , a Mixture-of-Experts architecture with intermediate CTC supervision that jointly promotes expert specialization and generalization. During training, accent-aware routing encourages experts to capture accent-specific patterns, which gradually transitions to label-free routing for inference. Each expert is equipped with its own CTC head to align routing with transcription quality, and a routing-augmented loss further stabilizes optimization. Experiments on the Mcv-Accent benchmark demonstrate consistent gains across both seen and unseen accents in low- and high-resource conditions, achieving up to 29.3% relative WER reduction over strong FastConformer baselines. 1 Introduction Automatic speech recognition (ASR) has reached near-human accuracy on benchmarks such as LibriSpeech 27 and Switchboard, driven by large-scale pretraining and advanced architectures 3 ; 33 . However, these advances do not generalize uniformly: performance drops markedly on accented speech, where acoustic and phonetic patterns differ from those dominant in training data. Because accented speech remains underrepresented in most corpora, current ASR systems under-perform for accented speakers, raising concerns of fairness and inclusivity 35 ; 8 ; 11 . Prior research has mainly followed two directions. Accent-agnostic approaches aim to learn robust, accent-invariant representations through self-supervised or adversarial objectives 15 ; 6 . While models such as XLS-R 2 and Whisper 31 improve general robustness, their performance still degrades for heavily accented or unseen varieties, indicating that full invariance is insufficient to capture accent diversity 45 . Accent-specific approaches , on the other hand, leverage accent labels for explicit specialization via fine-tuning 24 , data augmentation 9 , or parameter-efficient modules such as adapters and accent embeddings 37 ; 18 . These methods improve performance for known accents but depend on accent labels even at inference, limiting scalability and generalization to unseen speakers. A scalable alternative lies in Mixture-of-Experts (MoE) , which allows experts to specialize across domains. Since accent is an utterance-level attribute, sequence-level MoE provides a natural fit for accented ASR. Recent studies have explored MoE for multilingual or dialectal ASR 42 ; 19 ; 4 , yet existing methods either require accent labels at test time or fail to reliably route inputs to the appropriate experts. To address these challenges, we propose Moe-Ctc , a novel architecture that integrates MoE with expert-level intermediate CTC (Connectionist Temporal Classification) supervision. Our method learns accent-aware routing during training but transitions to accent-agnostic inference through a two-stage learning process. Each expert is equipped with an auxiliary CTC head, aligning routing decisions with transcription quality and improving optimization stability. Experiments on the Mcv-Accent benchmark 29 demonstrate that Moe-Ctc consistently outperforms prior accent-robust baselines, achieving substantial WER reductions across both seen and unseen accents. 2 Related Work 2.1 Accented Speech Recognition Accented speech has long been recognized as a major source of performance degradation in ASR systems. Analyses on corpora such as L2-ARCTIC 44 , CommonVoice 1 , and GLOBE 40 consistently show that models trained primarily on native speech exhibit substantially higher error rates for accented speakers 45 ; 35 . Even large-scale pretrained models such as XLS-R and Whisper reduce but do not eliminate the accent gap: Whisper shows higher WER on non-native accents 11 , while XLS-R/Whisper performance degrades on African-accented datasets 26 . Accent-agnostic Approaches. One research strand aims to improve robustness without relying on accent labels. Self-supervised pretraining with wav2vec 2.0 3 , HuBERT 15 , and related models has proven effective at learning accent-invariant representations. Similarly, large-scale multilingual pretraining (e.g., XLS-R, Whisper) improves cross-accent generalization. Other works adopt adversarial or domain generalization techniques—such as domain adversarial training 36 ; 6 , relabeling strategies 16 , or coupled/contrastive learning 38 —to enforce invariance. While these methods improve robustness overall, they still under-perform on heavily accented or unseen varieties compared to accent-aware models. Accent-specific Approaches. Another line of work explicitly incorporates accent supervision. Fine-tuning on accent-labeled corpora 24 often boosts performance for the target accent but compromises generalization to unseen varieties. Data augmentation synthesizes accented speech via phoneme perturbations or accent conversion 9 , though gains are limited by coverage and realism. Parameter-efficient adaptations such as residual adapters 37 , non-linear modules 30 , and accent embeddings 18 ; 5 ; 23 ; 39 offer lighter-weight alternatives. Rapid cross-accent adaptation has also been explored for low-resource settings 32 ; 41 . Recently, 4 organized accent-specific LoRA modules as a mixture of experts, showing that modular parameterization can balance specialization and scalability. 2.2 Mixture-of-Experts for Accent Adaptation While accent-specific methods achieve strong specialization, they depend on labeled accent data and generalize poorly to unseen accents. MoE offers a scalable alternative: by selectively activating specialized subnetworks, MoE increases model capacity without proportional computational cost 34 ; 10 . In ASR, MoE has been applied to multilingual and multi-domain recognition 42 ; 17 , where routing mechanisms distribute inputs across experts under diverse acoustic conditions. A key challenge for accent-aware MoE is routing when accent labels are unavailable at inference. To address this, 29 ; 28 introduced beam search–based expert (codebook) selection, while 19 proposed dialect-adaptive dynamic routing using feature–embedding combinations. 4 presented MAS-LoRA, where accent-specific LoRA modules act as experts; averages all LoRA experts equally when accent labels are unavailable. In contrast, our Moe-Ctc introduces accent-aware routing during early training to encourage expert specialization, then transitions to accent-agnostic training for inference. This avoids auxiliary routing strategies, requires no accent labels at test time, and enables effective generalization to unseen accents. 3 Background Our proposed framework, Moe-Ctc , is built upon a FastConformer 33 encoder with the CTC head for ASR, augmented with two key ideas: intermediate CTC supervision and the MoE architecture. Intermediate CTC provides auxiliary objectives that stabilize optimization and enhance representation learning, while MoE introduces capacity through expert specialization. In the following subsections, we review these two components to clarify the foundation of our approach. 3.1 Intermediate CTC Loss Connectionist Temporal Classification (CTC) 12 is a widely adopted training criterion for non-autoregressive ASR models. By marginalizing over all valid frame-to-label alignments, CTC enables end-to-end training without the need for frame-level annotations. Recent studies have shown that inserting intermediate CTC losses 22 at multiple encoder layers can improve training stability and accelerate convergence 20 ; 14 . These auxiliary objectives encourage hidden representations at lower layers to be predictive of output units, thereby mitigating vanishing gradients and facilitating optimization. Formally, let h^{(\ell)} denote the encoder states at layer \ell . An auxiliary CTC loss \mathcal{L}_{\text{CTC}}^{(\ell)} is computed, and the overall training objective is \mathcal{L}=\mathcal{L}_{\text{CTC}}^{(L)}+\sum_{\ell\in\mathcal{I}}\lambda_{\ell}\,\mathcal{L}_{\text{CTC}}^{(\ell)}, (1) where L is the final encoder layer, \mathcal{I} is the set of intermediate layers with auxiliary supervision, and \lambda_{\ell} are weighting coefficients. 3.2 Sequence-level Mixture-of-Experts Mixture-of-Experts (MoE) 34 ; 10 augments deep models with sparsely activated expert networks. A gating function routes inputs to a subset of experts, enabling large capacity without proportional computation. Conventional MoE performs token-level routing, where each audio frame in ASR may be directed to different experts, incurring high computation and switching overhead. In contrast, sequence-level MoE assigns an entire utterance to one or a few experts, reducing cost while preserving specialization. This is particularly suitable for accented speech, since accent is an utterance-level attribute. Given encoder input H\in\mathbb{R}^{B\times T\times D} , where B is the batch size, T the number of audio frames, and D the hidden dimension, we mean-pool across time: \bar{h}=\tfrac{1}{T}\sum_{t=1}^{T}H_{:,t,:}. (2) The routing network maps this pooled representation \bar{h}\in\mathbb{R}^{B\times D} to logits L\in\mathbb{R}^{B\times N} , where N is the number of experts and L_{i,j} is the logit for assigning sample i to expert j . The gating probabilities are then obtained via softmax: g_{i,j}=\frac{\exp(L_{i,j})}{\sum_{k=1}^{N}\exp(L_{i,k})} Finally, the MoE output for sample i is the weighted sum of expert outputs: \textsc{MoE}(H_{i})=\sum_{j=1}^{N}g_{i,j}\,\textsc{Expert}_{j}(H_{i}). Here, each \textsc{Expert}_{j} is implemented as a feed-forward block, which we describe in detail in the following subsection. 4 Method Figure 1: Overview of the proposed Moe-Ctc architecture. The \ell -th MoE module, inserted between encoder blocks, is illustrated with four experts (shown as an example). Each expert is equipped with an auxiliary CTC head, producing local supervision loss \mathcal{L}_{\text{local},\ell} . During training, the router incorporates accent-aware routing through Accent Biasing and Accent Classification loss ( \mathcal{L}_{\text{accent}} ). 4.1 Mixture-of-Experts Module We extend the FastConformer encoder by inserting MoE modules between encoder layers ( \ell -th), rather than replacing the internal feed-forward networks as in 17 . Each MoE module consists of N parallel experts, where each expert is a two-layer feed-forward block with ReLU activation: \textsc{Expert}_{j}(H)=\textsc{FFN}_{2,j}\big(\text{ReLU}(\textsc{FFN}_{1,j}(H))\big), with \textsc{FFN}_{1,j},\textsc{FFN}_{2,j}\in\mathbb{R}^{D\times D} and D the hidden dimension. The routing network maps the pooled encoder representation \bar{h}_{i} of sample i (Eq. 2 ) to a distribution g_{i}\in\mathbb{R}^{N} over N experts. To promote efficiency, we adopt top- K routing: only the K experts with the highest probabilities g_{i,j} are selected, and their weights are renormalized: \tilde{g}_{i,j}=\frac{g_{i,j}}{\sum_{k\in\mathcal{K}_{i}}g_{i,k}},\quad j\in\mathcal{K}_{i}, where \mathcal{K}_{i}\subseteq\{1,\dots,N\} denotes the selected top- K experts for sample i . The final MoE output for sample i is then the weighted combination: \textsc{Moe}(H_{i})=\sum_{j\in\mathcal{K}_{i}}\tilde{g}_{i,j}\,\textsc{Expert}_{j}(H_{i}). 4.2 Accent-Aware Routing Building on the generic MoE module, we introduce Accent-MoE , which incorporates explicit accent supervision. During training, each utterance is associated with an accent label a_{i}\in\{0,1,\dots,A-1\} . We align each accent to a designated expert, encouraging balanced utilization and accent-specific specialization. For example, align j -th expert for the Scotland accent ( a_{i}) . Formally, let L_{i,j} denote the gating logit for sample i and expert j , and a_{i} its accent index. We define the accent-biased logit: \tilde{L}_{i,j}=L_{i,j}+\alpha\cdot\mathbf{1}[j=a_{i}], (3) where \alpha is the bias strength and \mathbf{1}[\cdot] is the indicator function. The final routing distribution is then g_{i,j}=\frac{\exp(\tilde{L}_{i,j})}{\sum_{k=0}^{N-1}\exp(\tilde{L}_{i,k})}. To enable the router to select appropriate experts even without accent labels at test time, the gating weights are also interpreted as accent logits and trained with an auxiliary loss \mathcal{L}_{\text{accent}}=-\sum_{i}\log\frac{\exp(g_{i,a_{i}})}{\sum_{j=0}^{N-1}\exp(g_{i,j})}. (4) By applying the biasing term, we guide the model to route samples to accent-specific experts—so that each expert is primarily trained for its designated accent—while the auxiliary classification loss further regularizes the routing process. However, explicit accent supervision alone may not align routing with recognition quality, so we introduce expert-level CTC supervision to directly couple expert selection with transcription accuracy. 4.3 Expert-Level CTC Supervision ( Moe-Ctc ) Inspired by intermediate CTC loss (Section 3.1 ), we extend this principle to Accent-MoE by equipping each expert with its own auxiliary CTC head, as illustrated in Figure 1 . This design provides fine-grained supervision that guides both layer-wise representations and expert-level specialization, a strategy we denote as MoE-CTC . Let X\in\mathbb{R}^{B\times T\times D} denote the input to the MoE module, and \textsc{Expert}_{j}(X) the output of the j -th expert. Each expert is equipped with a CTC head that maps hidden states into the vocabulary space: \textsc{Ctc-Head}:\mathbb{R}^{B\times T\times D}\rightarrow\mathbb{R}^{B\times T\times V}, where V is the vocabulary size of Ctc-Head . Each expert-specific head yields an auxiliary CTC loss \mathcal{L}_{\text{CTC}}^{(\ell,j)} at layer \ell for expert j , evaluated against the ground-truth transcription. Following the intermediate CTC self-conditioning strategy of 25 , the logits from each expert’s CTC head are projected back into the hidden dimension through a learnable linear mapping, \textsc{Proj}:\mathbb{R}^{B\times T\times V}\rightarrow\mathbb{R}^{B\times T\times D}, then combined according to the gating weights and added via a residual pathway: \displaystyle H_{j} \displaystyle=\textsc{Expert}_{j}(X), \displaystyle C_{j} \displaystyle=\textsc{Ctc-Head}_{j}(H_{j}), \displaystyle P_{j} \displaystyle=\textsc{Proj}(C_{j}), \displaystyle\tilde{C} \displaystyle=\sum_{j=1}^{N}g_{j}\,P_{j}\in\mathbb{R}^{B\times T\times D}, \displaystyle X^{\prime} \displaystyle=X+\tilde{C}. Here, the auxiliary CTC losses are computed directly from C_{j} , while the residual pathway ensures that these CTC-informed signals are stably integrated back into the shared representation. In this way, expert behavior is aligned with transcription quality and CTC-informed feedback is propagated across layers. Finally, to further couple expert performance with routing, we define a local routing-augmented objective: \mathcal{L}_{\text{local}}=\sum_{\ell=1}^{L}\sum_{j=1}^{N}g_{j}^{(\ell)}\,\mathcal{L}_{\text{CTC}}^{(\ell,j)} (5) where \mathcal{L}_{\text{CTC}}^{(\ell,j)} denotes the CTC loss from expert j in layer \ell , and g_{j}^{(\ell)} is its routing probability, analogous to the intermediate CTC objective (Eq. 1 ). By minimizing this loss, the router is encouraged to assign higher weights to experts that yield lower CTC loss, thereby improving routing quality and enhancing overall recognition performance. The final training objective combines the global CTC loss, the accent-aware auxiliary loss, and the expert-level CTC supervision: \mathcal{L}=\mathcal{L}_{\text{CTC}}+\beta\,\mathcal{L}_{\text{local}}+\gamma\,\mathcal{L}_{\text{accent}}, (6) where \beta and \gamma are weighting coefficients. In this way, expert-level CTC supervision directly links routing to transcription quality, encouraging experts to specialize in improving ASR accuracy rather than only modeling accent distinctions. 4.4 Accent-Agnostic Training To stabilize routing and improve generalization, we adopt a two-stage training strategy. The first stage focuses on specialization: both Accent-Moe and Moe-Ctc are trained with accent-aware routing (Figure 1 , Section 4.2 ), where the router is guided by an explicit biasing term and an auxiliary accent classification loss to encourage expert alignment with specific accents. The second stage promotes generalization by removing these accent-specific signals, allowing the router to autonomously select experts. In this phase, Accent-Moe is optimized solely with the global CTC objective, while Moe-Ctc continues to leverage both global and expert-level CTC supervision ( \mathcal{L}_{\text{local}} ). This staged design enables the model to first learn accent-specific expertise and then generalize to unseen accents by aligning expert selection with recognition quality. Configuration Seen Accent Unseen Accent encoder size model parameter size AUS CAN UK SCT US Seen average AFR HKG IND IRL MAL NWZ PHL SGP WLS Unseen average FastConformer 12.78M 11.6 15.5 15.1 11.4 13.1 13.3 20.0 27.4 29.5 22.5 30.8 18.0 27.1 34.1 17.1 25.2 Inter-CTC 12.78M 11.3 14.3 15.1 11.0 13.2 13.0 19.4 26.6 29.3 21.4 30.1 18.1 26.9 33.8 16.9 24.7 MoE 13.72M 9.8 12.7 12.9 8.6 11.5 11.1 16.4 24.4 25.3 19.5 26.5 15.3 23.2 31.2 16.2 22.0 Accent Moe 13.72M 8.5 11.2 12.2 7.5 10.8 10.0 16.0 23.8 24.6 18.9 26.8 14.8 22.9 30.5 15.8 21.6 Small Moe-Ctc 16.62M 8.2 10.5 11.4 6.6 10.1 9.4 15.7 22.8 23.2 18.3 25.5 13.7 22.5 29.6 15.2 20.7 FastConformer 26.39M 7.3 11.4 10.3 5.7 10.0 8.9 15.4 22.6 22.8 16.5 24.3 12.8 21.4 28.6 14.1 19.8 Inter-CTC 26.39M 7.3 11.2 10.4 5.1 9.8 8.8 15.5 22.8 22.9 16.1 23.8 12.2 21.1 27.3 13.7 19.5 Moe 28.37M 7.0 10.3 9.7 5.0 9.9 8.4 15.5 22.4 22.1 16.2 23.6 11.9 20.4 26.4 12.9 19.0 Accent Moe 28.37M 6.3 9.3 8.5 4.7 8.0 7.4 13.8 21.3 20.0 15.5 21.3 11.1 19.9 24.9 10.7 17.6 Medium Moe-Ctc 32.58M 5.9 8.7 8.4 4.3 7.2 6.9 12.9 19.4 18.3 14.8 21.1 10.7 18.9 24.2 7.9 16.5 FastConformer 115.60M 5.5 9.4 7.8 3.9 7.7 6.9 13.3 19.8 18.5 15.6 23.6 10.8 19.2 24.2 10.4 17.3 Inter-CTC 115.60M 5.6 9.2 7.9 4.1 7.5 6.9 12.9 19.5 17.9 15.3 23.8 10.5 18.3 23.3 9.9 16.8 Moe 123.48M 5.3 8.8 7.3 4.0 6.8 6.4 12.5 19.9 17.8 14.5 19.5 8.8 17.4 20.6 7.5 15.4 Accent Moe 123.48M 4.9 7.9 6.8 3.6 6.1 5.9 12.4 16.3 17.0 13.0 18.3 8.2 15.2 20.2 7.0 14.2 Large Moe-Ctc 131.90M 4.4 7.8 6.1 3.2 5.9 5.5 11.4 12.2 16.8 12.6 15.1 7.7 12.6 17.9 6.3 12.5 Table 1: WER (%) on the MCV-Accent-Test set, comprising 5 seen and 9 unseen accents. All models are pretrained on LibriSpeech-960h and subsequently fine-tuned on MCV-Accent-100h . Highlighted cells indicate the best WER within each encoder size group. 5 Experimental Setup 5.1 Datasets We conduct experiments on the Mcv-Accent benchmark 29 , which is derived from the English portion of the Mozilla CommonVoice corpus 1 . CommonVoice is a large-scale crowd sourced dataset covering diverse speakers and accents worldwide. The MCV-Accent split provides two training subsets (100h and 600h), along with corresponding development and test sets. Summary statistics are presented in Table 6 and Table 7 in the Appendix. The training and development sets include five seen accents—Australia, Canada, England, Scotland, and the United States—while the test set introduces nine additional unseen accents to evaluate cross-accent generalization. To simulate realistic training strategies, all models are first pretrained on 960 hours of English speech dataset LibriSpeech-960h 27 and then fine-tuned on MCV-Accent , following a common pipeline where large-scale standard English pretraining enhances downstream recognition of accented speech. 5.2 Models All experiments are implemented using the NeMo framework 21 and trained on eight NVIDIA A100 GPUs (80GB each). Baseline Models. We first build baseline FastConformer encoders of three sizes— Small , Medium , and Large —as summarized in Table 8 . To ensure fair comparison with our MoE-based architectures, we also include intermediate FastConformer variants with approximately 46M and 76M parameters, matching the parameter scale of the MoE extensions. MoE-Based Models. On top of the baseline Conformers, we additionally build three MoE variants: a standard MoE without accent supervision, an accent-aware Accent-Moe , and an expert-level supervised Moe-Ctc (Table 9 ). Each MoE model contains L=3 MoE layers inserted at the 4th, 8th, and 12th encoder blocks, with N=5 experts per layer matching the number of seen accents . Unless otherwise specified, all MoE variants share the same default settings: K=2 for Top-K expert selection, accent prior strength \alpha=2 , local CTC loss coefficient \beta=\tfrac{1}{2\cdot(L\times N)} , and accent classification loss weight \gamma=0.1 . Tokenizer and Decoding. We employ a 1024-token BPE vocabulary trained on MCV-Accent-100h . For decoding, we apply greedy CTC decoding without any external language model or beam search. 5.3 Training Configuration All models are trained with a global batch size of 1024 using mixed-precision (FP16) training. During both LibriSpeech-960h pretraining and MCV-Accent fine-tuning, checkpoints are saved at the end of each epoch, and the best model is selected based on the lowest validation WER, with a maximum of 500 epochs. We employ the AdamW optimizer with an initial learning rate of 1\times 10^{-4} for pretraining and 1\times 10^{-5} for fine-tuning, using a cosine annealing schedule with warmup. For Accent-MoE and Moe-Ctc , we use a two-stage training strategy (Section 4.4 ). Models are first trained with accent-aware routing, and the best checkpoint from this stage is then fine-tuned for 20 additional epochs without accent supervision. The final model is selected based on the best validation WER from this second stage. 6 Results 6.1 Main Result Table 1 presents the Word Error Rate (WER) of all models across seen and unseen accents under various encoder sizes (Table 8 , 9 ). Across all settings, the proposed Moe-Ctc achieves the lowest WER, demonstrating clear improvements over the FastConformer baselines. With the Small encoder, it yields relative WER reductions (WERR) of 29.3% on seen and 17.3% on unseen accents, and with the Large encoder, 20.3% and 27.8% , respectively. Notably, the gains on unseen accents increase with model capacity, suggesting that our approach generalizes more effectively as encoder size grows. To further examine the effect of intermediate supervision, we introduce an Inter-CTC variant that adds intermediate CTC losses to the FastConformer baseline. This variant yields consistent yet modest improvements, confirming that intermediate supervision stabilizes training and enhances recognition quality. Beyond this baseline, we compare the MoE and Accent-MoE variants, which share identical parameter sizes but differ in whether accent-aware supervision is applied. Across small, medium, and large encoders, Accent-MoE achieves average WERRs of 9.9%, 11.9%, and 7.8% on seen accents, and 1.8%, 7.4%, and 7.8% on unseen accents, respectively—highlighting the consistent advantage of accent-aware routing. Building on these results, Moe-Ctc demonstrates the most robust and generalizable performance, outperforming Accent-MoE by an additional 6.5% WERR on seen and 7.5% on unseen accents across all encoder sizes. These findings suggest that expert-level CTC supervision enables Moe-Ctc to further enhance recognition accuracy across both seen and unseen accents. To isolate the effect of model capacity, we control for parameter size in subsequent analyses. For fairness, we also compare Moe-Ctc and FastConformer under equal parameter budgets, along with prior Mcv-Accent benchmark results, as detailed in the following subsection. model parameter Seen ALL Unseen ALL Conformer 13 \dagger 43M 14.0 23.7 MTL 43 \dagger 43M 14.1 23.7 DAT 7 \dagger 43M 14.0 23.4 29 \dagger 46M 13.6 22.9 FastConformer 46.89M 13.8 23.7 Moe-Ctc 46.91M 12.7 22.3 Table 2: Weighted average WER (%) on the Mcv-Accent-Test set. All models are trained on the Mcv-Accent-100h training split. \dagger indicates results reported by 29 . 6.2 Benchmark To compare with prior studies on Mcv-Accent , we evaluate two benchmark settings. The first uses a 46M-parameter model trained solely on the Mcv-Accent-100h split without Librispeech pretraining, following 29 (as shown in Table 2 ). The second employs a 76M-parameter model trained on the larger Mcv-Accent-600h split, also without pretraining, following 28 (Table 3 ). The Seen-ALL and Unseen-ALL scores are calculated as weighted averages based on the number of samples per accent (Table 7 ), which differs from the unweighted averages reported in Table 1 . As shown in Table 2 , removing Librispeech-960h pretraining leads to higher WER across all models. Our proposed Moe-Ctc still outperforms other baselines and 29 , although the improvement is relatively modest, achieving a WERR of 6.6% on seen and 2.6% on unseen accents. When compared to our FastConformer baseline, the gains are slightly larger (8.0% on seen and 5.9% on unseen), but remain smaller than those observed in Table 1 . This indicates that training with only 100 hours of data is insufficient to fully exploit the advantages of the proposed method. In contrast, utilizing the larger Mcv-Accent-600h training set yields a substantial performance improvement. As shown in Table 3 , Moe-Ctc outperforms previous best model 28 with larger WERR margins of 18.4% on seen and 5.9% on unseen accents. model parameter Seen ALL Unseen ALL HuBERT 15 \dagger 74M 3.87 9.49 MTL 43 \dagger 74M 3.76 9.37 DAT 7 \dagger 74M 3.83 9.30 28 \dagger 76M 3.80 9.19 FastConformer 76.70M 3.53 9.52 Moe-Ctc 76.26M 3.10 8.65 Table 3: Weighted average WER(%) on the Mcv-Accent-Test set. All models are trained on the Mcv-Accent-600h training split. \dagger indicates results reported by 28 . 7 Analysis 7.1 Effect of Accent-Agnostic Training We further investigate the impact of accent-agnostic training (Section 4.4 ) for both Accent-Moe and Moe-Ctc under the Large encoder configuration. Table 4 presents the results across different training stages. For Accent-Moe , the accent-agnostic only training corresponds to the standard MoE baseline reported in Table 1 . Introducing an initial accent-aware stage yields consistent gains on both seen and unseen accents (6.4 \rightarrow 6.1 and 15.4 \rightarrow 15.1, respectively). When followed by accent-agnostic fine-tuning, the model achieves further improvements, reducing WER to 5.9 (↓ 3.8%) on seen and 14.2 (↓ 6.0%) on unseen accents. These results confirm that accent-agnostic training helps the router generalize beyond explicit accent supervision. A similar trend is observed in Moe-Ctc , where incorporating accent-agnostic fine-tuning after accent-aware pretraining leads to a more pronounced improvement—achieving 5.5 (↓ 5.2%) on seen and 12.5 (↓ 12.0%) on unseen accents. This indicates that our expert-level CTC supervision is especially effective when combined with accent-agnostic training, allowing the model to retain accent sensitivity while improving generalization to unseen accents. Model Training Stage Seen Avg. Unseen Avg. MoE Accent-agnostic only 6.4 15.4 Accent-Moe Accent-aware 6.1 15.1 +Accent-agnostic 5.9 ( \downarrow 3.8 %) 14.2 ( \downarrow 6.0 %) Moe-Ctc Accent-aware 5.8 14.2 +Accent-agnostic 5.5 ( \downarrow 5.2 %) 12.5 ( \downarrow 12.0 %) Table 4: Average WER(%) for seen and unseen accents under different training stages. Green text indicates WERR compared to the accent-aware stage. 7.2 Accent Routing Performance While 45 report 95% accuracy on a 16-way accent classification task using w2v2-XLSR (315M) trained on the CommonAccent dataset, their model is optimized solely for accent identification. In contrast, our router is jointly trained with ASR, making the task considerably more challenging. Under this multitask setting, the 76M Moe-Ctc model (in Table 3 ), trained on 600 hours of data, attains 72% top-1 routing accuracy on the final MoE layer before accent-agnostic training. A similar observation was made by 29 , who found that auxiliary accent classifiers for expert selection offered limited benefit over inference-time beam search, suggesting that perfect accent discrimination is neither achievable nor necessary within ASR training. After accent-agnostic fine-tuning, the router in Moe-Ctc still preserves meaningful accent organization, achieving 66.2% top-1 accuracy (Figure 2 ). The confusion matrix reveals that Canadian speech is often routed to the U.S. expert and Scottish to the England expert, indicating that routing aligns with acoustically and geographically correlated accents even without explicit supervision. This adaptive redistribution promotes better generalization by guiding expert selection toward recognition-oriented specialization (Section 7.1 ). Figure 2: Matrix of router gating probabilities ( g_{i,j} ) at the final ( 3rd ) Moe-Ctc module of the 76M model trained on Mcv-Accent-600h . Each row represents an accent, and the probabilities in each row sum to \sum_{j=1}^{N=5}g_{i,j}=1 . 7.3 Oracle Accent Routing Setting To estimate the upper bound of our architecture under ideal conditions, we conduct an oracle experiment where ground-truth accent labels are provided at inference time. Specifically, we enforce hard expert routing by applying the biasing term (Eq. 3 ) with maximum strength ( \alpha=\infty ), ensuring that each accent exclusively activates its designated expert. As shown in Table 5 , providing oracle labels yields the lowest WER of 5.2% under accent-aware training and 5.3% when combined with accent-agnostic fine-tuning, compared to 5.8% and 5.5% without label access. These results confirm that oracle routing achieves the best performance by leveraging accent-specific specialization, while accent-agnostic training slightly reduces specialization but improves generalization for label-free inference. model Training Stage Accent Label Seen Average MoE-CTC Accent-aware No 5.8 (5.82) MoE-CTC Both No 5.5 (5.48) MoE-CTC Accent-aware Yes 5.2 (5.20) MoE-CTC Both Yes 5.3 (5.32) Table 5: WER(%) under the oracle accent routing setting with Large encoder. Parentheses denote values reported to two decimal places. 8 Conclusion We presented Moe-Ctc , a Mixture-of-Experts architecture enhanced with intermediate CTC supervision for accent-robust ASR. By combining expert-level CTC heads with a two-stage transition from accent-aware to accent-agnostic training, the model achieves relative WER reductions of up to 29.3% and 27.8% on seen and unseen accents, respectively, on the Mcv-Accent benchmark. These results highlight that aligning expert routing with transcription quality substantially improves both robustness and generalization in accented speech recognition. Limitations While Moe-Ctc demonstrates substantial improvements in accented speech recognition, its sequence-level routing assumes discrete accent boundaries, which may not generalize to mixed or code-switched speech. The model also relies on accent labels during early training, limiting applicability in unsupervised settings. Although sparsely activated, the additional MoE modules increase training cost and latency. Finally, experiments are limited to English accents on Mcv-Accent ; broader multilingual evaluation and analysis of expert interpretability remain future work. Appendix A Appendix A.1 Mcv-Accent Dataset For English text normalization, we adopt the whisper-normalizer 1 1 1 https://kurianbenoy.github.io/whisper_normalizer/ toolkit to ensure consistent text preprocessing across both the training (Table 6 ) and evaluation (Table 7 ) sets. Accent Mcv-Accent-100h Mcv-Accent-600h Count Duration (h) Count Duration (h) US 46,385 64.08 289,177 399.88 England 14,590 19.51 90,081 119.93 Australia 5,018 6.95 31,355 43.56 Canada 4,815 6.79 29,436 41.13 Scotland 1,669 2.69 10,123 16.21 Table 6: Comparison of accent distribution and duration in Mcv-Accent-100h and Mcv-Accent-600h subsets from 29 . Accent Mcv-Accent-dev Mcv-Accent-test Count Duration (h) Count Duration (h) US 7,050 8.32 4,295 4.87 England 2,775 3.22 1,490 1.65 Australia 3,004 4.33 414 0.46 Canada 966 1.16 902 1.21 Scotland 233 0.23 132 0.16 New Zealand – – 1,620 2.11 Ireland – – 1,423 1.94 African – – 1,213 1.71 Philippines – – 631 0.90 Indian – – 492 0.58 Singapore – – 477 0.64 Hongkong – – 409 0.52 Malaysia – – 262 0.39 Wales – – 219 0.27 Table 7: Statistics of validation ( Mcv-Accent-dev ) and test ( Mcv-Accent-test ) sets. A.2 Model Configuration Details Table 8 summarizes the configurations of the FastConformer baselines, including the total number of parameters, encoder layers, hidden dimension, and multi-head attention heads. The corresponding configurations for Moe-Ctc are presented in Table 9 . The Small , Medium , and Large variants share identical FastConformer settings as in Table 8 , with additional MoE modules inserted between encoder blocks. For the 46M and 76M variants, the base FastConformer encoders are slightly downsized—by adjusting the number of layers and hidden dimensions—to ensure comparable total parameter counts after adding the MoE layers. Name Params Layers d_model Heads Small 12.78M 16 176 4 Medium 26.39M 16 256 4 46M 46.89M 18 324 4 76M 76.70M 18 416 4 Large 115.60M 18 512 8 Table 8: FastConformer configurations with parameter size, depth, hidden dimension ( d_{\text{model}} ), and multi-head attention heads. Name Params Layers d_model Heads Small 16.62M (12.78M) 16 176 4 Medium 32.58M (26.39M) 16 256 4 46M 46.91M (38.85M) 16 312 4 76M 76.26M (65.51M) 18 384 4 Large 123.48M (115.60M) 18 512 8 Table 9: Moe-Ctc model configurations. Notation follows Table 8 . Values in parentheses indicate the base FastConformer parameters, excluding Moe-Ctc modules. A.3 Adding Spare Experts By default, we set the number of experts to 5 , matching the number of seen accents for accent-aware routing. In the Large configuration, we further increase the capacity by adding three additional experts, resulting in a total of 8 experts. During training, these extra experts are randomly assigned to approximately 20% of the training samples from all seen accents, encouraging exposure to diverse acoustic patterns without explicit bias. As shown in Table 10 , increasing the number of experts consistently improves recognition performance. For Accent-MoE , expanding from 5 to 8 experts reduces WER from 5.86 to 5.83 ( 0.5% relative) on seen accents and from 14.18 to 14.12 ( 0.4% ) on unseen accents. The improvement is more pronounced for Moe-Ctc , achieving a WER reduction from 5.48 to 5.39 ( 1.6% ) on seen and from 12.51 to 12.28 ( 1.9% ) on unseen accents. These consistent gains indicate that the additional experts act as global experts, learning accent-invariant representations that complement accent-specific specialization and enhance overall generalization. Model Number of Experts Parameter size Seen average Unseen average Accent-Moe 5 123.48M 5.86 14.18 Accent-Moe 8 129.89M 5.83 ( \downarrow 0.5%) 14.12 ( \downarrow 0.4%) Moe-Ctc 5 131.90M 5.48 12.51 Moe-Ctc 8 141.37M 5.39 ( \downarrow 1.6%) 12.28 ( \downarrow 1.9%) Table 10: WER(%) comparison between 5- and 8-expert variants. Green text denotes relative WER reduction (WERR) of 8-expert over 5-expert models. A.4 Ctc-Head sharing Moe-Ctc (Figure 1 ) equips each expert with its own Ctc-Head , enabling expert-specific alignment but also increasing the overall parameter count. To examine more lightweight alternatives, we explore several Ctc-Head sharing strategies. As summarized in Table 11 , Full separation denotes the default configuration where each expert has an independent head. In the Layer-wise sharing setup, a single Ctc-Head is shared among all experts within the same MoE layer, while Global sharing removes intermediate heads entirely, using only the final global CTC head for all experts. Layer-wise sharing slightly degrades WER on both seen (+2.2%) and unseen (+4.5%) accents, yet it remains competitive and far superior to Accent-Moe while reducing parameters by about 5%. In contrast, global sharing results in a more substantial degradation (+8.0% on seen, +16.1% on unseen), even worse than Accent-Moe with no expert-specific head. We hypothesize that each MoE layer produces representations of varying transcription quality—earlier layers generate coarse alignments, while deeper layers refine them—thus separate heads at each layer help preserve alignment fidelity and stabilize expert specialization. Model CTC-Head Sharing Parameter size Seen average Unseen average Moe-Ctc Full separation 131.90M 5.48 12.51 Layer-wise sharing 125.59M 5.60 ( \uparrow 2.2%) 13.07 ( \uparrow 4.5%) Global sharing 124.01M 5.92 ( \uparrow 8.0%) 14.52 ( \uparrow 16.1%) Accent-Moe No Head 123.48M 5.86 14.18 Table 11: WER(%) comparison of different CTC head sharing strategies in Moe-Ctc . Red text indicates relative WER degradation compared to the full separation baseline.