Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, [2pt] Juncheng Billy Li, Heather Miller, [6pt] Carnegie Mellon University · 2026
Read Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning by Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, [2pt] Juncheng Billy Li, Heather Miller, [6pt] Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23036.txt.
Introduction
He and seven colleagues tackle supervised fine-tuning for mixture-of-experts models with fragile router layers. Dense mixing and load-balancing mitigate collapse but add noisy gradients that hurt performance. Pruning tests show rare experts encode useful knowledge, though discarding them degrades performance despite super experts activating more often. Their auxiliary-loss-free framework combines bias-driven sparsification with always-active gated condenser experts, preserving long-tailed expert information and beating state-of-the-art baselines by more than 2.5 percent on average across math and commonsense benchmarks.
Catalog record
| CREATOR | Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, [2pt] Juncheng Billy Li, Heather Miller, [6pt] Carnegie Mellon University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90049 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23036
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, [2pt] Juncheng Billy Li, Heather Miller, [6pt] Carnegie Mellon University Abstract Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMixer and ESFT mitigate router collapse with dense mixing or auxiliary load-balancing losses, but these introduce noisy gradients that often degrade performance. In preliminary experiments, we systematically pruned experts and observed that while certain “super experts” are activated far more frequently, discarding less used experts still leads to notable performance degradation. This suggests that even rarely activated experts encode non-trivial knowledge useful for downstream tasks. Motivated by this, we propose an auxiliary-loss-free MoE SFT framework that combines bias-driven sparsification with always-active gated condenser experts. Rather than enforcing balanced activation across all experts, our method encourages task-relevant experts to remain active while pushing long-tailed experts toward inactivity. The condenser experts provide a persistent, learnable pathway that alleviates gradient starvation and facilitates consolidation of information that would otherwise remain fragmented across sparsely activated experts. Analysis further suggest that this design better preserves long-tailed expert information under sparse routing. Experiments on large-scale MoE models demonstrate that our approach outperforms state-of-the-art SFT baselines such as DenseMixer and ESFT, achieving average gain of 2.5%+ on both mathematical reasoning and commonsenseQA benchmarks. 1 Introduction Mixture-of-Experts (MoE) models scale language models efficiently by activating only a small subset of experts per token, enabling massive capacity without increasing per-token compute. Yet the same sparse routing that drives their success also makes them fragile: MoE relies on a non-differentiable Top-K router, which blocks straightforward gradient flow and makes post-training, such as supervised fine-tuning (SFT), far more difficult than for dense LLMs. Over the years, researchers have sought to stabilize MoE training through progressively refined routing strategies. GShard ( Lepikhin et al. 2020 ) introduced top-2 gating with heavy auxiliary balancing losses, while Switch Transformers ( Fedus et al. 2022 ) simplified this to a single expert per token. More recent work, such as DeepSeek-MoE ( Wang et al. 2024a ) and DeepSeek-V3 ( Liu et al. 2024a ) , explored bias-based routers and minimized auxiliary losses to reduce gradient noise and improve efficiency. However, these advances primarily address pre-training. In the post-training setting, SFT remains underexplored: ESFT ( Wang et al. 2024b ) routes all gradients to the most-activated expert, while DenseMixer ( Yao et al. 2025 ) improves slightly by applying a Straight-Through Estimator (STE) ( Bengio et al. 2013 ) to approximate updates for inactive experts, yet STE introduces biased gradients. In parallel, recent studies have identified the existence of “super experts” Su et al. 2025 or “super weights” Yu et al. 2025 , whose activations dominate the routing and whose removal leads to sharp performance degradation. These findings suggest that a small subset of experts carries disproportionate importance. However, our observations reveal a complementary phenomenon: even the rarely activated experts encode indispensable information, and pruning them also causes substantial performance decline. This highlights the need for fine-tuning strategies that not only preserve the capacity of frequently activated super experts, but also consolidate information from the long tail of rarely activated experts. Motivated by these observations, we introduce condenser experts : experts that are always selected yet remain gated per token. This creates a learnable shared pathway that alleviates gradient starvation, narrows the train–inference routing gap, and facilitates preservation of information that would otherwise remain distributed across long-tailed experts. Experimental results show that our method consistently outperforms SoTA MoE SFT methods by 4+ points when post-training popular MoE LLMs on commonsense reasoning (PIQA, ARC, SIQA) and math reasoning benchmarks (MATH-500, AIME-25, GPQA, GSM8K, etc). Our implementation is open-source 1 1 1 https://github.com/HectorHHZ/ExpertCondenser . 1.1 Contributions • We extend scaling-law analysis to MoE compression by relating performance to the number of expert parameters retained. Through this analysis, we find that performance degrades substantially when rarely activated experts are removed, showing that these long-tailed experts encode indispensable knowledge beyond the most frequently activated “super experts.” • We propose ExpertCondenser , a MoE post-training method combines auxiliary-free bias-based routing with always-active gated condenser experts to alleviate gradient starvation, narrow the train–inference routing gap, and facilitate consolidation of information from long-tailed experts. 2 Does Saving ”Super experts” mean saving model performance? Figure 1: Illustration of the three expert scaling strategies. ( Small-Dense ) Experts are pruned, and all remaining experts are always activated. ( Small MoE ) Experts are pruned and only a subset of them are activated per token. ( Inference Reduce ) Fewer experts are selected per token while the expert pool is unchanged. A substantial performance gap remains between the base model and pruned variants under Small Dense and small MoE categories. Previous work ( Lu et al. 2024 ; Su et al. 2025 ) shows that pruning frequently activated “super experts” leads to significant performance degradation, highlighting their importance. However, it remains unclear whether retaining only these experts is sufficient to preserve model quality. To answer this, we study how performance scales with the total number of retained expert parameters. We consider three compression strategies (Fig. 1 ) that vary the expert pool size n and activation budget k . For token t , the selected experts are S_{t}=\mathrm{TopK}(\{s_{j,t}\}_{j=1}^{n},k), where s_{j,t} denotes the gating score. We examine: (i) Dense conversion via pruning: n\to n^{\prime} , k=n^{\prime} , activating all remaining experts (dense model); (ii) Smaller MoE via pruning: n\to n^{\prime} , k<n^{\prime} , preserving sparsity within a reduced expert pool; (iii) Reduced activation budget: n fixed, k\to k^{\prime}<k , increasing sparsity without changing model size. We use ES-Act and ES-Mag ( Wang et al. 2024b ) to select experts prior to pruning, with ES-Act as default (details in Appendix E ). To conduct a scaling-law analysis of experts, we design experiments using GPT-OSS ( OpenAI 2025 ) and DeepSeek-Coder-V2-Lite ( Liu et al. 2024b ) . In figure 1 , we summarize the results across the three strategies (more details in Appendix D , Tab. 7 and 8 ). Even though expert activation is highly skewed and a few “super experts” dominate routing (shown in Appendix P , Fig. 9 ), a substantial performance gap still remains between the base model and pruned variants. Retaining the top 75% of experts still results in more than a 10% drop. These results show that preserving ”Super Experts” is not sufficient to preserve model performance. Crucially, our experiments reveal that rarely activated experts nonetheless encode indispensable domain knowledge, which is distributed in a fragmented manner across long-tailed experts. 3 Proposed Method Our findings in Section 2 suggest that effective post-training should preserve knowledge from infrequently activated experts. This motivates a mechanism that consolidates such long-tailed information into a small set of persistent components. While traditional shared experts provide persistent computation, they are ungated and mix signals uniformly across inputs, limiting expressiveness. More generally, persistent computation paths are more expressive when they remain learnably mixed rather than fixed. A detailed discussion between condenser experts and shared experts is in Appendix B . We therefore introduce condenser experts : experts that are always selected yet remain gated per