X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang · 2025
Read X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms by Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13337.txt.
Description
Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, and Minjia Zhang present a training system for very large AI models in a 2025 paper. The models use a Mixture-of-Experts design, where only some parts fire for each input. New designs split experts into finer parts, which boosts quality. But they also use more memory and need costly data swaps.
The system, called X-MoE, cuts waste in three ways. It trains without padding, skips needless data sends, and mixes ways to split work across chips. On the Frontier supercomputer, it trained models up to 545 billion parameters across 1,024 graphics chips. That is ten times larger than prior methods on the same hardware. Speed stayed high at that scale.
Catalog record
| CREATOR | Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13337