X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms

Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang · 2025

Read X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms by Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13337.txt.

Description

Yueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, and Minjia Zhang present a training system for very large AI models in a 2025 paper. The models use a Mixture-of-Experts design, where only some parts fire for each input. New designs split experts into finer parts, which boosts quality. But they also use more memory and need costly data swaps.

The system, called X-MoE, cuts waste in three ways. It trains without padding, skips needless data sends, and mixes ways to split work across chips. On the Frontier supercomputer, it trained models up to 545 billion parameters across 1,024 graphics chips. That is ten times larger than prior methods on the same hardware. Speed stayed high at that scale.

Catalog record

CREATORYueming Yuan, Ahan Gupta, Jianping Li, Sajal Dash, Feiyi Wang, Minjia Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13337

Browse the full library · About the Tower of Babel