LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training Conference: Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2; March 22–26, 2026; Pittsburgh, PA, USA Proceedings of the 31st ACM I
Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li, Bin Cui · 2026
Read LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training Conference: Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2; March 22–26, 2026; Pittsburgh, PA, USA Proceedings of the 31st ACM I by Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li, Bin Cui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11686.txt.
Description
Training mixture-of-experts models splits experts across devices, but routing is uneven: a few overloaded experts slow every iteration and become the bottleneck. Xinyi Liu and six co-authors answer with LAER-MoE, built on a new parallel scheme that fully shards each expert across devices and rebuilds partial experts at expert granularity during training. That flexibility allows expert parameters to be rearranged as loads shift.
Fine-grained scheduling of communication keeps overhead low, and a load-balancing planner designs rearrangement and token-routing plans. On a cluster of A100 graphics processors the system runs nearly seventy percent faster than current state-of-the-art training systems. For large expert models, the work suggests layout should adapt during training, not stay fixed from the start.
Catalog record
| CREATOR | Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li, Bin Cui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-11686