MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

Dennis Liu, Zijie Yan *, Xin Yao, Tong Liu, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Chandler Zhou, David Wu, Xipeng Li, June Yang · 2025

Read MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core by Dennis Liu, Zijie Yan *, Xin Yao, Tong Liu, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Chandler Zhou, David Wu, Xipeng Li, June Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14960.txt.

Description

Dennis Liu and co-authors built in 2025 a training framework that lets giant mixture-of-experts language models learn across a thousand graphics processors without wasting them. Such models route each word piece to a few specialist subnetworks, and splitting that work across machines strains old schemes. Their answer is five-way parallelism with a twist.

The twist, parallel folding, lets the attention layers and the expert layers each pick their own best splitting pattern instead of sharing one. A token dispatcher that works across all five dimensions supports both dropping and keeping every token. On flagship open models the framework reached about forty to forty-nine percent of the hardware's theoretical peak. It held that pace at long context lengths. The code ships inside a public training library.

Catalog record

CREATORDennis Liu, Zijie Yan *, Xin Yao, Tong Liu, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Chandler Zhou, David Wu, Xipeng Li, June Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14960

Browse the full library · About the Tower of Babel