Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism

Chenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana María Tárano, Hannah Kerner · 2026

Read Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism by Chenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana María Tárano, Hannah Kerner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04870.txt.

Description

Chenwei Cui and 4 co-authors redesign how mixture-of-experts language models train across many processors in a 2026 paper. Expert parallel training is standard, but its message cost grows with the count of active experts, loads imbalance, and routing needs metadata exchange.

Their Multi-Head LatentMoE design with Head Parallel training keeps message cost constant no matter the active expert count. Traffic is fully balanced and fixed in advance, yet the scheme stays compatible with expert parallelism. IO-aware routing and expert computation speed it further. Against standard expert parallel training, the new method trained much faster at the same quality. With doubled granularity, it did better overall while still training faster. The authors aim at more open foundation model research.

Catalog record

CREATORChenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana María Tárano, Hannah Kerner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04870

Browse the full library · About the Tower of Babel