Accelerating MoE Model Inference with Expert Sharding CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Machine learning Conference: The 5th Workshop on Machine Learning and Systems ; March 30-April 3 2025; Rotterdam, Netherlands The 5th Workshop on Machin

Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic · 2025

Read Accelerating MoE Model Inference with Expert Sharding CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Machine learning Conference: The 5th Workshop on Machine Learning and Systems ; March 30-April 3 2025; Rotterdam, Netherlands The 5th Workshop on Machin by Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08467.txt.

Description

Oana Balmau and co-authors speed up mixture-of-experts models by slicing experts across graphics cards in a 2025 paper. The catalog title carries pasted conference classification text. These models route each token to a few specialist sub-networks, but routing is lopsided: some cards idle while others drown.

Their system, MoEShard, splits each expert's matrices by rows and columns across all cards, so every card shares every expert's work. No tokens are dropped and balance is perfect by construction. Fusing the sliced computations cuts kernel launch overhead as well. Against a leading baseline on encoder models, time to first token improved up to 6.4-fold. The authors argue sharding, not luck, should carry expert parallelism.

Catalog record

CREATOROana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08467

Browse the full library · About the Tower of Babel