Accelerating MoE Model Inference with Expert Sharding CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Machine learning Conference: The 5th Workshop on Machine Learning and Systems ; March 30-April 3 2025; Rotterdam, Netherlands The 5th Workshop on Machin
Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic · 2025
Read Accelerating MoE Model Inference with Expert Sharding CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Machine learning Conference: The 5th Workshop on Machine Learning and Systems ; March 30-April 3 2025; Rotterdam, Netherlands The 5th Workshop on Machin by Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08467.txt.
Description
Oana Balmau and co-authors speed up mixture-of-experts models by slicing experts across graphics cards in a 2025 paper. The catalog title carries pasted conference classification text. These models route each token to a few specialist sub-networks, but routing is lopsided: some cards idle while others drown.
Their system, MoEShard, splits each expert's matrices by rows and columns across all cards, so every card shares every expert's work. No tokens are dropped and balance is perfect by construction. Fusing the sliced computations cuts kernel launch overhead as well. Against a leading baseline on encoder models, time to first token improved up to 6.4-fold. The authors argue sharding, not luck, should carry expert parallelism.
Catalog record
| CREATOR | Oana Balmau, Anne-Marie Kermarrec, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, Milos Vujasinovic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08467