Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang · 2025
Read Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading by Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05370.txt.
Description
Hanfei Yu and co-authors present FineMoE, a system to serve large mixture-of-experts language models with less waste. Their paper appeared in twenty twenty-five. In such models, just a few expert parts fire for each prompt, yet all parts sit in fast GPU memory. Past fixes move idle parts to slower CPU memory, but coarse moves add lag or still waste space.
FineMoE learns fine patterns in which experts fire, plus hints from each prompt's sense. Those clues guide what to fetch early, keep close, or move out. A test rig with six GPUs ran real work loads. Lag fell by near half and expert hits rose by a wide share over top past tools.
Catalog record
| CREATOR | Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05370