Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading

Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang · 2025

Read Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading by Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05370.txt.

Description

Hanfei Yu and co-authors present FineMoE, a system to serve large mixture-of-experts language models with less waste. Their paper appeared in twenty twenty-five. In such models, just a few expert parts fire for each prompt, yet all parts sit in fast GPU memory. Past fixes move idle parts to slower CPU memory, but coarse moves add lag or still waste space.

FineMoE learns fine patterns in which experts fire, plus hints from each prompt's sense. Those clues guide what to fetch early, keep close, or move out. A test rig with six GPUs ran real work loads. Lag fell by near half and expert hits rose by a wide share over top past tools.

Catalog record

CREATORHanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05370

Browse the full library · About the Tower of Babel