Cache Management for Mixture-of-Experts LLMs – extended version

Spyros Angelopoulos, Loris Marchal, Adrien Obrecht, Bertrand Simon · 2025

Read Cache Management for Mixture-of-Experts LLMs – extended version by Spyros Angelopoulos, Loris Marchal, Adrien Obrecht, Bertrand Simon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02408.txt.

Description

Spyros Angelopoulos, Loris Marchal, Adrien Obrecht, and Bertrand Simon wrote a 2025 computer science paper on memory management for mixture-of-experts language models. Such models activate only some expert subnetworks per token, so the active experts should sit in fast cache memory.

The authors define a new paging problem that captures layered model structure and expert reuse. They prove lower bounds on how well any deterministic or randomised strategy can compete with an ideal one. Those bounds suggest least-recently-used style policies perform well in theory. A layer-aware version of that policy is then proposed. Simulations on synthetic data and real usage traces show it beats classic paging policies, helping large models run within tight memory limits.

Catalog record

CREATORSpyros Angelopoulos, Loris Marchal, Adrien Obrecht, Bertrand Simon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02408

Browse the full library · About the Tower of Babel