SpecMD: A Comprehensive Study On Speculative Expert Prefetching

Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple · 2026

Read SpecMD: A Comprehensive Study On Speculative Expert Prefetching by Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03921.txt.

Description

Duc Hoang and 3 co-authors benchmark how mixture-of-experts models should cache their specialists. These models activate only a few expert subnetworks per input, so keeping the right experts in memory decides speed. Their 2026 framework, SpecMD, compares caching policies across hardware setups under realistic limits.

Experiments showed expert access ignores the recency patterns classic caches assume. Their new Least-Stale policy exploits predictable access instead, cutting collision misses dramatically against the standard rule. Hit rates above eighty-eight percent came with much faster first-token times while using only a small slice of video memory.

Catalog record

CREATORDuc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03921

Browse the full library · About the Tower of Babel