SpecMD: A Comprehensive Study On Speculative Expert Prefetching
Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple · 2026
Read SpecMD: A Comprehensive Study On Speculative Expert Prefetching by Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03921.txt.
Description
Duc Hoang and 3 co-authors benchmark how mixture-of-experts models should cache their specialists. These models activate only a few expert subnetworks per input, so keeping the right experts in memory decides speed. Their 2026 framework, SpecMD, compares caching policies across hardware setups under realistic limits.
Experiments showed expert access ignores the recency patterns classic caches assume. Their new Least-Stale policy exploits predictable access instead, cutting collision misses dramatically against the standard rule. Hit rates above eighty-eight percent came with much faster first-token times while using only a small slice of video memory.
Catalog record
| CREATOR | Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho, Apple |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03921