A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
Chengyi Nie, Nian Si, Zijie Zhou · 2026
Read A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints by Chengyi Nie, Nian Si, Zijie Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04595.txt.
Description
Chengyi Nie, Nian Si, and Zijie Zhou build the first queueing-theory framework for analysing whether a large language model serving system can keep up with demand. Serving is limited not only by computation but by the memory used to cache past tokens, which speeds generation yet quickly fills graphics memory.
Their framework includes both constraints and yields rigorous conditions separating stable operation from queues that grow without bound. Operators can combine an estimated arrival rate with the derived stable service rate to size a cluster correctly, avoiding both wasteful over-buying and service failures from under-buying. Extensive experiments on real systems validate the theoretical predictions.
Catalog record
| CREATOR | Chengyi Nie, Nian Si, Zijie Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04595