A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

Chengyi Nie, Nian Si, Zijie Zhou · 2026

Read A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints by Chengyi Nie, Nian Si, Zijie Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04595.txt.

Description

Chengyi Nie, Nian Si, and Zijie Zhou build the first queueing-theory framework for analysing whether a large language model serving system can keep up with demand. Serving is limited not only by computation but by the memory used to cache past tokens, which speeds generation yet quickly fills graphics memory.

Their framework includes both constraints and yields rigorous conditions separating stable operation from queues that grow without bound. Operators can combine an estimated arrival rate with the derived stable service rate to size a cluster correctly, avoiding both wasteful over-buying and service failures from under-buying. Extensive experiments on real systems validate the theoretical predictions.

Catalog record

CREATORChengyi Nie, Nian Si, Zijie Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04595

Browse the full library · About the Tower of Babel