inference-fleet-sim : A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference
Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University · 2026
Read inference-fleet-sim : A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference by Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16054.txt.
Description
Chen and co-authors size GPU fleets for language model serving with queueing math. Fleet choices lack closed answers and hinge on token length spreads. Routing rules and queue dynamics twist the math further. Their inference-fleet-sim tool blends M/G/c queueing with event simulation. A physics-based speed model covers A10G, A100, and H100 chips. Layouts span single pool, two-pool routed, and disaggregated forms. Seven planning cases run on public and synthetic traces. Each case shows a call that plain analysis gets wrong.
Catalog record
| CREATOR | Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35831 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16054