inference-fleet-sim : A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference

Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University · 2026

Read inference-fleet-sim : A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference by Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16054.txt.

Description

Chen and co-authors size GPU fleets for language model serving with queueing math. Fleet choices lack closed answers and hinge on token length spreads. Routing rules and queue dynamics twist the math further. Their inference-fleet-sim tool blends M/G/c queueing with event simulation. A physics-based speed model covers A10G, A100, and H100 chips. Layouts span single pool, two-pool routed, and disaggregated forms. Seven planning cases run on public and synthetic traces. Each case shows a call that plain analysis gets wrong.

Catalog record

CREATORHuamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35831
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16054

Browse the full library · About the Tower of Babel