FleetOpt : Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University · 2026
Read FleetOpt : Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism by Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16514.txt.
Introduction
Chen and co-authors right-size GPU fleets for language model serving. Fleets are usually built for worst-case prompt lengths. Most requests never come close, so cache slots sit idle. Their FleetOpt derives the cheapest fleet from the workload's length spread. Queue models show a two-pool design wins: short and long contexts apart. A cost cliff at the pool boundary resists that split. Compress-and-route trims borderline prompts to soften the cliff. Production traces saved 6 to 82 percent.
Catalog record
| CREATOR | Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41248 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16514
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FleetOpt : Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism Huamin Chen, Xunzhuo Liu, Yuhan Liu, [4pt] Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project, [1pt] University of Chicago, [1pt] MBZUAI / McGill University Abstract Modern LLM GPU fleets are provisioned for worst-case context lengths that the vast majority of requests never approach, wasting GPU capacity on idle KV-cache slots. We present FleetOpt , a framework that starts from first principles: given a workload’s prompt-length CDF and a P99 TTFT target, derive the minimum-cost fleet analytically, then deploy it in practice. The analytical core models each pool as an M/G/ c c queue and derives that the minimum-cost fleet is a two-pool architecture — a short-context pool 𝒫 s \mathcal{P}_{s} and a long-context pool 𝒫 l \mathcal{P}_{l} — with an optimal boundary B short ∗ B_{\text{short}}^{*} satisfying an equal marginal GPU cost condition across both pools. The fundamental barrier to achieving B short ∗ B_{\text{short}}^{*} is the cost cliff : a hard routing step where requests just above B short ∗ B_{\text{short}}^{*} consume 8 × 8\times – 42 × 42\times more GPU capacity than requests just below it (depending on the context window ratio), creating a structural disincentive to lower the boundary. Compress-and-Route ( C&R ) is the implementation mechanism that resolves this barrier. Gateway-layer extractive compression trims borderline requests below B short ∗ B_{\text{short}}^{*} before the engine ever sees them, converting the hard hardware boundary into a software parameter read from the workload CDF. The two components are unified in the FleetOpt offline planner: given a CDF and SLO, it returns the optimal ( n s ∗ , n l ∗ , B short ∗ , γ ∗ ) (n_{s}^{*},n_{l}^{*},B_{\text{short}}^{*},\gamma^{*}) in under 1 ms. On three production traces, the combined framework reduces total GPU cost by 6–82% versus a homogeneous fleet, with C&R contributing 1–44 percentage points beyond plain pool routing depending on workload archetype. The analytical model is validated against a discrete-event simulator ( inference-fleet-sim ) with ≤ 3 % \leq 3\% error on predicted GPU utilization across all pools and workloads. 1 Introduction Modern LLMs support context windows of 128K tokens or more, yet production traces reveal a persistent mismatch: in the Azure LLM Inference Trace, 80% of requests use fewer than 2K tokens ( 15 ; 21 ; 3 ) . A homogeneous fleet provisioned for worst-case context length allocates KV-cache capacity that goes almost entirely unused for the vast majority of requests. Pool routing ( 5 ) addresses this by splitting the fleet into short- and long-context pools, cutting GPU cost by 16–38% in scenarios studied by prior work. The residual problem: the cost cliff. Even with pool routing, a structural inefficiency remains at the pool boundary B_{\text{short}} . The routing decision is binary: a request at L_{\text{total}}=B_{\text{short}} fills one of n_{\max}^{(s)} short-pool slots; a request one token longer must enter the long pool, which offers only n_{\max}^{(l)}=16 slots per GPU — a throughput-capacity penalty of 8\times – 42\times for a single token (Section 2.2 ). Requests in the borderline band (B_{\text{short}},\,\gamma B_{\text{short}}] are not genuinely long; they are RAG payloads with one extra paragraph, or multi-turn sessions whose history just crossed the threshold. For the workloads we study, 4.6–11.2% of all traffic falls in this band. Each borderline request is assigned a long-pool slot provisioned for 64K tokens while using at most 1/\rho of its KV budget. Our approach: analytical model first, then implement. Prior work treats pool routing and Compress-and-Route ( C&R ) as independent operational interventions: deploy pool routing, then optionally retrofit C&R on the existing fleet. This framing leaves value on the table. We take a different path. We start from the analytical optimum : given a workload CDF and a P99 TTFT target, what is the minimum-cost fleet? The M/G/ c queuing model answers this precisely: a two-pool architecture with a specific boundary B_{\text{short}}^{*} . But the cost cliff is a barrier to achieving B_{\text{short}}^{*} — lowering the boundary forces borderline requests to the expensive long pool. C&R is the implementation mechanism that resolves this barrier. By compressing borderline requests below B_{\text{short}}^{*} at the gateway, C&R makes the analytically-optimal boundary achievable in practice. The combined framework does not merely retrofit compression onto an existing fleet; it provisions the correct fleet from the start, using C&R to enforce the boundary the model prescribes. Key results. 1. The minimum-cost fleet is analytically derived. Under the M/G/ c model (with server count c=n_{\text{gpus}}\times n_{\max} KV slots), the optimal two-pool fleet satisfies an equal marginal GPU cost condition across both pools. GPU counts come from Erlang-C inversion at each pool’s arrival rate and service distribution (Section 3 ). 2. The cost cliff is a barrier to achieving B_{\text{short}}^{*} without compression. At the optimal boundary, the borderline band contains 43–76% of above-threshold traffic for real workloads (4.6–11.2% of all traffic). Without compression, the operator must either raise B_{\text{short}} (losing savings) or over-provision the long pool (paying more) to absorb borderline load. 3. C&R converts the hard boundary into a software knob. Gateway-layer extractive compression trims borderline prompts in 2–7 ms. The compressed token budget is chosen so KV overflow is impossible by construction (Section 5 ). The effective routing boundary shifts from B_{\text{short}} to \gamma B_{\text{short}} , where \gamma^{*} comes from the planner sweep. 4. Combined savings: 6–82% vs. homogeneous. On three production traces, the FleetOpt framework achieves 6.7–82.4% GPU cost reduction versus homogeneous deployment. C&R contributes an incremental 1.2 pp (Agent-heavy), 15.9 pp (LMSYS), and 43.7 pp (Azure) beyond plain pool routing (Section 7 ). 5. Analytical model validated by DES. inference-fleet-sim ( 6 ) confirms analytical GPU utilization predictions match simulation within 3% across all workloads and pools, operating in the many-server regime where queueing delays are negligible. Paper organization. Section 2 characterizes the cost cliff and workload archetypes. Section 3 develops the analytical fleet model. Section 4 derives optimal fleet sizing and the optimal boundary. Section 5 presents C&R as the implementation mechanism. Section 6 gives the FleetOpt offline planner. Section 7 evaluates on production traces. Sections 8 and 9 discuss related work and conclude. 2 Background: The Cost Cliff and Workload Archetypes 2.1 Pool Routing Basics A request r is assigned a total token budget L_{\text{total}}=\lceil|r|/\hat{c}_{k}\rceil+r.\texttt{max\_output\_tokens} , where \hat{c}_{k} is a per-category bytes-per-token EMA estimate. Pool routing routes r to \mathcal{P}_{s} if L_{\text{total}}\leq B_{\text{short}} , else \mathcal{P}_{l} . The short pool is sized for C_{\max}^{(s)} tokens (e.g., 8K) with n_{\max}^{(s)} concurrent sequences per GPU; the long pool for C_{\max}^{(l)} tokens (e.g., 64K) with n_{\max}^{(l)} . 2.2 KV-Cache Memory and the Cost Cliff For Llama-3-70B in fp16, the KV cache grows at 320 KB per token across 80 layers. A long-pool slot sized for 64K tokens requires 64{,}000\times 320\,\text{KB}\approx 20.5 GB of GPU memory; the same hardware configured for 8K can host \rho=8\times as many concurrent sequences, delivering \rho=8\times the throughput. The GPU savings formula for pool routing is \alpha(1-1/\rho) ( 5 ) , where \alpha is the fraction of traffic routed to \mathcal{P}_{s} . The cost cliff is the discontinuity at B_{\text{short}} : a request at B_{\text{short}}+1 tokens must enter the long pool, which hosts only n_{\max}^{(l)}=16 concurrent slots per GPU (sized for 64K tokens),