Equinox: Holistic Fair Scheduling in Serving Large Language Models

Zhixiang Wei, James Yen, Jingyi Chen, Ziyang Zhang, Zhibai Huang, Chen Chen, Xingzi Yu, Yicheng Gu, Chenggang Wu, Yun Wang, Mingyuan Xia, Jie Wu, Hao Wang, Zhengwei Qi · 2025

Read Equinox: Holistic Fair Scheduling in Serving Large Language Models by Zhixiang Wei, James Yen, Jingyi Chen, Ziyang Zhang, Zhibai Huang, Chen Chen, Xingzi Yu, Yicheng Gu, Chenggang Wu, Yun Wang, Mingyuan Xia, Jie Wu, Hao Wang, Zhengwei Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16646.txt.

Introduction

Zhixiang Wei and co-authors build a fair scheduler in 2025 for serving large language models to many users at once. Serving farms must share costly graphics chips fairly, yet fairness means one thing to users, who want fast first words, and another to operators, who want high chip use. Scores for both sides are known only after a job runs, which creates a scheduling paradox.

Their system predicts the scores ahead of time with a team of prediction experts, then blends them into one fairness score for live decisions. Tests on real chat traces show higher throughput, much lower wait for the first token, and fairer sharing than a leading rival, while keeping chips busy.

Catalog record

CREATORZhixiang Wei, James Yen, Jingyi Chen, Ziyang Zhang, Zhibai Huang, Chen Chen, Xingzi Yu, Yicheng Gu, Chenggang Wu, Yun Wang, Mingyuan Xia, Jie Wu, Hao Wang, Zhengwei Qi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Equinox: Holistic Fair Scheduling in Serving Large Language Models Zhixiang Wei, James Yen, Jingyi Chen, Ziyang Zhang, Zhibai Huang, Chen Chen, Xingzi Yu, Yicheng Gu, Chenggang Wu, Yun Wang, Mingyuan Xia, Jie Wu, Hao Wang, Zhengwei Qi Abstract . Large Language Model serving faces unprecedented demand, yet existing schedulers struggle to allocate resources fairly across diverse workloads. Current deployment strategies like First-Come-First-Served offer no client isolation against monopolization, while Requests Per Minute policy wastes resources during off-peak hours. The Virtual Token Counter attempts fair-sharing by tracking token consumption but inadequately captures LLM resource dynamics. The Transformer architecture’s prefill-decode bifurcation creates conflicting resource patterns: memory-bound decode operations dominate latency, compute-bound prefill operations determine throughput, and GPU utilization follows independent batch-refresh patterns, making single-metric fairness unachievable. We address these limitations with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to 1.3 × 1.3\times higher throughput, 60% lower time-to-first-token latency, and 13% higher fairness versus VTC while maintaining 94% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms. 1. Introduction Figure 1 . Token-count scheduling is unfair for independent, single-turn requests: with equal total tokens, many short vs. few long requests yield very different user latency, GPU utilization, and throughput. Batching mainly shifts the advantage—short identical requests benefit in-batch; without batching, queued short requests can lose. Setup: S-LoRA ( Babakniya et al. 2023 ) on LMSYS Chat-1M ( Zheng et al. 2024a ) , A100-80GB, matching VTC ( Sheng et al. 2024 ) . (a) FCFS/VTC shows non-linear latency variation across token ranges. (b) VTC exhibits non-linear throughput patterns across token ranges. (c) VTC demonstrates step-like GPU resource usage across token intervals. Figure 2 . Experiments on NVIDIA A100-80GB use real-world trace (lmsys-chat-1m) for (a,b) and synthetic workload for (c) with fixed total tokens (RPS \times tokens/request) and 1:1 input:output ratio (prefill:decode). Real-world traces across vLLM and SGLang ( Figure 16 ) confirm these patterns stem from architectural properties, not implementation choices. Large Language Models (LLMs) have reshaped artificial intelligence by combining modern neural networks with scalable computing and vast datasets. Their versatility spans tasks from content generation to software engineering, driving rapid adoption ( Brown et al. 2020 ; Devlin et al. 2019 ; Radford and Narasimhan 2018 ; Vaswani et al. 2023 ; Sutskever et al. 2014 ) . For example, ChatGPT gained 1 million users in five days—and 100 million in two months—while DeepSeek topped the U.S. iOS chart shortly after its January 2025 release ( Hu 2023 ; Kerr 2025 ) . Yet, most prior research has primarily targeted improvements in model quality, inference speed ( Zhong et al. 2024 ; Wu et al. 2024 ) , and GPU utilization ( Yu et al. 2022 ; Gong et al. 2025 ) , leaving fair multi - tenant serving largely unexplored. This gap prompts a central question: How can we allocate resources both efficiently and equitably as LLM demand and workload diversity grow? Most production deployments ( Ouyang et al. 2025 ; Ganguli et al. 2022 ; Rapp et al. 2025 ; Schaeffer et al. 2023 ; Lin et al. 2024 ; Wang et al. 2025 ) rely on First Come First Served (FCFS) scheduling, which processes requests strictly in arrival order, thereby offering no client isolation and allowing compute-intensive requests to monopolize resources and degrade co-tenants. To mitigate this, providers often impose static Requests Per Minute (RPM) quotas ( OpenAI 2024 ) , a rate-limiting mechanism that caps the number of requests a client can send within a one-minute interval; however, these waste capacity during off-peak periods as the fixed allowance goes unused when demand is low. The Virtual Token Counter (VTC) ( Sheng et al. 2024 ) addresses these scheduling limitations by adapting fair-share scheduling from network queuing and operating systems ( Demers et al. 1989a ; Kobus and Szklarski 2009 ; Nagle 1987 ; Demers et al. 1989b ; Golestani 1994 ; Goyal et al. 1996 ) . In traditional domains, resource consumption patterns allow a single metric like CPU seconds or transmitted bytes to effectively represent overall usage, as these metrics naturally correlate with actual resource costs. VTC attempts to replicate this approach in LLM serving by adopting token count as its scheduling metric. The scheduler tracks cumulative input and output tokens per user and implements a work-conserving policy ( Jin et al. 2004 ) that prioritizes users with lower token totals. This approach solves both prior failures: it prevents FCFS starvation by ensuring proportional service allocation while avoiding RPM’s capacity waste through dynamic resource allocation instead of static quotas. Key Insight. Although VTC improves upon FCFS and RPM, it inadequately adapts fair-share scheduling principles to LLM serving environments. Unlike traditional domains where one metric comprehensively captures resource usage, LLM serving exhibits metrics that actively conflict rather than correlate. The Transformer architecture’s bifurcated nature drives this conflict because prefill processes prompts in parallel with superlinear compute-bound costs, while decode generates tokens sequentially with cache-dominated memory-bound costs ( Zhong et al. 2024 ) . Equal token counts thus yield vastly different resource consumption. As Figure 1 demonstrates, identical token counts produce divergent outcomes across latency , throughput , and GPU utilization due to opposing computational constraints. Latency grows monotonically as a memory-bound operation, with decode consuming over 90% of end-to-end time ( 2(a) ). Throughput exhibits compute-bound non-monotonic patterns, initially increasing then declining after 1,000 tokens ( 2(b) ). GPU utilization shows stepwise plateaus because shorter requests trigger frequent batch refreshes, introducing idle intervals during CPU-bound operations ( 2(c) ). These observations show that token count cannot serve as a universal fairness metric and motivate a shift from single metric fairness to an explicitly multi objective formulation. Key Idea. Motivated by this conflict, we propose a dual counter framework comprising the User Fairness Counter (UFC) and the Resource Fairness Counter (RFC) that acknowledges distinct stakeholder needs. The UFC ensures equal tenant Quality of Service (QoS) by combining weighted tokens with user perceived latency, with weighting justified because decode dominates both pricing and latency. The RFC ensures equitable operator allocation using GPU utilization and throughput, without weighting since throughput bottlenecks arise in prefill. We define holistic fairness (HF) as the joint consideration of UFC and RFC, reframing scheduling as the co-optimization of user experience and system efficiency rather than the division of a single resource. Implementing the dual-counter framework requires solving two key challenges: a paradox where computing UFC and RFC needs metrics that schedulers cannot predict, and the resulting tension between minimizing tenant latency and maximizing operator utilization. Our Approach. We address both challenges through our open-source system Equinox,

Read the complete text