TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
Ruiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov · 2025
Read TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity by Ruiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03416.txt.
Description
Serving language models to many users needs smart scaling. Ruiqi Lai and co-authors present TokenScale in this 2025 work. Modern serving splits prefill and decode stages, which improves use of hardware but struggles with bursty demand.
Older autoscaling policies rely on lagging signals like GPU use or rough request counts. TokenScale uses token velocity, a timely signal tied to real work. Faster, more accurate scaling keeps response times low without wasting machines. That balance saves cost for firms running large AI services. Token velocity reacts at once to load, unlike utilization gauges that lag behind bursts.
Catalog record
| CREATOR | Ruiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03416