TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity

Ruiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov · 2025

Read TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity by Ruiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03416.txt.

Description

Serving language models to many users needs smart scaling. Ruiqi Lai and co-authors present TokenScale in this 2025 work. Modern serving splits prefill and decode stages, which improves use of hardware but struggles with bursty demand.

Older autoscaling policies rely on lagging signals like GPU use or rough request counts. TokenScale uses token velocity, a timely signal tied to real work. Faster, more accurate scaling keeps response times low without wasting machines. That balance saves cost for firms running large AI services. Token velocity reacts at once to load, unlike utilization gauges that lag behind bursts.

Catalog record

CREATORRuiqi Lai † \dagger, Hongrui Liu † \dagger, Chengzhi Lu, Zonghao Liu, Siyu Cao *, Siyang Shao *, Yixin Zhang *, Luo Mai, Dmitrii Ustiugov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03416

Browse the full library · About the Tower of Babel