ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

Yuxing Xiang 1,2 {}^{\text{1,2}} Xue Li 2 {}^{\text{2}} Kun Qian 2 {}^{\text{2}} Wenyuan Yu 2 {}^{\text{2}} Ennan Zhai 2 {}^{\text{2}} Xin Jin 1 {}^{\text{1}} 1 {}^{\text{1}} School of Computer Science, Peking University 2 {}^{\text{2}} Alibaba Group · 2025

Read ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production by Yuxing Xiang 1,2 {}^{\text{1,2}} Xue Li 2 {}^{\text{2}} Kun Qian 2 {}^{\text{2}} Wenyuan Yu 2 {}^{\text{2}} Ennan Zhai 2 {}^{\text{2}} Xin Jin 1 {}^{\text{1}} 1 {}^{\text{1}} School of Computer Science, Peking University 2 {}^{\text{2}} Alibaba Group (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09999.txt.

Description

Yuxing Xiang and co-authors study the real workload of serving large language models in a live cloud service, in a paper from 2025. Serving means answering user requests with a trained model, at scale and with low delay. Good test workloads are key to building fast, well sized systems.

The team drew data from a world-wide cloud service and mapped patterns for text models as well as newer image-aware and reasoning models. Past studies were smaller and missed many of these traits. Based on their findings, the authors built ServeGen, a tool that creates realistic workloads client by client. In a real use case, the tool avoided a large shortfall in capacity that a naive method would have caused. The tool is shared in a public link.

Catalog record

CREATORYuxing Xiang 1,2 {}^{\text{1,2}} Xue Li 2 {}^{\text{2}} Kun Qian 2 {}^{\text{2}} Wenyuan Yu 2 {}^{\text{2}} Ennan Zhai 2 {}^{\text{2}} Xin Jin 1 {}^{\text{1}} 1 {}^{\text{1}} School of Computer Science, Peking University 2 {}^{\text{2}} Alibaba Group
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-09999

Browse the full library · About the Tower of Babel