Reasoning Language Model Inference Serving Unveiled: An Empirical Study
Qi Li, Junpan Wu, Xiang Liu, Yuxin Wang, Zeyu Li, Zhenheng Tang, Yuhan Chen, Shaohuai Shi, Xiaowen Chu · 2025
Read Reasoning Language Model Inference Serving Unveiled: An Empirical Study by Qi Li, Junpan Wu, Xiang Liu, Yuxin Wang, Zeyu Li, Zhenheng Tang, Yuhan Chen, Shaohuai Shi, Xiaowen Chu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18672.txt.
Introduction
Qi Li and his eight co-authors study reasoning models in real service. It is the first broad study of its kind. Reasoning models think through problems step by step. They excel at math and coding. Their serving behavior was still unexplored. A pilot test compared them with ordinary language models. Four differences stood out. Memory use was heavy and unstable. Some requests straggled and held others up. Running times adapted to the problem. The models also showed domain preferences.
The team then tested standard efficiency tricks. Compressing the model improved efficiency at small accuracy cost. Speculative decoding helped in the same way. Prefix caching could hurt accuracy or speed. Cache compression could hurt as well. Small reasoning models were hit hardest. Tests under a realistic workload confirmed the pattern. Builders of serving systems gain concrete guidance from the study.
Catalog record
| CREATOR | Qi Li, Junpan Wu, Xiang Liu, Yuxin Wang, Zeyu Li, Zhenheng Tang, Yuhan Chen, Shaohuai Shi, Xiaowen Chu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Language Model Inference Serving Unveiled: An Empirical Study Qi Li, Junpan Wu, Xiang Liu, Yuxin Wang, Zeyu Li, Zhenheng Tang, Yuhan Chen, Shaohuai Shi, Xiaowen Chu Abstract The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving performance and behavior of RLLM remains unexplored , which may undermine the deployment and utilization of RLLM in real-world scenario. To close this gap, in this paper, we conduct a comprehensive study of RLLM service. We first perform a pilot study on comparing the serving performance between RLLM and traditional LLM and reveal that there are several distinct differences regarding serving behavior: (1) significant memory usage and fluctuations ; (2) straggler requests ; (3) adaptive running time ; (4) domain preference . Then we further investigate whether existing inference optimization techniques are valid for RLLM. Our main takeaways are that model quantization methods and speculative decoding can improve service system efficiency with small compromise to RLLM accuracy, while prefix caching, KV cache quantization may even degrade accuracy or serving performance for small RLLM. Lastly, we conduct evaluation under real world workload modeled by Gamma distribution to verify our findings. Empirical results for real world workload evaluation across different dataset are aligned with our main findings regarding RLLM serving. We hope our work can provide the research community and industry with insights to advance RLLM inference serving. The reproduction details can be found in § F . * * footnotetext: Equal Contribution. $\dagger$ $\dagger$ footnotetext: Corresponding Author. 1 Introduction Large language models (LLM) such as GPT ( gpt-4 ) , Claude ( claude3 ; claude4 ) , Gemini ( gemini ) , Llama ( llama3-herd-models ) have emeraged as powerful knowledge bases through pre-training. These models, trained on vast Internet-crawled corpora such as C4 ( c4 ) , PILE ( pile-dataset ) and guided by scaling law ( scaling-law ; scaling-law-gopher ) , have accumulated large-scale knowledge, and exhibited remarkable performance on various knowledge extensive tasks. Despite these advancements, LLMs are criticized for their unsatisfactory capabilities on complex reasoning tasks, e.g., challenging mathematics, and programming tasks. Recently, reasoning large language models (RLLM) like OpenAI o1 ( openai-o1 ) , DeepSeek R1 ( deepseek-r1 ) , Qwen-3 ( qwen3 ) have sparked a growing body of research into test time scaling ( test-time-scaling ; s1 ) via long chain-of-thought reasoning ( chain-of-thought ) , significantly improving their mathematical reasoning, coding tasks and knowledge reasoning capabilities, e.g., even a 1.5B open source RLLM can surpass giant cutting-edge LLMs like GPT-4o on math tasks ( deepseek-r1 ) . Such achievements make it possible to deploy a small to medium RLLM as a powerful assistant to light the burden of workload for the staff of small entities or even for person, democratizing the use of cutting-edge RLLMs. Hence, it is desirable for small entity with limited GPU resources to efficiently deploy RLLM with inference engine privately for internal use. Nevertheless, current LLM serving engine, e.g. vLLM ( vllm ) , LMDeploy ( lmdeploy ) , Tensor-RT ( tensor-rt ) , are initially designed for traditional LLM , other than for RLLM. Though optimization techniques for LLM serving (§ 2 ) have been extensively studied, it remains largely unexplored whether RLLM exhibits distinct serving characteristics from LLM. If so, directly applying existing LLM serving techniques to RLLM may leave sub-optimal serving performance. Thus, it is natural to ask the following critical research question: Is there any distinct difference in serving behaviors between LLM and RLLM? To answer the above question, we perform systematic study of efficient RLLM serving. We first establish the ASU assessment framework (§ 3.2 ) for assessing RLLM serving. To justify whether there exists a distinct difference in serving behavior between RLLM and LLM, we design a benchmark suite named ASU-Perf and conduct a pilot investigation with it on different scale LLM and RLLM (§ 4 ). We found that when requests arrive in batches, the serving behavior of RLLMs differ significantly from LLMs, and the main findings can be primarily summarized in the following aspects: (1) RLLM exhibits significant KV Cache fluctuations and usage; (2) long tail distribution of requests running time caused by slow requests; (3) RLLM solves different difficulty level problems with adaptive running time; (4) RLLM excels LLM on math reasoning while on-par on knowledge intensive tasks. To understanding RLLM serving further, we first conduct extensive evaluations with various optimization techniques across diverse benchmarks (§ 5 ). We find that the model quantization and speculative decoding integrated in serving engine can improve serving efficiency and performance with only small compromising on accuracy of RLLM. However, prefix caching, and KV cache quantization do not always improve serving efficiency. They degrade the accuracy or serving performance for small RLLM, e.g., 7B model. Lastly, we conduct evaluation ( § 6 ) under real world workload modeled by Gamma distribution to verify our findings with different scale language models across different domain. Empirical results of real world workload evaluation indicate that the serving behaviors of RLLM are distinct from the LLM and are aligned with our main findings regarding RLLM serving. We hope our work can provide the research community and industry with insightful perspectives to help advance studies in efficient RLLM serving. To the best of our knowledge, we are the first to dissect the RLLM serving performance. The main contributions of this paper are the following. • Conceptually, we propose ASU , a framework to assess RLLM serving, which considers accuracy of response, RLLM service-provider side metric, and user side performance metrics together (§ 3 ). • Technically, we introduce ASU-Perf , a benchmarking suite for evaluating RLLM serving (§ 3 ). • Empirically, we reveal key differences of serving behaviors between RLLM and LLM: Significant Memory Fluctuations and Usage , Straggler Requests , and Adaptive Running Time (§ 4 ). • We conduct extensive experiments on some RLLM serving optimization techniques (§ 5 ). • We empirically validate our findings in real-world workload and verify their generalization (§ 6 ). 2 Preliminaries In this section, we provide preliminaries of RLLM, LLM serving and its metric. For comprehensive introduction of LLM serving optimization and recent advancement, please refer to Appendix E . RLLM and LLM. LLMs have demonstrated remarkable capabilities across various natural language processing tasks. However, standard LLMs often encounter difficulties when faced with complex problems that require multi-step reasoning, planning, and deeper cognitive processes, sometimes referred as “System-2 tasks” ( li2025system ) . To address these limitations, RLLMs have emerged, specifically engineered to enhance these deliberative reasoning abilities. A key technique employed by RLLMs is the “long Chain of Thought” (long CoT) prompting strategy ( shao2024deepseekmathpushinglimitsmathematical ) . This approach encourages the model to generate extended, explicit step-by-step reasoning pathways, breaking down complex problems into more manageable parts. Unlike standard LLMs that might provide more direct or less detailed answers, RLLMs utilizing long CoT can better navigate the intricacies of tasks, leading to more accurate and justifiable solutions by methodically thinking through the problem. This distinction allows RLLMs to tackle challenges in domains like advanced mathematics, intricate logical puzzles, and long-horizon planning more effectively than