LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen · 2026
Read LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems by Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08305.txt.
Introduction
Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, and coauthors release a benchmark for tuning the knobs of real language model systems. Such systems mix model settings with serving settings, their fidelity choices interact in nonlinear ways, and each configuration costs a different amount to measure, none of which older tuning benchmarks capture.
LLMSYS-HPOBench profiles hundreds of thousands of configurations, each with twelve to twenty-three hyperparameters plus several fidelity dimensions, giving hundreds of distinct evaluation settings. The suite is presented as the first live benchmark of hyperparameter optimization for real-world LLM systems. It gives the AutoML community realistic ground to test search methods against.
Catalog record
| CREATOR | Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08305
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen Abstract Large Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench , covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12–23, 3–5 dimensions of fidelity factor leading to 932 settings, 3–9 inference objective metrics, and 2–10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench . 1 Introduction The frontier of AI has evolved from machine/deep learning models into large language models (LLMs) that are often combined with planning, memory, retrieval, and tool usage 6 , dubbed the system of LLM, or simply the LLM systems. Their real-world effectiveness and efficiency, however, are governed not only by core LLM capabilities, but also by the vast other system components that interact with the LLM, e.g., tools, database, and hardware containers. This leads to a multifaceted hyperparameter space, including the AI parts (e.g., LLM temperature and prompting choices) and non-AI parts (e.g., retrieval and tooling strategies) 1 1 1 We treat the AI hyperparameters as those that are directly attached to the underlying LLM and can influence its behaviors; otherwise they are non-AI hyperparameters. 1 . It has been shown that different hyperparameter configurations of LLM systems can lead to substantially different inference behaviors in terms of task quality, latency, robustness, and monetary cost 6 . For example, in retrieval-augmented generation (RAG), increasing retrieval top- k items and enabling a stronger reranker can improve response quality, but often cause extra query time and token cost due to longer context and extra LLM calls 6 . The inherent black-box nature of LLM systems and their hyperparameter characteristics lead to a great opportunity for the AutoML community to conduct hyperparameter optimization (HPO), as shown in recent work 7 . However, despite the increasing importance of optimizing LLM systems’ hyperparameters, we have yet to witness considerable research progress/innovation, primarily because the measurement of hyperparameter configurations over the LLM systems is extremely expensive and effort-heavy: measuring one hyperparameter configuration may require tens of minutes to hours and a non-trivial API/compute budget; scaling to thousands of configurations can consume hundreds of GPU-hours 74 ; 37 . This is in addition to the complexity of deploying real LLM systems. The above is a corollary of lacking dedicated benchmark suite/datasets on configuration and HPO for LLM systems. Indeed, existing benchmarks/datasets for AutoML 44 ; 23 ; 54 ; 5 and for computing systems 50 cannot fulfill the needs of HPO for LLM systems due to several reasons: • They primarily focus on either classic machine/deep learning models or computing systems, which fail to capture the unique structure, AI/non-AI hyperparameter interaction, and fidelity implication for LLM systems. For example, LLM temperature and backend scheduling policy can significantly, sparsely, and jointly affect both quality and latency in a LLM system, while many classic HPO benchmarks assume a model-centred pipeline 72 ; 55 or are systems only 50 . • They focus on simple fidelity settings, while the tasks that LLM systems deal with are often much more complex due to the presence of natural language-based tasks. For example, the fidelity may vary with query difficulty, prompt length, and domain shift, which can non-monotonically influence the cost and objectives. In contrast, classic HPO benchmarks 72 ; 23 often assume a monotonic relationship and are limited to dataset size or training epochs. • The diverse costs of measuring every hyperparameter configuration, which can differ for LLM systems, are often limited in existing benchmarks. For example, the cost of LLM systems can include build time and the query tokens required to complete a set of tasks. To bridge this gap, this paper presents LLMSYS-HPOBench , the first live benchmark suite with measured hyperparameter configurations and their inference results over real-world LLM systems, together with other relevant data, e.g., hardware and logs. At the time of writing this paper, LLMSYS-HPOBench contains data collected by running and profiling seven LLM systems that are of various categories: LLM agent, retrieval-augmented generation (RAG) with LLM, and LLM inference engine, leading to 364,450 hyperparameter configurations with a dimensionality of 12–23, 3–5 fidelity factors of 932 settings, 2–9 objective metrics and 2–10 cost metrics, together with hardware metrics and generated logs from measuring the LLM systems. What we advocate to provide the AutoML community is not merely an efficient suite to compare the better or worse among the HPO algorithms for LLM systems, but also paving the way toward new research opportunities and directions. It is worth noting that preparing and collecting LLMSYS-HPOBench requires great efforts and resources: by May 2026, the data available are the results of running on 10 servers with \approx 2{,}277{,}600 CPU core-hours and \approx 94{,}900 GPU-hours. Yet, LLMSYS-HPOBench is a live benchmark suite where new data/LLM systems are continually incorporated. LLMSYS-HPOBench can be accessed at: https://github.com/ideas-labo/llmsys-hpobench . In the remainder of the paper, we start by presenting the preliminaries and discuss how LLMSYS-HPOBench differs from the related work in Section 2 . We specify the creation procedure of LLMSYS-HPOBench , together with its dataset formats and API access in Section 3 . This then follows a case study of using LLMSYS-HPOBench to run the well-known HPO algorithms, leading to some new insights in Section 4 . Finally, in Section 5 , we discuss how LLMSYS-HPOBench can assist in shaping the future research opportunities for AutoML community before concluding the paper. 2 Preliminaries 2.1 Black-box Hyperparameter Optimization Black-box hyperparameter optimization (HPO) aims to identify a hyperparameter configuration that optimizes target performance objective(s) when the objective function can only be accessed through evaluation of the model, without gradient information or exploitable structure assumptions. Formally, given a hyperparameter space \bm{\mathcal{X}} , the problem is defined as: \bm{x}^{*}=\arg\min_{\bm{x\in\mathcal{X}}}f(\bm{x})\text{ or }\bm{x}^{*}=\arg\max_{\bm{x\in\mathcal{X}}}f(\bm{x}), (1) where \bm{x}=(x_{1},x_{2},\dots,x_{n}) denotes a concrete hyperparameter configuration, and f(\bm{x}) represents the performance objective of interest, e.g., error or runtime. In general, \bm{\mathcal{X}} may comprise continuous, integer, ordinal, and categorical hyperparameters. This is a well-formulated problem that has been tackled by the AutoML community 23 ; 33 ; 18 and the configurable software/system community 52 ; 75 . Such a formulation naturally applies to LLM systems, whose behavior is governed by a rich set of tunable hyperparameters, such as model selection, prompting strategies, and memory mechanisms. A practical extension of black-box HPO is multi-fidelity hyperparameter optimization,