LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling
Xin Wang, Zhenhao Li *, Zishuo Ding * · 2025
Read LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling by Xin Wang, Zhenhao Li *, Zishuo Ding * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16070.txt.
Introduction
Xin Wang, Zhenhao Li, and Zishuo Ding test a new use for language models in a paper on software engineering released on arXiv in 2025. Modern software has huge numbers of settings, and its speed depends on them in tangled ways. Building a model of that behavior needs smart sampling, but classic samplers cannot read manuals or balance several goals at once.
The team built LLM4Perf, a feedback-driven framework where a language model picks which configurations to test. Across four real, highly configurable systems, the method won in 77 of 112 scenarios, about 68.8 percent. Its edge came from pruning the search space and refining strategy from feedback. That pruning even improved classic baselines in about 91.5 percent of cases. The paper maps which model and setting choices make the sampler work best.
Catalog record
| CREATOR | Xin Wang, Zhenhao Li *, Zishuo Ding * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling Conference: 48th International Conference on Software Engineering; April 2024; Rio De Janeiro, Brazil Thanks: * Corresponding authors. Xin Wang, Zhenhao Li *, Zishuo Ding * Abstract . The performance of modern software systems is critically dependent on their complex configuration options. Building accurate performance models to navigate this vast space requires effective sampling strategies, yet existing methods often struggle with multi-objective optimization and cannot leverage semantic information from documentation. The recent success of Large Language Models (LLMs) motivates the central question of this work: Can LLMs serve as effective samplers for multi-objective performance modeling? To explore this, we present a comprehensive empirical study investigating the capabilities and characteristics of LLM-driven sampling. We design and implement LLM4Perf , a feedback-based framework, and use it to systematically evaluate the LLM-guided sampling process across four highly-configurable, real-world systems. Our study reveals that the LLM-guided approach outperforms traditional baselines in most cases. Quantitatively, LLM4Perf achieves the best performance in nearly 68.8% (77 out of 112) of all evaluation scenarios, demonstrating its superior effectiveness. We find this effectiveness stems from the LLM’s dual capabilities of configuration space pruning and feedback-driven strategy refinement. The effectiveness of this pruning is further validated by the fact that it also improves the performance of the baseline methods in nearly 91.5% (410 out of 448) of cases. Furthermore, we show how the LLM choices for each component and hyperparameters within LLM4Perf affect its effectiveness. Overall, this paper provides strong evidence for the effectiveness of LLMs in performance engineering and offers concrete insights into the mechanisms that drive their success. 1. Introduction Modern software systems are highly configurable, offering users flexible control through a wide range of options, such as algorithmic choices, thresholds, and resource settings. While this configurability enhances adaptability, it also presents challenges to identifying configurations that ensure high performance. The combination of options leads to a combinatorial explosion in the configuration search space, where each configuration (i.e., a specific combination of option values) can have an unpredictable effect on key performance metrics like latency and memory usage, directly impacting the user experience. As a result, developers rely on performance testing to find effective configurations. However, exhaustively evaluating all combinations is computationally infeasible. This challenge is further compounded by complex trade-offs, where improving one objective may degrade another, making it difficult to deliver consistently high-performing systems. A common approach to managing configuration complexity is to build predictive models that estimate system performance based on a small sample of tested configurations. Research in this area has advanced along two directions. On one hand, efforts have focused on the predictive models themselves, where researchers have proposed a range of approaches, including traditional machine learning techniques, as well as deep learning models like DeepPerf ( 15 ) and Perf-AL ( 46 ) . On the other hand, researchers have recognized that the accuracy of these models depends heavily on the quality of the sampled configuration data. This has led to the development of various sampling strategies to guide configuration selection. For example, diversity-driven methods aim to maximize coverage of the configuration space ( 22 ) , search-based methods seek high-performing configurations ( 13 ; 27 ; 57 ) , and feedback-based approaches leverage feedback loops to iteratively refine the sampling process ( 49 ; 18 ; 56 ) . However, existing sampling methods have several limitations. First, they typically treat all configuration options as equally important, failing to leverage domain knowledge, such as developer documentation, to identify and prune performance-insensitive options. These options expand the configuration space used for sampling, thereby hindering the sampling effectiveness. Second, diversity-based strategies, though effective for ensuring broad coverage of the configuration space ( 22 ; 57 ) , may be less suitable for performance modeling tasks. In practice, different configurations may still exhibit similar performance, leading to redundant measurements. While feedback-driven strategies ( 36 ; 13 ; 56 ) address this by adapting the sampling process based on observed performance results, most are designed to optimize a single performance metric. This poses a challenge in real-world performance tuning, which often involves balancing complex multi-objective trade-offs, such as latency versus memory usage. Applying these single-objective methods in such cases requires running the entire sampling process separately for each metric, which is inefficient and fails to capture the interactions between different performance goals. Recent advances in LLMs have motivated extensive research into applying LLMs across a wide range of software engineering tasks, such as code generation ( 54 ; 58 ; 2 ; 3 ) , detecting software defects ( 17 ; 53 ) , program repair ( 52 ; 60 ) , as well as those related to performance analysis and optimization ( 55 ; 42 ; 41 ; 6 ; 43 ) . Inspired by this trend, we would like to explore whether the reasoning and generalization capabilities of LLMs can be leveraged to overcome the aforementioned limitations by incorporating domain knowledge, utilizing feedback, and supporting multi-objective optimization. To this end, we conduct a comprehensive study to assess the feasibility and effectiveness of using LLMs as configuration samplers for performance modeling. In this study, we design and implement LLM4Perf , a novel framework for LLM-driven configuration sampling. In LLM4Perf , the LLM analyzes performance documentation as domain knowledge to prune the configuration space and uses iterative feedback to refine the sampling strategy, thereby guiding the sampling toward more effective regions of the configuration space. The framework is specifically designed to handle multi-objective performance scenarios and aims to produce a sampling strategy that is both effective and efficient. Meanwhile, to support our evaluation, we construct a new multi-objective performance dataset based on four real-world systems. Unlike existing datasets, ours records performance metrics under both the full configuration space and a pruned space where performance-insensitive options are removed. This design allows us to examine the effects of configuration space pruning and assess the framework’s ability to handle multi-objective optimization. Our empirical evaluation, conducted using the LLM4Perf framework and our newly constructed dataset, addresses four key research questions: (i) Effectiveness (RQ1): We compare LLM4Perf against existing sampling baselines. The results show that both traditional machine learning model (i.e., XGBoost) and deep learning models (i.e., DeepPerf), when trained on configuration samples generated by LLM4Perf , generally achieve better performance across multiple metrics. (ii) Core Mechanisms (RQ2): We analyze the underlying factors contributing to LLM4Perf ’s effectiveness and identify two key sources: the LLM’s ability to prune the configuration space and its capability to iteratively refine the sampling strategy based on feedback. (iii) Impact of LLM Selection (RQ3): We evaluate the impact of each component’s LLM choice on the overall effectiveness of LLM4Perf . The results indicate that equipping reasoning - intensive components with high - capability LLMs improves overall effectiveness, whereas components with modest reasoning requirements can employ smaller models without perceptible degradation. (iv) Impact of Hyperparameter (RQ4): We investigate how LLM4Perf ’s hyperparameters settings influence its behavior. In summary, this paper makes the following contributions: • We propose LLM4Perf , an LLM-driven framework that leverages LLMs to prune the configuration space and adaptively guide the sampling process using iterative feedback. LLM4Perf is specifically designed to support multi-objective performance