LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference
Katelyn Crumpacker, Dimitrios Nikolopoulos · 2026
Read LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference by Katelyn Crumpacker, Dimitrios Nikolopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27032.txt.
Introduction
Crumpacker and Nikolopoulos build a human-in-the-loop flow for tuning language-model inference settings to cut energy use. Picking runtime values usually needs deep app knowledge or slow searches taking days. With human-crafted feedback prompts, chat-based models iteratively find energy-efficient settings faster than classic search, tailoring answers to hardware and other system limits. The enhanced prompt template converged below threshold in fewer prompts than the baseline, with lower final energy per token, and it beat Sobol sampling on speed.
Catalog record
| CREATOR | Katelyn Crumpacker, Dimitrios Nikolopoulos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25589 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference Katelyn Crumpacker, Dimitrios Nikolopoulos Abstract Large Language Models (LLMs) have become an integral part of many real-world workflows. However, LLMs consume a lot of energy, which becomes a large concern in the scale of the demand for these tools. As LLMs become integrated into different workflows, different applications have arisen to deal with the challenge of running inference for these tools. This raises another issue of choosing the runtime parameter values for these services in order to minimize the energy consumption. Oftentimes this requires deep knowledge of the application or traditional optimization methods that can take days to find optimal values. In this work, we created a human-in-the-loop flow with LLM-assisted runtime parameter optimization in order to solve this issue. With human-created, specific feedback prompting methods, chat-based LLMs can iteratively find energy-efficient inference parameters faster than traditional search methods. LLMs can also tailor their solutions to different hardware setups and easily take into account other system constraints. The enhanced prompt template was able to converge below the threshold at an average of 3.4 prompts compared to the baseline, which converged in an average of 5.2 prompts, and consistently achieved lower final energy per token. The enhanced prompt template also outperformed Sobol sampling in convergence speed. 1 Introduction LLM-Assisted Runtime Optimization Framework: We propose a feedback-driven framework that leverages LLMs to iteratively optimize runtime parameters for energy-efficient LLM inference. Energy-Aware Prompt Encoding Strategy: We design prompt templates using specific prompting techniques that improve LLM performance in the proposed optimization task. Cross-Backend Empirical Evaluation: We evaluate the proposed approach on two inference systems, vLLM and PyTorch-based deployments, demonstrating generality. Reduced Search Overhead: We show that LLM-guided tuning converges to near-optimal energy per token configurations with fewer evaluations than black-box optimization techniques. 2 Related Work 2.1 Energy-Efficient Model Serving Systems Runtime parameter optimization is one method to improve the energy efficiency of LLM inference; there are other techniques that can be used alongside or instead of it. One such solution is ELLIE, which dynamically maps inference phases across heterogeneous hardware for energy reduction. It will make these decisions per prompt and per model, resulting in significant decreases in energy consumption [ 3 ] . This solution is very effective; however, it can be hardware-dependent or complicated to include in the inference workflow. In a paper by Stojkovic et al., multiple methods, related to techniques like input length and batching, that decrease energy consumption in inference are presented [ 6 ] . 2.2 Runtime Parameter Optimization There have been other researched methods in parameter optimization. Bergstra and Bengio provide an argument for random search, as they found it can exceed or match grid search with less computation [ 2 ] . Wang et al. propose EcoOptiGen, combining Bayesian optimization with cost-aware pruning for LLM inference [ 7 ] . Zhang et al. use an LLM as the optimizing tool for machine learning models, achieving competitive performance with Bayesian optimization [ 10 ] . However, their work focuses on optimizing machine learning model hyperparameters rather than runtime parameters of inference systems. 3 Design The goal of our design is to evaluate whether LLMs can guide runtime parameter optimization for energy-efficient LLM inference under realistic deployment constraints. We use the LLM as an adaptive decision agent, iteratively proposing configurations based on observed system feedback rather than relying on exhaustive search. To validate the LLM’s ability to optimize runtime parameters for inference engines, we selected vLLM to run on a single 16GB GPU for text-based prompts. vLLM is memory-efficient, which allows for a larger search space. For further validation, we used PyTorch for image and text workloads. PyTorch and vLLM exhibit complex cross-layer interactions between runtime scheduling, memory usage, and computation that motivate an LLM-guided black-box optimization strategy. Our search space includes parameters that fall into four categories: (i) memory utilization, (ii) batch and execution control, (iii) precision and computation, and (iv) hardware constraints. We selected these because they influence energy consumption through their effects on GPU occupancy, device utilization, computational cost, and power limits. Although evaluated on vLLM and PyTorch, we designed the framework to remain runtime-agnostic and generalizable to other inference systems. 3.1 LLM-Guided System-Level Parameter Optimization The system involves using an LLM to improve inference of other LLMs. LLMs maintain context across iterations, allowing them to incorporate prior information when proposing new configurations [ 9 ] . This differs from search methods like Sobol sampling. We chose Sobol sampling as a baseline comparison because it deterministically finds configurations that cover the search space uniformly. We chose Claude as the online LLM because it is freely accessible via a web interface. Claude offers a project feature for chats to have isolated memory [ 1 ] . Figure 1: Human-in-the-loop workflow for optimizing runtime inference parameters using an external LLM. Energy metrics from each inference run are used in subsequent prompts to guide parameter exploration. 3.2 Prompting Construction and Feedback Encoding We developed a workflow (Figure 1 ) that includes a search space, human-in-the-loop prompting of LLMs, and energy metric collection. We incorporated human oversight to monitor proposals and enforce deployment constraints [ 4 ] . We constructed prompts based on energy metrics of the previous configuration to guide the LLM toward more energy-efficient parameter configurations. We tested prompt techniques to determine which combinations helped the LLM produce the most effective results. We explored repetition, along with adding background information in the first prompt, which the LLM can use throughout iteration. We also tested structured versus unstructured prompts [ 8 ] . For the structured prompt strategy, we used search-process constraints in guiding the LLM and keeping its focus on a single task (enhanced prompt template). We also experimented with unstructured prompts that leave the LLM open to focus on more multi-task optimizations (baseline prompt template). 4 Implementation Figure 2: System architecture for measuring energy consumption during LLM inference. An experiment runner executes prompts through vLLM while a PyNVML-based script samples GPU power to compute energy metrics. Parameter configurations are provided from the defined search space or suggested by an external LLM. 4.1 System Setup Figure 2 shows the end-to-end system architecture used in the experiments to see the effect prompts have on the LLM’s ability to generate candidate runtime parameters. 4.1.1 Hardware and Software Stack We collected data on a server with a single NVIDIA Tesla V100 PCIe GPU with 16 GB of device memory. We set the gpu-memory-utilization flag to 0.55 to reduce excessive OOM errors. vLLM serves the decoder-only LLM used for inference. The client concurrently issues inference requests, while batching decisions are handled entirely by vLLM at the token level. The system runs on top of PyTorch with CUDA-enabled kernels. An experiment runner orchestrates the execution loop. This runner is responsible for loading prompt data, configuring inference parameters, issuing batched inference requests to vLLM, and logging measurements in a .csv file. 4.1.2 Model and Dataset We evaluate Llama-3.2-3B served through vLLM. Inference requests are generated from a fixed dataset of 1,000 prompts. We reuse the dataset across all experimental runs to ensure comparability across different parameter configurations. 4.1.3 Energy Measurement The recorded metrics are total energy consumption in joules (J), wall time in seconds