Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference CCS: Hardware CCS: Power and energy CCS: Impact on the environment
Patrick Wilhelm, Thorsten Wittkopp, Odej Kao · 2026
Read Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference CCS: Hardware CCS: Power and energy CCS: Impact on the environment by Patrick Wilhelm, Thorsten Wittkopp, Odej Kao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20224.txt.
Introduction
Wilhelm, Wittkopp, and Kao argue that energy per token belongs beside accuracy in judging language model inference. Small models often answer simple requests well, at far lower cost than large ones. Reasoning tricks like chain of thought can lift small models toward large model quality. Those tricks also burn extra energy, creating a trade off to manage. Their MMLU analysis maps this energy and accuracy balance across test time strategies. They propose steering reasoning depth with operating curves. An energy aware router would then pick models and depth per request.
Catalog record
| CREATOR | Patrick Wilhelm, Thorsten Wittkopp, Odej Kao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27551 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference CCS: Hardware CCS: Power and energy CCS: Impact on the environment Patrick Wilhelm, Thorsten Wittkopp, Odej Kao Abstract . Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose energy efficiency metrics , including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment. 1. Introduction As AI models become more accessible and integrated into IT systems, sustainability and computational concerns are growing. Recent research shows, that the energy demand of global data centers is projected to reach 1,000 TWh by 2026 1 1 1 IEA Electricity Report 2024, https://www.iea.org/reports/electricity-2024 , fueled in part by the rapid expansion of AI technologies. Additionally, related carbon emissions are estimated to account for up to 8 % of global emissions in the next decade ( 5 ) . The energy demand of Large Language Models (LLMs) are primarily driven by their parametric size and computational requirements. Neural scaling laws, initially introduced by Kaplan ( 11 ) , offered a foundational framework for optimizing model performance by balancing model parameter size, dataset scale, and computational power, leading to a constant model parameter size increasement. However, these scaling laws often neglect inference, a phase with different and task-specific computational demands. Research from cloud providers such as AWS and Google confirms that inference frequently surpasses training in energy consumption, especially in high-demand, low-latency applications ( 22 ) . Although these frameworks have guided efficient training practices, only recent work accounts for the inference demand in the training phase, moving towards development of Small Language Models ( 23 ) . Recent research breakthrough in cost efficient LLM training by DeepSeek does not necessarily reduce resource consumption - instead, it often drives greater adoption and usage, known as the Jevons paradox. ( 4 ) . This emphasizes the need to evaluate computing efficiency during inference alongside traditional metrics like accuracy ( 17 ; 19 ; 18 ; 20 ) . One might expect that Energy Consumption per Token during inference would be consistent for models with identical parameter sizes. In this paper, we show that they are, in fact, not : LLMs reveal different energy efficiencies across the same tasks on the same hardware. Our key findings include the following: • Different LLM architectures have different energy efficiencies in token processing. • LLM computations are influenced by the transformer-based autoregressive generation of tokens. These dynamics produce characteristic nonlinear energy operating curves over the number of generated tokens across LLMs. • Test-time strategies like Chain-of-Thought Prompting boost accuracy in smaller models but come with high energy costs, making larger models more energy efficient in comparison. Our findings are based on several experiments, detailed throughout the remainder of this paper. After discussing the relevant related work in Section 2 , we provide a brief overview of the evaluated LLMs, the datasets, and the benchmarks used in Section 3 . In Section 4 , we investigate energy efficiency of LLM during inference, highlighting how prompts lead to differences in energy consumption between models. We further characterize these variations by analyzing the input-output token relationships and their impact on the hardware energy operating curves. Lastly, in Section 5 , we propose a routing design balancing accuracy and sustainability in LLM inference. 2. Balancing Compute in LLM Training and Inference Scaling laws provide a structured framework for optimizing the balance between model size, training data, and computational resources. Kaplan et al. ( 11 ) demonstrated that increasing model size significantly improves performance, shaping the development of large-scale language models. Building on this, Hoffmann et al. ( 8 ) refined these principles with the Chinchilla scaling laws, advocating for a proportional increase in model parameters and dataset size to achieve optimal training efficiency. While Chinchilla emphasized efficiency in pretraining, recent models such as LLaMA 2 and LLaMA 3 have adopted a different strategy, prioritizing extensive training token counts—2 trillion and 15 trillion tokens, respectively—over major architectural changes ( 27 ; 28 ) . This shift highlights a trade-off where higher training costs are offset by reduced inference costs and greater adaptability to deployment scenarios ( 23 ) . Additionally, emerging research suggests that further refinements to scaling laws may integrate considerations of both model quality and inference demands, optimizing LLM designs for specific operational requirements ( 23 ) . Optimizing LLMs requires balancing computational demands between training and inference. While early scaling approaches focused on maximizing model performance during training, recent research extends these laws to consider inference efficiency ( 23 ) . Compression techniques, including quantization and pruning, are commonly employed to mitigate inference costs. Quantization reduces model precision while maintaining accuracy ( 9 ; 16 ) , and pruning eliminates redundant parameters, lowering computational complexity ( 6 ; 26 ) . These techniques enable smaller, more efficient models that preserve performance while reducing resource demands. Moreover, advancements in test-time computation, such as dynamic inference optimization, continue to shape the trade-offs between training and inference efficiency, pointing to a future where these aspects become increasingly intertwined ( 31 ; 24 ) . Advances in inference-time optimization improve efficiency without requiring larger models. Chain-of-Thought (CoT) prompting enables stepwise reasoning, improving performance on complex tasks ( 35 ; 34 ) . Variants such as Majority Voting and Best-of-N further enhance accuracy by generating multiple responses and selecting the most probable one ( 15 ; 29 ; 33 ) . More advanced inference-time strategies dynamically allocate computational resources. Beam Search expands multiple reasoning paths in parallel, selecting the most probable sequence based on cumulative likelihood. Monte Carlo Tree Search (MCTS) further refines this by exploring solution paths, evaluating their expected quality, and backpropagating optimal results ( 3 ; 32 ; 37 ) . Despite these improvements, most studies do not report computational costs or energy usage during inference, leaving a critical gap in understanding inference efficiency. The absence of standardized metrics for inference-time compute further complicates direct comparisons between different optimization techniques. Transformers, the backbone of LLMs, pose additional computational challenges. The self-attention mechanism scales quadratically with sequence length, increasing memory and energy consumption ( 30 ; 21 ) . Sequential decoding exacerbates this problem, as token generation requires recurrent attention to past outputs, leading to significant computational overhead. Efforts to