Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models † \dagger

Soham Poddar*, Paramita Koley*, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh · 2025

Read Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models † \dagger by Soham Poddar*, Paramita Koley*, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05610.txt.

Introduction

Six researchers benchmark the energy cost of running large language models in a 2025 paper. Training costs draw most attention, yet inference runs constantly in deployment and its energy bill is poorly mapped. The team measured energy across models, tasks, prompts, and system settings.

Energy tracked output token length and response time closely. Quantization, tuned batch sizes, and targeted prompt phrases each cut usage sharply. The authors call the study the first broad benchmark of inference energy across so many factors. They close with deployment advice for lower energy operation. The findings frame inference, not just training, as a design target for sustainable language technology.

Catalog record

CREATORSoham Poddar*, Paramita Koley*, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05610

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models † \dagger Soham Poddar*, Paramita Koley*, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh Abstract Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment. † † \dagger This paper has been accepted at NAACL main 2025. † † * Equal Contribution 1 Introduction Fine- Top- Task Response Model size Batch Quanti- Targeted grained I/O level I/O Complexity time & family size zation phrase Our Work Yes Yes Yes Yes Yes Yes Yes Yes Luccioni et al. (2024) No Yes No No Yes No No No Li et al. (2024) No No No No No No No Yes Everman et al. (2023) No No No No Yes No No No Desislavov et al. (2021) No No No No Yes No No No Samsi et al. (2023) No No No No Yes No No No Table 1: Comparing our approach with existing literature on energy for specific experimental settings Recent discussions on the energy and carbon impact of machine learning (ML) algorithms have mainly concentrated on quantifying energy usage during the training phase of these models Dodge et al. (2022) ; Luccioni et al. (2023) ; Patterson et al. (2021) ; Raffel et al. (2020) . Studies on inference energy are much rarer because a single inference operation consumes considerably less energy and resources. However, under deployment, inference is performed many more times, making its energy impact significant and warranting separate investigation Wu et al. (2022) ; Patterson et al. (2022) . For example, 90% of total cloud computing demand for AWS, the largest global cloud provider, were for model inference purpose Barr (2019) . Moreover, a key motivation for training large language models is that a single model can achieve state-of-the-art performance across diverse NLP tasks due to its impressive zero-shot and few-shot capabilities, making it energy-efficient from a training perspective. However, when we consider the total carbon footprint of the entire lifetime of the model, the energy requirement for model inference plays a significant role, considering the number of inferences that are carried out during the model’s lifetime. Thus it is crucial to conduct a systematic study to quantify the energy requirements and carbon emissions for model inference across various models and tasks. 1.1 Literature survey Existing works have attempted to study the inference energy of LLMs from various perspectives. Everman et al. (2023) evaluated energy usage of GPT models on question-answering tasks, employing Software Carbon Intensity (SCI) released by green software. Samsi et al. (2023) provides a detailed account of energy usage during inference for question-answering tasks for Llama models on various GPU architectures. Liu et al. (2022) study the energy consumption trade-off for fine-tuning vs few-shot learning for both encoder-decoder and decoder-only models on SuperGLUE and RAFT tasks, measuring energy in FLOPS. These preliminary works mostly ignore the performance-energy tradeoff. In recent times, Li et al. (2024) compare the effect of prompt directives on inference energy and performance for Llama-2 models for 3 question-answering tasks. Luccioni et al. (2024) offers a comparatively detailed account of inference energy usage while choosing LLMs and tasks from a diverse range, showing the dependency of energy with model size and architecture, task type, and output token length. While these works provide an initial account of inference energy usage of LLMs in different configurations, they are often limited by the number and diversity of the models and tasks (Check Appendix A for details). 1.2 Our contributions & differences with prior works In this work, we present a comprehensive study of LLMs, running models from both encoder-decoder and decoder-only models on both discriminative and generative NLP tasks, while analyzing the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, (i) We start with a detailed analysis of various model-related factors that affect the inference energy of LLMs, where we systematically study the correlation between inference energy and influencing factors like input and output token length, response time, model size and complexity. (ii) We conduct various experiments to study the connection between inference energy and batch size, level of quantization, and prompt editing. (iii) We then complement our analysis by introducing the Normalized Accuracy metric, providing an accuracy-energy usage tradeoff analysis across tasks and models. (iv) Finally, we present a list of efficiency guidelines in Section 4 . In particular, our contributions include an energy analysis from eight aspects, as given in Table 1 . Out of these aspects, several were primarily unexplored in the literature, as per our knowledge - these are indicated in the table. We vary the input and output token length at a finer granular level while reporting the variation in energy to have a better understanding of the correlation between energy and these factors, revealing several valuable insights (Section 3.2 ). A more thorough analysis of energy with variation of input or output while keeping the other fixed provides a more accurate insight into their comparative influence in inference energy, implying immediate solutions for energy-efficient inference approaches. The correlation between energy and response time validates response time as a good proxy of inference energy in black-box models (Section 3.1 ). Our work first explores task complexity’s effect with inference energy (Section 3.3 ). We also provide the performance change for quantization (Section 3.5 ) and use of targeted phrases (Section 3.6 ) along with improvement in energy. While the effects of quantization and increasing batch size have been studied individually, we show that using them together under fixed memory constraints can help speed up the computations. Finally, we explore each of the above aspects for a diverse range of LLMs from both architecture families along with a diverse range of tasks, which was mostly lacking in previous attempts. 2 Experimental setup In this section, we describe the models, datasets and various settings we use for our experiments. 2.1 Models We select 6 popular and recent GPT-style models from the decoder-only family and 4 Flan-T5 models from the encoder-decoder family, adding to 10 models in total (details in Appendix B ). Decoder-only Models generate output in an autoregressive manner by predicting the next token in the sequence based on the context (key-value-query) vectors corresponding to the input and previously generated tokens. We consider the following models from decoder family in our study. (D1) Tiny-LLama (1.1B params); (D2) Phi-3-mini (3.8B params); (D3) Mistral-7B (7.2B params);

Read the complete text