An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Automated software engineering CCS: Computing methodologies Artificial intelligence CCS: Hardware Power and energy
Pragati Kumari, Novarun Deb · 2026
Read An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Automated software engineering CCS: Computing methodologies Artificial intelligence CCS: Hardware Power and energy by Pragati Kumari, Novarun Deb (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02754.txt.
Introduction
Kumari and Deb measure the environmental cost of prompt-driven test script generation with small language models, a topic prior work largely ignores in favor of large models. Using HumanEval, adaptive prompt variants, and CodeCarbon tracking, they compare models of 2 to 8 billion parameters on energy, carbon, duration, and test coverage. Different models show distinct profiles: some save energy and run fast, others hold steadier coverage.
Catalog record
| CREATOR | Pragati Kumari, Novarun Deb |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30496 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Automated software engineering CCS: Computing methodologies Artificial intelligence CCS: Hardware Power and energy Pragati Kumari, Novarun Deb Abstract . The increasing use of language models in automated test script generation raises concerns about their environmental impact, yet existing sustainability analyses focus predominantly on large language models. As a result, the energy and carbon characteristics of small language models (SLMs) during prompt-driven unit-test script generation remain largely unexplored. To address this gap, this study empirically examines the environmental and performance trade-offs of SLMs (in the 2B–8B parameter range) using the HumanEval benchmark and adaptive prompt variants (based on the Anthropic template). The analysis uses CodeCarbon to characterize energy consumption carbon emissions and duration under controlled conditions, with unit-test script coverage serving as an initial proxy for generated test quality. Our results show that different SLMs exhibit distinct sustainability profiles: some favor lower energy use and faster execution, while others maintain higher stability or coverage under comparable conditions. Overall, this work provides focused empirical evidence on sustainable SLM-based test script generation, clarifying how prompt structure and model selection jointly shape environmental and performance outcomes. 1. Introduction The integration of Large Language Models (LLMs) into automated software testing has significantly advanced test script generation, bridging semantic gaps that traditional symbolic or search-based techniques often struggle to address. However, the deployment of massive foundation models incurs a substantial “ environmental tax ” due to high energy consumption and carbon intensity during inference. While most sustainability research in Software Engineering (SE) focuses on these large-scale models, the environmental characteristics of Small Language Models (SLMs)—specifically those in the 2B–8B parameter range—remain under-explored. In resource-constrained environments or privacy-sensitive domains where local execution is preferred, understanding the trade-offs between the energy efficiency of SLMs and their test generation performance is critical for sustainable development. The primary objective of this research is to empirically evaluate the sustainability-to-performance ratio of SLMs in prompt-driven unit-test script generation. We seek to move beyond traditional accuracy metrics by establishing how prompt engineering and quantization levels influence the physical resource costs of test generation. By utilizing SLMs, we prioritize scientific reproducibility and local control over the computational footprint, addressing the growing need for Green AI practices within the software testing lifecycle. Our investigation is guided by the following research questions: • RQ-1: How do different prompt structures and the geographical migration of workloads across regional power grids (with varying carbon intensities) jointly affect the energy consumption (Wh) and the overall carbon footprint (gCO 2 eq) of SLMs during the generation of unit-test scripts? • RQ-2: Is there a holistic mechanism to rank SLMs for unit-test script generation such that both code coverage and environmental overheads are taken into account? • RQ-3: What are the trade-offs between test coverage and environmental sustainability when employing varying quantization levels (4-bit vs. 8-bit vs. unquantized) across SLM architectures? • RQ-4: Can we give software solution architects a mechanism to prioritize between test coverage and environmental sustainability while choosing an SLM for unit-test script generation? As a direct outcome of this empirical exploration, we define and introduce two novel trade-off metrics: the Sustainability Velocity Index (SVI) and the Green F_{\beta} score (GF β ). SVI measures overall sustainability performance, favoring models that are fast, stable, low-emission, and broadly effective in terms of code coverage. Its multiplicative design prevents high performance in one area from fully compensating for poor performance in another, promoting balanced eco-efficiency. The GF β score is a composite index that balances environmental impact and code coverage, where tuning \beta allows practitioners to prioritize eco-efficiency ( \beta<1 ) or code coverage ( \beta>1 ). This empirical analysis revealed that sustainability in unit-test script generation is not merely a function of model size, but is significantly shaped by the interplay between prompt complexity, model quantization, and regional grid carbon intensity. We demonstrate how our proposed SVI and GF β metrics can be used to rank SLMs, identifying configurations that achieve high coverage with minimal environmental impact. The novelty of this work lies in its specific focus on SLMs, its multi-regional carbon analysis, and the introduction of energy-normalized performance metrics. It provides actionable evidence for selecting model-prompt configurations that balance testing efficacy with environmental responsibility in a global context. The remainder of the paper is structured as follows: Section 2 documents the related works and gaps in the existing literature. Section 3 presents the empirical study pipeline, followed by Section 4 that documents the experimental setup, results and discussion. Section 5 discusses the threats to validity of the proposed study. Finally, Section 6 concludes the paper. 2. Related Work Recent work has explored sustainability in prompting and code generation. Studies such as ( Wu et al. 2023 ; Cappendijk et al. 2025 ) introduced prompt-based techniques to reduce emissions at inference time. These works evaluated structured prompts across various language models (e.g., GPT-3, Code Llama, DeepSeek-Coder) and demonstrated that prompt phrasing can impact energy usage without compromising output quality. In addition, the influence of prompting patterns on sustainability and model efficiency has been examined ( Oprescu et al. 2023 ) , offering structured perspectives on energy-aware prompt engineering. Efforts to model and monitor carbon emissions throughout the lifecycle of language models are presented ( Deng et al. 2024 ; Luccioni et al. 2023 ) . The corresponding tools and frameworks, including LLMCarbon and ML Bloom, provide lifecycle-aware and real-time tracking of emissions, enhancing transparency and enabling sustainability-informed development decisions. The environmental impact of automated test script generation has been assessed ( Li et al. 2023 ; Sharma et al. 2023 ) . These studies analyzed energy usage patterns using models such as CodeT5, GraphCodeBERT, and quantized SLMs, highlighting key considerations in emission-aware testing. Work such as ( Fraser and Arcuri 2011 ) introduced EvoSuite, an evolutionary testing tool targeting branch coverage, though without a focus on energy consumption. Two key reviews ( Lago et al. 2014 ; Mourão et al. 2018 ) classified green software metrics and sustainability practices, emphasizing the absence of integrated sustainability metrics in early phases of the software lifecycle and advocating for tool-supported green engineering methodologies. Further extensions to sustainability-focused quality frameworks have been proposed ( Kapoor 2024 ) , incorporating maintainability and energy efficiency into software development processes. Sustainability in large language models has also been examined from operational, deployment, and educational perspectives. Studies such as ( Iyer et al. 2023 ; Verdecchia et al. 2021 ; Vartziotis et al. 2024 ) analyzed real-world usage patterns, prompting strategies, and organizational interventions aimed at improving energy and emissions awareness in cloud-hosted AI and code generation contexts. Broader industry guidance is provided through initiatives such as ( Green Software Foundation 2021 ) , which promote best practices and standards for carbon-aware software development. Foundational analyses of energy and carbon costs in NLP tasks were presented ( Strubell et al. 2019 ; Rashkin et al. 2021 ) , establishing relationships between batch size, model configuration, task