Reflecting on Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models Conference: arXiv Preprint; 2025; Online arXiv Preprint

David Williams, Max Hort, Maria Kechagia, Aldeida Aleti, Justyna Petke, Federica Sarro · 2025

Read Reflecting on Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models Conference: arXiv Preprint; 2025; Online arXiv Preprint by David Williams, Max Hort, Maria Kechagia, Aldeida Aleti, Justyna Petke, Federica Sarro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26538.txt.

Introduction

David Williams, with five co-authors, reflects on how software engineering research is coping with large language models. Studies that use these models face new worries: benchmark quality, contamination, where test data leaks into training, hard replication, and the money and energy costs of running big models.

Williams and the team survey current practice at ICSE, a leading venue in the field. They give a structured picture of what researchers are doing well and where gaps persist. Their closing advice aims to firm up benchmark rigour and make studies easier to repeat. It also asks the field to face the financial and environmental costs of model-based research head-on, rather than treating them as side notes.

Catalog record

CREATORDavid Williams, Max Hort, Maria Kechagia, Aldeida Aleti, Justyna Petke, Federica Sarro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26538

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reflecting on Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models Conference: arXiv Preprint; 2025; Online arXiv Preprint David Williams, Max Hort, Maria Kechagia, Aldeida Aleti, Justyna Petke, Federica Sarro Abstract . Software Engineering (SE) research involving the use of Large Language Models (LLMs) has introduced several new challenges related to rigour in benchmarking, contamination, replicability, and sustainability. In this paper, we invite the research community to reflect on how these challenges are addressed in SE. Our results provide a structured overview of current LLM-based SE research at ICSE, highlighting both encouraging practices and persistent shortcomings. We conclude with recommendations to strengthen benchmarking rigour, improve replicability, and address the financial and environmental costs of LLM-based SE. 1. Introduction Large Language Models (LLMs) are now being integrated into software development practices, influencing safety-critical applications, projects, and codebases. As these systems become more prevalent, Software Engineering (SE) researchers are uniquely positioned to address challenges and opportunities across the entire software lifecycle. In fact, despite the potential to augment human capabilities and streamline software development workflows, the integration of LLMs into software systems raises critical concerns. For instance, benchmarking should be performed systematically to ensure fair evaluation, particularly in light of potential data leakage from training corpora that can skew performance metrics ( Sallou et al. 2024 ) . Reproducibility and replicability should become mandatory to address the stochastic nature of LLMs and the lack of standardised evaluation protocols ( Liang et al. 2024 ) . Further efforts should be made to reduce the computational costs associated with training and inference – both in terms of energy and infrastructure – which pose significant barriers to accessibility and sustainability ( Hort et al. 2023 ) . As LLM-based SE becomes increasingly prevalent, addressing these challenges is crucial to harnessing the full potential of LLMs while maintaining scientific rigour, ethical standards, and equitable access. To understand how our community is tackling the aforementioned challenges, we conduct an initial analysis of the empirical research literature published in the research track of the top SE conference, the International Conference on Software Engineering (ICSE), between 2023 and 2025. We identified 177 relevant papers and manually extracted several key pieces of information, including the SE tasks addressed, the types of models used, the experimental protocols, the benchmarks, and other details relevant to empirical studies. As expected, the proportion of published papers at ICSE that include an empirical evaluation using LLMs has grown dramatically, from 15.2% of accepted papers in 2023 to 36.6% in 2025. We invite the community to reflect on crucial empirical and ethical aspects of current SE research involving LLMs. 2. Research Questions We analyse relevant ICSE papers to establish: RQ1. Models & Benchmarking Which LLMs are used in SE research and how are they benchmarked? We want to understand whether current studies provide reliable evidence of improvement over non-LLM techniques, and to highlight gaps where more rigorous evaluation protocols are needed. We check (i) whether open or closed models are used, (ii) which model families are chosen, (iii) whether comparisons include non-LLM baselines, and (iv) which programming languages are targeted. RQ2. Contamination. How well do authors tackle the problem of data leakage/contamination? Data leakage and contamination directly threaten the validity of empirical findings. If evaluation datasets overlap with model pre-training data, performance may reflect memorisation rather than genuine reasoning capabilities. We check i) whether authors mention data contamination as a threat to the validity of the results and ii) the strategy proposed to mitigate contamination. RQ3. Replicability. How replicable are LLM-based studies? Replicability underpins scientific credibility. LLM-based SE research is challenged by the stochastic nature of model outputs, opaque APIs, and frequent model updates. We check (i) whether LLM configuration details are reported and (ii) whether artefacts are shared and recognised with a badge. RQ4. Sustainability. What are the costs of LLM-based SE research? Training and deploying LLMs is resource-intensive, with high financial and environmental costs. By examining how sustainability is addressed, we aim to raise awareness of these hidden costs and encourage the community to adopt more transparent, responsible practices that balance innovation, accessibility, and environmental responsibility. We check (i) which experimental costs are reported, (ii) the costs of LLM-based SE research, and (iii) how they are sustained. 3. Methodology To answer our RQs, we used a mixed-methods approach, integrating quantitative and qualitative data gathered through a literature review and user survey. We incorporate temporal analyses, where possible, to analyse how trends have shifted. Article Search. The study is conducted on the 692 articles published at the technical research track of ICSE 2023, 2024, and 2025. As the premier venue in SE, ICSE sets the benchmark for methodological rigour and research quality in the field. Consequently, the insights we draw from this body of work can be regarded as a reliable reflection of the community’s current priorities and practices. Moreover, these years coincide with the widespread advent of LLMs, following the release of ChatGPT in late 2022. The initial list of 692 papers was filtered based on the following list of keywords: “ LLM ”, “ language model ”, “ conversational AI ”, “ chatbot ”, “ genAI ”, “ generative ”, “ AI-assisted ”, “ foundation model ”, “ BERT ”, “ encoder ”, “ decoder ”, “ autoencoder ”, “ transformer ”, and “ agent ”. A title and abstract search using these terms returned 304 articles from the initial 692. Among these 304, we manually identified relevant publications matching the following inclusion criteria: articles that present an empirical experiment involving at least one LLM, either as the main approach or as a baseline . For example, we exclude papers focusing solely on qualitative insights derived from LLM user studies. When referring to LLMs, we use the classification proposed by Fan et al. ( Fan et al. 2023 ) , namely encoder-only (e.g., BERT, DeBERTa, RoBERTa), encoder-decoder (e.g., T5, BART), and decoder-only (e.g., GPT, LLaMA, Claude). We followed a three-step process, widely adopted in previous surveys (e.g. ( Martin et al. 2016 ; Hort et al. 2021 ; Moussa and Sarro 2022 ) ): i) We first excluded all publications whose titles clearly did not meet our inclusion criteria; ii) next, we judged inclusion based on the abstract; iii) finally, we reviewed the content of all remaining articles from the previous two steps and excluded those that did not meet our inclusion criteria. This process resulted in a total of 174 articles meeting the inclusion criteria and thus selected for data extraction. To mitigate the risks of missing relevant articles, we (a) checked whether we missed any keywords in our search by computing word clouds from the abstracts of these 174 articles (see ( Williams et al. 2025 ) for full results), finding no additional relevant keywords were missing; (b) randomly sampled 20% of the 338 papers not selected by our keyword search for each year. This comprised a total of 80 papers (32 for 2023, 28 for 2024, and 20 for 2025), which we manually inspected using the process described above. We found that only three of these articles were false negatives and subsequently included them in the survey. The resulting number of relevant papers we

Read the complete text