Guidelines for Empirical Studies in Software Engineering involving Large Language Models

Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner · 2025

Read Guidelines for Empirical Studies in Software Engineering involving Large Language Models by Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15503.txt.

Introduction

Sebastian Baltes and co-authors write the rulebook for studying software with large language models, in a 2025 guidelines paper. Language models now appear throughout software engineering research, but their randomness, hidden training data, and rapid updates make studies hard to repeat. Twenty-two researchers joined forces to organise the field. They sorted studies into seven types, by how the model is used, and wrote eight guidelines for designing and reporting them.

Each guideline separates firm requirements from softer recommendations and says which study types it binds. The rules start with declaring the model's role, then demand reporting of versions, settings, and any fine-tuning. They cover documenting prompts and system design beyond the model itself. The authors aim to rescue reproducibility before a literature of unrepeatable results hardens.

Catalog record

CREATORSebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE132 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15503

Browse the full library · About the Tower of Babel