Guidelines for Empirical Studies in Software Engineering involving Large Language Models
Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner · 2025
Read Guidelines for Empirical Studies in Software Engineering involving Large Language Models by Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15503.txt.
Introduction
Sebastian Baltes and co-authors write the rulebook for studying software with large language models, in a 2025 guidelines paper. Language models now appear throughout software engineering research, but their randomness, hidden training data, and rapid updates make studies hard to repeat. Twenty-two researchers joined forces to organise the field. They sorted studies into seven types, by how the model is used, and wrote eight guidelines for designing and reporting them.
Each guideline separates firm requirements from softer recommendations and says which study types it binds. The rules start with declaring the model's role, then demand reporting of versions, settings, and any fine-tuning. They cover documenting prompts and system design beyond the model itself. The authors aim to rescue reproducibility before a literature of unrepeatable results hardens.
Catalog record
| CREATOR | Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 132 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15503