A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering

Matteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, Valentina Lenarduzzi · 2025

Read A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering by Matteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, Valentina Lenarduzzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12728.txt.

Introduction

Six authors wrote a paper in two thousand twenty-five calling for reform in how software engineering research handles data. The field relies on experiments and statistics, yet worries about misapplied methods date back decades. The authors set out to measure the problem and to test whether experts can spot it.

They surveyed about twenty-seven thousand empirical studies, using language models to class their statistics as sound or not. They also gave thirty primary studies to thirty-three experts in a workshop. Serious statistical flaws showed up in the studies, and the experts caught and fixed few of them, raising concern about the field's statistical skills.

Catalog record

CREATORMatteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, Valentina Lenarduzzi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12728

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering Matteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, Valentina Lenarduzzi Abstract Context. Empirical Software Engineering (ESE) is the core of innovation in the broader SE fields via its qualitative and quantitative empirical studies. Albeit empirical methodologies have driven the advances in SE practices and research, concerns over their correct application have sparked as early as 2006 in the Dagstuhl seminar on SE. So far, advancements would be expected since some previous studies have investigated it in specific software engineering areas, highlighting the misconceptions about using inadequate statistics strategies to analyze data. However, it is unclear how it has evolved since then and how it could influence confidence in decision-making based on the results of primary studies. Objective. To analyze three decades of SE research and look for eventual mistakes in choosing adequate statistics strategies to support the data analysis, i.e., flawed statistical techniques for analyzing data reported in the technical literature. Besides, it will evaluate current experts’ accuracy in detecting and addressing these issues. Methods. To perform a literature survey (pilot study) in the technical literature to collect a large sample of empirical studies, i.e., circa 27k works, reporting data analysis, and filtering them using LLM to identify adequate (“good”) and inadequate (“bad”) statistical methodological patterns. To randomly select 30 primary studies (15 good and 15 bad) and conduct a focus group-based workshop with 33 Empirical Software Engineering experts to observe their capabilities in identifying and remediating those methodological statistics issues. Results. Our findings reveal widespread statistics issues within the published primary studies regarding empirical software engineering studies. Besides, the performance of the experts in identifying eventual mistakes in the use of statistical methods, models, and tests in these primary sources raises serious concerns about the general ability of empirical software engineering researchers to spot them and suggest proper adjustments. Conclusions. Despite our study’s eventual limitations, its results shed light on recurring issues from promoting information copy-and-paste from past authors’ works and the continuous publication of inadequate approaches that promote dubious results and jeopardize the spread of the correct statistical strategies among researchers. Besides, it justifies further investigation into empirical rigor in software engineering to expose these recurring issues and establish a framework for reassessing our field’s foundation of statistical methodology application. Therefore, this work calls for critically rethinking and reforming data analysis in empirical software engineering, paving the way for our work soon. 1 Introduction The term “Software Engineering” first emerged in the mid-20th century, with early usages including a 1965 letter from ACM president Anthony Oettinger ( Meyer, 2013 ; Tedre, 2014 ) and lectures by Douglas T. Ross at MIT in the 1950s ( Mahoney, 1990 ) . These pioneers recognized the need for a disciplined approach to software development. Margaret H. Hamilton, when working on the Apollo Guidance Computer, coined the term “software engineering” (SE) to denote the rigor and importance of the practices involved ( Rayl, 2008 ; Hamilton, 2018 ) . SE was popularized during the 1968 NATO Software Engineering Conference, which addressed the “software crisis”, i.e., the challenges of developing complex software systems that were over budget, behind schedule, and low-quality ( Hack Reactor, 2020 ) . In response to these challenges, researchers recognized the need for a more scientific approach to software development. This led to adopting empirical methods, such as controlled experiments, case studies, and surveys, to systematically evaluate software processes and tools. Victor Basili, an empirical software engineer pioneer, advocates for using empirical studies to build a science of computer science ( Basili et al., 1994 ) . Over the decades, Empirical Software Engineering (ESE) has evolved to encompass various research methods, including quantitative and qualitative approaches Felderer and Travassos (2020) . This evolution reflects a growing recognition of the importance of empirical evidence in understanding and improving software engineering practices. However, early warnings regarding the uncritical adoption of empirical methodologies (EM), including the tendency to replicate methodological issues through widespread copy-paste practices, were raised as early as 2006 during the Dagstuhl seminar on Software Engineering ( Basili et al., 2007 ) . The legacy of such a seminar was considered in Reyes et al. (2018) , where they conducted an empirical study to assess the extent of statistical errors in software engineering experiments. The authors compiled a list of frequent statistical mistakes from various experimental disciplines and systematically reviewed all papers published at ICSE. Finally, in Vegas and Elbaum (2023) , they analyzed pitfalls in experiments with Deep Neural Networks (DNNs) in SE. Vegas (2024) warned the community that specific methodological issues could push SE to the situation faced by psychology, i.e., a reproducibility crisis. Stemming from the daunting need to verify and evaluate the degree to which our field falls into a methodological crisis, we investigate using methodological statistical strategies in ESE studies, distinguishing between adequate patterns that adhere to best practices and inadequate patterns. On the other hand, we evaluate the experts’ inadequate methodological pattern detection and remediation capabilities. The study draws on a rich corpus of ESE literature spanning the last 30 years (1994–2023). These studies encompass various domains and methodologies, reflecting the evolution of empirical data analysis practices in ESE. Therefore, we split our study into two sub-studies. To evaluate the actual trends over the year, we devised a pilot study that surveys a portion of the state of the art. Based on 60.381 original studies, our pilot findings alarmingly highlighted many works with methodological issues published in the last 30 years. Stemming from these findings, we conducted a workshop with 33 ESE specialist to evaluate their abilities in determining statics misuses when analyzing empirical data. Our findings reveal widespread methodological issues within the ESE field, including misidentifying statistical methods, models, and tests. This raises serious concerns about experts’ performance in spotting them and indicates risks of spreading false evidence regarding software phenomena. Shortly, we aim to expose these recurring issues extensively and establish a framework for reassessing the very foundation of research in our field. Therefore, this work calls for critically rethinking and reforming data analysis in ESE, paving the way for our future work. Paper Structure Our study is composed of two sub-studies, namely a pilot study investigating the usage of methodological patterns in ESE studies, distinguishing between adequate (“good”) patterns that adhere to best practices and inadequate (“bad”) pattern and a workshop evaluating the experts’ inadequate methodological pattern detection and remediation capabilities. This section briefly presents the structure of our study. Study Overview . Section 2 presents the background, Section 3 presents the related work, and Section 4 describes the overall study design. Pilot Study . Section 5 presents the pilot study. Specifically, Section 5.1.4 presents the pilot study design, Section 7 presents the obtained results, Section 5.3 discusses them, and Section 5.4 highlights the threats to the validity of our pilot study. Workshop . Section 6 presents the workshop. Specifically, Section 6.1.4 presents the workshop study design, Section 6.2.2 presents the results, Section 6.3 discusses them, and Section 6.4 highlights the threats to the validity of our workshop. Conclusions . Section 7 discusses the key insights of our overall study and the main limitations. Finally,

Read the complete text