Empirical Analysis of Temporal and Spatial Fault Characteristics in Multi-Fault Bug Repositories

Dylan Callaghan, Alexandra van der Spuy, Bernd Fischer · 2025

Read Empirical Analysis of Temporal and Spatial Fault Characteristics in Multi-Fault Bug Repositories by Dylan Callaghan, Alexandra van der Spuy, Bernd Fischer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08872.txt.

Introduction

Bug databases built for research usually pretend each broken version holds exactly one fault, an assumption real software laughs at. Dylan Callaghan, Alexandra van der Spuy, and Bernd Fischer audited sixteen open-source Java and Python projects in 2025, from the standard Defects4J and BugsInPy collections, to see how faults really live.

Many faults turn out to be long-lived, so most versions in fact carry several coexisting bugs. The faults huddle in a small subset of the code, yet spread fairly evenly inside that subset, leaving few true hotspots. The findings matter because testing and repair tools tuned on single-fault assumptions can misjudge their own scores. The paper is a reality check for a whole subfield's benchmarks.

Catalog record

CREATORDylan Callaghan, Alexandra van der Spuy, Bernd Fischer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Empirical Analysis of Temporal and Spatial Fault Characteristics in Multi-Fault Bug Repositories Dylan Callaghan, Alexandra van der Spuy, Bernd Fischer Abstract . Fixing software faults contributes significantly to the cost of software maintenance and evolution. Techniques for reducing these costs require datasets of software faults, as well as an understanding of the faults, for optimal testing and evaluation. In this paper, we present an empirical analysis of the temporal and spatial characteristics of faults existing in 16 open-source Java and Python projects, which form part of the Defects4J and BugsInPy datasets, respectively. Our findings show that many faults in these software systems are long-lived, leading to the majority of software versions having multiple coexisting faults. This is in contrast to the assumptions of the original datasets, where the majority of versions only identify a single fault. In addition, we show that although the faults are found in only a small subset of the systems, these faults are often evenly distributed amongst this subset, leading to relatively few bug hotspots. 1. Introduction Throughout the lifetime of a software system, a large number of faults manifest themselves, where a fault (also called a bug or defect) is defined as a deviation between the requirements and the actual system behavior. The removal of faults forms a large part of the software maintenance process ( 1 ) . A variety of factors affect the time of discovery of a fault in the system ( 2 ) and the cost of software debugging (i.e., removing faults), and many faults remain in the system for long periods of time ( 3 ) . In order to validate and understand this phenomenon better, we conduct an empirical study using two datasets of software faults mined from real-world open-source software systems, namely Defects4J ( 4 ) and BugsInPy ( 5 ) . These datasets are commonly used in the validation of software maintenance and debugging techniques. We build on work by Callaghan and Fischer ( 6 ) and analyze these repositories based on the identification of multiple faults across the lifetime of the software, enabling an evaluation of the faults throughout the evolution of the software system. Our results confirm the findings in current literature ( 3 , 7 , 8 , 9 ) that a large portion of faults are fixed within a short time after they are introduced, typically within a day to a week. However, the fault lifetimes exhibit a long-tail characteristic, and many faults remain in the system for months or even years after their introduction. The fault lifetimes roughly follow the expected negative exponential distribution; however, we show that their logarithms have a zero-inflated, left-skewed distribution shape, which indicates that a more complex mathematical model may yield a better fit and thus allow better predictions. We show that, correlated with (and presumably caused by) the large fault lifetimes, the majority of system versions contain multiple simultaneous faults, with between 3 and 10 faults identified on average in each version. This is in contrast to the assumptions of the original datasets, where the majority of versions only identify a single fault, but consistent with the results shown by An et al. ( 10 ) for the Java-based Defects4J benchmark suite. Our analysis further shows that the number of faults that occur in a module over the system’s lifetime is not correlated with the module’s size; instead, our findings indicate that the overall system faultiness can be attributed to the evolution of the system and, specifically, to different stages in the development workflow. Finally, we show that only a small percentage (16–17%) of the modules in each project contain identified faults at all, tightening earlier observations about the Pareto-principled nature of fault distributions (i.e., that 80% of the faults occur in only 20% of the modules) ( 11 , 12 , 13 , 14 ) . We also show that the number of faults per module varies substantially per project: a large number of projects have only a single fault per module, while other projects have one or more modules with much higher fault occurrences. We then give an intuitive definition of bug hotspots and show that they occur only rarely, with only 5 bug hotspots in 4 of the 16 projects. To facilitate open science, we make our results publicly available in a full replication package ( 15 ) . 2. Background In this study, we consider the open-source datasets Defects4J ( 4 ) and BugsInPy ( 5 ) , which are real-world repositories of Java and Python software projects with faults, respectively. We choose these datasets firstly due to their popularity in the field of software debugging, as any findings would directly affect this field, providing useful insights. We choose them secondly due to their maturity and support, which enables a detailed analysis. The datasets are comprised of bug entries, which refer to two versions in the underlying project repository, a faulty version with a single fault and fixed version, as well as the test suite exposing the fault. The entries for each project are numbered from 1 to N , where a lower number corresponds to a more recent version pair. Table 1 gives the overall dataset statistics for the projects in Defects4j and BugsInPy. Note that we do not consider in our analysis projects with 10 or fewer versions from BugsInPy ( 5 ) , due to their limiting size. In order to aid in the analysis of this study, we build upon the work of An et al. ( 10 ) and Callaghan and Fischer ( 6 ) . An et al. identify which Defects4J faults may be present in Defects4J versions other than the initially identified version for that fault. They check for exposure of each fault (i.e., failure of at least one test case related to the fault) in each version, and deem a fault to be present in a version if it is exposed . Callaghan and Fischer build on the work of An et al. by including the source code locations of these faults in each version they are present, as well as extending the technique to the BugsInPy dataset. Table 1 . Dataset statistics. N is the number of versions in the project; the number of faults and program sizes are averaged over all project versions. Modules Methods KLOC Project N Min Mean Max Min Mean Max Min Mean Max Chart 1 1 1 https://www.jfree.org/jfreechart 26 553 573.2 633 7129 7582.6 8661 203.3 208.7 232.4 Closure 2 2 2 https://developers.google.com/closure/compiler/ 106 298 383.3 413 3704 5523.4 6229 99.4 208.4 269.2 Lang 3 3 3 https://commons.apache.org/proper/commons-lang/ 65 76 89.0 108 1857 2036.5 2217 48.0 53.2 61.1 Math 4 4 4 https://commons.apache.org/proper/commons-math/ 106 136 508.5 808 1138 4321.2 6603 30.5 121.7 185.3 Time 5 5 5 https://www.joda.org/joda-time/ 27 156 156.3 157 3573 3657.6 3744 70.2 78.0 99.2 Total Java 330 76 362.8 808 1138 4431.9 8661 30.5 134.0 269.2 ansible 6 6 6 https://github.com/ansible/ansible 18 439 3702.1 5190 3287 28613.7 39898 101.7 1124.7 1590.1 black 7 7 7 https://github.com/psf/black 23 14 20.4 24 211 288.2 404 5.2 66.5 96.0 fastapi 8 8 8 https://github.com/tiangolo/fastapi 16 21 27.6 40 80 110.8 130 2.8 4.2 5.0 keras 9 9

Read the complete text