What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study

Domenico Cotroneo 1 , Giuseppe De Rosa 2 3 , Cristina Improta 2 , Benedetta Gaia Varriale 2 · 2026

Read What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study by Domenico Cotroneo 1 , Giuseppe De Rosa 2 3 , Cristina Improta 2 , Benedetta Gaia Varriale 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26672.txt.

Introduction

Cotroneo, De Rosa, Improta, and Varriale characterize defects that escape testing and surface after release across C, C++, and Java systems, drawing on more than 14,000 defects from open-source projects. Post-release defects cluster in older, frequently changed, high-churn components and demand longer, more complex fixes than pre-release ones. The pattern suggests residual faults stem more from evolutionary and process dynamics than from code structure alone. Reliability work should therefore target testing at mature, complex regions.

Catalog record

CREATORDomenico Cotroneo 1 , Giuseppe De Rosa 2 3 , Cristina Improta 2 , Benedetta Gaia Varriale 2
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70429
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26672

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study Domenico Cotroneo 1 , Giuseppe De Rosa 2 3 , Cristina Improta 2 , Benedetta Gaia Varriale 2 Abstract Understanding how software defects manifest and evolve in production environments is critical for improving reliability. While previous research has largely focused on pre-release defects, the nature of residual faults , i.e., those escaping testing and surfacing post-release, remains poorly understood. This paper presents a large-scale characterization of pre- and post-release defects across C/C++ and Java systems, encompassing over 14k defects mined from open-source projects. We employ a broad suite of software metrics to capture diverse code attributes such as complexity, size, structure, and development history. Results show that post-release defects are concentrated in older, frequently modified, and high-churn components, typically requiring longer and more complex fixes than pre-release ones. These findings highlight that residual defects arise more from evolutionary and process dynamics than code structure alone, suggesting that reliability engineering should prioritize targeted testing in mature and complex code regions. I Introduction Software failures in deployed systems remain a primary threat to dependability, often arising from residual faults, i.e., bugs that evade rigorous testing and manifest only in production [ 31 , 33 ] . These latent defects have caused high-impact economic, operational, and safety incidents [ 46 , 38 ] . The urgency of understanding these escaping defects is increasing. As the industry shifts toward AI-assisted development, analysts predict that “by 2027, 25 percent of software defects escaping to production will result from a lack of human oversight of AI-generated code” [ 16 ] . Yet, a critical gap persists: we still lack a fundamental understanding of what properties allow certain bugs to escape testing, even in traditional, human-authored software. Current research largely treats software faults as a homogeneous category, drawing on datasets dominated by easy-to-detect pre-release bugs [ 56 , 49 ] . This introduces a survivorship bias: we design and optimize testing tools based on the faults they already catch, rather than on those that evade them. Without establishing a solid empirical baseline of how residual faults differ from non-residual ones across product, process, and historical dimensions, efforts to secure future high-velocity development pipelines are proceeding blindly. We address this gap by treating residual faults as a distinct class, isolating the specific characteristics that allow them to evade testing. This distinction is critical for reliability engineering: it shifts the focus from generic bug prevention to targeting the specific patterns, often invisible to standard metrics that manifest only in production. To this aim, we constructed a large-scale, balanced dataset from mature C/C++ and Java projects. Identifying residual faults is challenging due to their rarity and the noise in issue trackers; we filtered over 200k candidates to identify approximately 7k confirmed residual defects. We then reconstructed the faulty and fixed versions of each function, determining the escape timing by triangulating issue metadata, report text, and reporter identity. To quantify differences between residual and non-residual faults, we extract a set of product, process, and statistical metrics of the code involved in each defect. Product metrics capture structural properties of the affected functions, while process metrics reflect their commit history, developer activity, and related aspects. We analyze these metric profiles to uncover systematic differences between pre-release and post-release faults, how these differences vary across languages, and relate to the complexity and effort of fixing each defect. The contributions of this work are threefold. (i) We introduce a methodology for collecting, classifying, and reconstructing residual faults at the function level across heterogeneous systems. (ii) We construct two large-scale datasets enriched with product, process, and statistical metrics. (iii) We carry out a comprehensive evaluation and characterization of pre- and post-release defects. Overall, our analysis reveals three main findings: 1. Residual and non-residual defects are influenced by similar metric families, but the relationships among these metrics differ substantially: residual faults exhibit more heterogeneous and unstable metric profiles than defects detected during testing. 2. Distributional differences show that residual and non-residual defects diverge mainly along process and project-history dimensions, with structural metrics playing a smaller role, particularly in Java. 3. In C/C++, residual defects are not only harder to find but significantly more expensive to fix, requiring more complex code changes and time compared to pre-release bugs. In contrast, Java shows a more uniform fixing effort. To foster academic and industry research, we publicly release the datasets, the results, and code used in this work [ 3 ] . The remainder of this paper is organized as follows: § II introduces background concepts and discusses previous work; § III describes the dataset construction process and the software metrics used; § IV reports the experimental evaluation; § VI analyzes threats to validity; § VII concludes the paper. II Background & Related Work Software dependability research often uses the terms fault , defect , and bug interchangeably. Following Laprie’s terminology [ 27 ] , we refer to a fault as a latent imperfection in the code or design that may lead to failure once activated, while we use defect and bug as synonyms commonly adopted in empirical software engineering. A failure denotes the externally observable incorrect behavior that results from fault activation. A release represents a version of the software delivered to end users. Defects discovered before this point, in the testing phase, are pre-release defects or non-residual , whereas those that escape verification activities and manifest during operational use are post-release , or residual defects [ 27 , 42 ] . Unlike pre-release defects, residual defects tend to require specific runtime conditions to emerge [ 17 , 13 ] and often result in reliability and economic consequences [ 42 , 46 ] . Research on software defects spans several directions, but only a limited portion directly addresses the post-release faults. A first line of work investigates how software components accumulate pre- and post-release defects. Early studies show that defect patterns observed before release do not necessarily translate to failures in the field [ 15 ] . Other works highlight the role of code churn, historical change patterns, and product metrics in correlating with post-release defects across large industrial systems [ 36 , 35 , 34 , 57 ] . These studies collectively indicate that post-release defects are influenced by both structural and historical properties but do not provide a direct characterization of how these faults differ from pre-release ones. A broader body of research focuses on predicting post-release defect-proneness using product and process metrics. Systematic analyses show that process and change-related metrics often outperform traditional complexity-based ones for identifying defect-prone components [ 39 ] . Later studies explored uncommon metric families such as cohesion measures and code-review characteristics [ 54 , 26 ] . These approaches provide valuable predictors but aim at post-release prediction and rely on metric families that are costly to compute at scale. Another research direction examines the conditions activating residual faults. Laprie [ 27 ] emphasizes that certain faults activate only under specific environmental or workload factors not exercised during testing. Empirical analyses of field failures corroborate this view, showing that many faults leading to operational failures are triggered by rare execution scenarios or stress conditions, often require larger changes, and

Read the complete text