Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems

Giuseppe De Rosa 1 2 , Pietro Liguori 1 · 2026

Read Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems by Giuseppe De Rosa 1 2 , Pietro Liguori 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26667.txt.

Introduction

De Rosa and Liguori ask whether post-release faults in Python systems can be predicted from software metrics. Using a balanced set of more than 4,000 labeled faults with 83 metrics spanning product, process, statistical, and Python-specific traits, plus normalized code representations, they run cross-project experiments. Large language models and unsupervised models fail to separate residual faults from others, while supervised metric-based models reach recall between 85 and 90 percent and cut missed faults roughly tenfold. Process metrics - age, churn, and developer activity - plus class and file size prove most predictive.

Catalog record

CREATORGiuseppe De Rosa 1 2 , Pietro Liguori 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70258
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems Giuseppe De Rosa 1 2 , Pietro Liguori 1 Abstract Python’s dynamic nature complicates testing and increases the possibility that some defects evade detection, so an effective fault prediction becomes essential. We examine whether post-release faults can be predicted using modern ML and DL. Using a balanced dataset of over 4,000 labeled faults with 83 product, process, statistical, and Python-specific metrics plus normalized code representations, we conduct cross-project experiments. LLMs and unsupervised models fail to distinguish residual from non-residual faults, while supervised metric-based models (RandomForest, XGBoost, CatBoost) perform far better, yielding a 0.85-0.9 recall and cutting false negatives by an order of magnitude. Process metrics, especially age, churn, and developer-activity, alongside class and file size, consistently prove most predictive. Notably, the Principal Component Analysis shows that metrics and code embeddings occupy distinct regions of the representation space, suggesting that they capture complementary rather than redundant information. I Introduction Software failures continue to impose severe operational and economic consequences across modern digital infrastructures. From the $460 million trading loss at Knight Capital [ 19 ] to large-scale transportation disruptions and safety-critical incidents [ 48 , 66 ] , the inability to prevent defects from reaching deployment environments remains a central challenge for software dependability. Industry analyses echo this urgency: according to Gartner, organizations that invest in “digital immunity”, i.e., the systematic detection, containment, and mitigation of defects, can reduce downtime by up to 80%, significantly improving resilience and service continuity [ 29 ] . Accurately identifying fault-prone components before release, therefore, plays a decisive role in reducing maintenance effort, improving testing strategies, and increasing trust in the ever-growing ecosystem of software-driven systems. Despite substantial advances in defect prediction, one category of faults continues to elude even rigorous testing pipelines: residual faults , i.e., defects that remain unnoticed until the software is already in production. In the reliability literature, these are also referred to as post-release faults , since the defining characteristic of a residual defect is precisely that it survives all pre-release verification activities and manifests only after deployment [ 46 , 63 ] . Using post-release failures as an operational definition is, therefore, standard practice: a defect revealed in the field is, by construction, one that the testing and quality-assurance processes failed to expose. These faults are uniquely problematic. They often surface only under realistic operational conditions that are difficult or impossible to reproduce in controlled test suites; they tend to arise from subtle interactions, rare execution paths, or latent corner cases; and, critically, they rarely exhibit clear syntactic or structural patterns that distinguish them from non-residual defects. This makes predicting which faults will escape pre-release detection intrinsically more challenging than predicting fault-proneness in general. Recent research in machine learning (ML) and deep learning (DL) for defect prediction has shown promising results by leveraging combinations of product and process metrics, along with learned code representations [ 21 ] . Yet several long-standing issues remain unresolved, and they become even more critical when the target is residual faults rather than generic defect-proneness. First, cross-project generalization is notoriously weak: models trained on one project frequently underperform when applied to others [ 14 ] . This limitation is exacerbated by the rarity of residual faults and their project-specific characteristics. Second, DL approaches often lack interpretability, limiting their usefulness in safety-critical settings where understanding why a component is predicted as risky is as important as the prediction itself [ 21 , 15 ] . Third, residual faults are inherently rare, creating a challenging class imbalance that distorts learning and inflates apparent performance [ 24 , 11 ] . These limitations motivate the need for carefully designed datasets and a broad spectrum of software metrics to capture language-specific behaviors that may influence residual fault manifestation. This paper investigates residual fault prediction. Predicting residual faults is not equivalent to standard defect prediction: traditional defect predictors estimate whether a defect exists in a component, implicitly conflating two statistically different populations: faults that will be caught during testing and faults that will escape into production. We formulate a distinct prediction target: estimating the probability that a defect, once introduced, survives all pre-release validation activities and reaches production. This distinction is operationally significant because it enables targeted interventions such as selective manual review or fuzzing only for components with high escape risk rather than the entire codebase. To support this task, we leverage a diverse range of traditional software metrics, including product metrics that quantify size, control-flow complexity, documentation quality, coupling, and inheritance structures; Python-specific syntactic indicators; statistical dispersion measures; and process metrics that capture code churn, temporal evolution, commit activity, and developer behavior. We focus on Python because of its widespread adoption in modern software infrastructures and its dynamic, interpreted nature, which makes defects more likely to manifest only at runtime [ 68 ] . These characteristics increase the probability that subtle faults escape pre-release testing, making Python a particularly relevant and challenging setting for studying residual defects [ 61 , 73 ] . Our study yields three key contributions: 1. A new dataset of residual and non-residual Python faults. We build a balanced and reproducible dataset of more than 4,000 classified faults from real-world Python projects, complemented by 500 additional cases from BugsInPy. Each instance includes code, software metrics, and version-control information, enabling systematic experimentation on residual-fault prediction and supporting a repeatable evaluation. 2. A systematic assessment of LLM-based code representations for residual fault prediction. We evaluate both proprietary and fine-tuned open LLMs under uniform conditions. Although their embeddings contain meaningful signals, the models themselves perform unreliably: closed-source systems achieve F1-scores between 0.43 and 0.72, often collapsing into degenerate behavior, while fine-tuned models reach only 0.35–0.38 and yield a recall of 0.3 on average. A representation-space analysis shows that embeddings remain largely orthogonal to metrics (maximum correlation 0.27; mean 0.24), indicating that LLMs encode complementary but insufficient information for this task. 3. An investigation of software metrics through lightweight ML models. We assess how far one can go using only traditional software metrics and simple numerical models. Unsupervised anomaly detectors perform poorly, confirming that residual faults are not statistical outliers. In contrast, supervised learners such as RandomForest, XGBoost, and CatBoost achieve F1-scores of 0.71–0.73 and reach 0.85-0.9 recall, reducing false negatives by an order of magnitude compared to LLMs. Feature-importance and SHAP (i.e., a method that assigns each feature a quantitative contribution to the model’s prediction) analyses highlight process, size, and developer-activity metrics as the most informative families, revealing stable structural and historical patterns that support effective residual-fault prediction. To foster both academic and industrial research on this topic, we publicly release our replication package [ 60 ] , which includes the datasets and scripts used for the analyses in the paper. The remainder of this paper is organized as follows. Section II introduces the background and motivates our focus on residual faults. Section III details the data collection, metric extraction, and model design. Section IV describes the experimental setup, while Section V presents our evaluation results. Section VI discusses the results. Related work is discussed in Section VII , and threats to validity are outlined in Section VIII . Finally, Section IX concludes

Read the complete text