Automatically Learning a Precise Measurement for Fault Diagnosis Capability of Test Cases

Yifan Zhao, Zeyu Sun, Guoqing Wang, Qingyuan Liang, Yakun Zhang, Yiling Lou, Dan Hao, Lu Zhang · 2025

Read Automatically Learning a Precise Measurement for Fault Diagnosis Capability of Test Cases by Yifan Zhao, Zeyu Sun, Guoqing Wang, Qingyuan Liang, Yakun Zhang, Yiling Lou, Dan Hao, Lu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02216.txt.

Introduction

Yifan Zhao and co-authors propose, in a 2025 paper, a learned score for how well a software test exposes bugs. Fault hunting tools lean on tests, and a test's power to pinpoint faults guides which tests to keep or craft. Past scores rest on hand-set rules and prove rough.

Their score, RLFDC, learns from feedback instead. Fault hunting results act as rewards, and a reinforcement learning model learns to predict each test's fault-finding power. Unlike stronger past scores, it needs no pass or fail results as input, so it can steer test creation too. Tests in the study check how well the learned score lifts fault hunting over rule-based rivals. A learned gauge may give debugging teams a sharper way to spend test effort.

Catalog record

CREATORYifan Zhao, Zeyu Sun, Guoqing Wang, Qingyuan Liang, Yakun Zhang, Yiling Lou, Dan Hao, Lu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02216

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatically Learning a Precise Measurement for Fault Diagnosis Capability of Test Cases Yifan Zhao, Zeyu Sun, Guoqing Wang, Qingyuan Liang, Yakun Zhang, Yiling Lou, Dan Hao, Lu Zhang Abstract . Prevalent Fault Localization (FL) techniques rely on tests to localize buggy program elements. Tests could be treated as fuel to further boost FL by providing more debugging information. Therefore, it is highly valuable to measure the Fault Diagnosis Capability (FDC) of a test for diagnosing faults, so as to select or generate tests to better help FL (i.e., FL-oriented test selection or FL-oriented test generation). To this end, researchers have proposed many FDC metrics, which serve as the selection criterion in FL-oriented test selection or the fitness function in FL-oriented test generation. Existing FDC metrics can be classified into result-agnostic and result-aware metrics depending on whether they take test results (i.e., passing or failing) as input. Although result-aware metrics perform better in test selection, they have restricted applications due to the input of test results, e.g., they cannot be applied to guide test generation. Moreover, all the existing FDC metrics are designed based on some predefined heuristics and have achieved limited FL performance due to their inaccuracy. To address these issues, in this paper, we reconsider result-agnostic metrics (i.e., metrics that do not take test results as input), and propose a novel result-agnostic metric RLFDC which predicts FDC values of tests through reinforcement learning. In particular, we treat FL results as reward signals, and train an FDC prediction model with the direct FL feedback to automatically learn a more accurate measurement rather than design one based on predefined heuristics. Finally, we evaluate the proposed RLFDC on Defects4J by applying the studied metrics to test selection and generation. According to the experimental results, the proposed RLFDC outperforms all the result-agnostic metrics in both test selection and generation, e.g., when applied to selecting human-written tests, RLFDC achieves 28.2% and 21.6% higher acc@1 and mAP values compared to the state-of-the-art result-agnostic metric TfD. Besides, RLFDC even achieves competitive performance compared to the state-of-the-art result-aware metric FDG in test selection. DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Software testing and debugging 1. Introduction Software debugging is a critical but painstaking process that costs tremendous resources every year ( Vessey, 1985 ; Wong et al., 2016 ) . To alleviate this problem, Fault Localization (FL) techniques aim to automatically localize the buggy elements ( Wong et al., 2016 ) and thus reduce the cost. Prevalent FL techniques ( Naish et al., 2011 ; Abreu et al., 2009 ; Jones and Harrold, 2005 ; Wong et al., 2013 ; Lou et al., 2021 ; Li et al., 2021 ; Li et al., 2019 ; Zhang et al., 2019 ; Papadakis and Le Traon, 2015 ; Hong et al., 2015 ; Moon et al., 2014 ; Papadakis and Le Traon, 2012 ) rely on the coverage information and execution results of tests as input to predict a suspicious value for each code element, indicating their possibility of harboring faults. Consequently, the effectiveness of FL techniques heavily relies on the quality of the tests, as testing information provides critical clues for fault diagnosis. In other words, the fault diagnosis capability (FDC) of a test suite might potentially become the bottleneck of the FL effectiveness ( Perez et al., 2017 ; Campos et al., 2013 ) . Unfortunately, in practice, tests with high FDC are not always available, especially considering the heavy expense for manually writing tests ( Dinella et al., 2022 ) . To mitigate this issue, researchers propose FDC metrics to measure the diagnostic capability of a test or a test suite, and apply these FDC metrics to test generation ( Perez et al., 2017 ; Baudry et al., 2006 ; Campos et al., 2013 ) or selection ( Hao et al., 2010 ; An and Yoo, 2022 ) for improving FL. Fig. 1 presents the workflow of these applications, where FDC metrics play a central role. Given a fault-triggering test (which composes an initial test suite), both of these two tasks aim to augment the initial test suite by generating or selecting more additional tests to more accurately pinpoint the fault. In particular, test generation for FL aims to automatically generate more high-quality tests. It utilizes FDC metric as a fitness function to guide the evolutionary algorithm of test generation tools and generate more tests with high FDC. These tests are then used to construct an augmented test suite to boost the performance of FL techniques. Test selection for FL aims to select valuable tests with high FDC from a given test pool. The selected tests are then used to construct an augmented test suite to boost FL. Selection is performed here because the existing test generation tools cannot generate accurate test oracles and thus the oracles are usually manually labelled ( Campos et al., 2013 ) . To alleviate human efforts on oracle labeling, FDC metrics are applied to select a subset of valuable tests from the whole set of tests generated by test generation tools. The selected tests are then labeled with oracles and provides additional debugging information for FL techniques. In summary, FDC metric serves as a fitness function in test generation and a selection criterion in test selection. Generally, test-based FL techniques, especially spectrum-based FL ( Abreu et al., 2006 ; Abreu et al., 2009 ) , predict suspicious values of code elements based on the diagnosis capability of tests, and FDC metric is proposed to quantify this capability. Figure 1. The overall workflow of FDC metric Existing FDC metrics can be classified into result-agnostic ( Perez et al., 2017 ; Baudry et al., 2006 ; Campos et al., 2013 ; Gonzalez-Sanchez et al., 2011 ) and result-aware metrics ( An and Yoo, 2022 ; Bandyopadhyay and Ghosh, 2011 ; Hao et al., 2010 ; Yoo et al., 2013 ) . Result-agnostic metrics calculate the FDC value based on test coverage, while result-aware metrics calculate the FDC value based on test coverage and previous test execution results. The extra input of execution results makes result-aware metrics perform better but limits their application scenarios. For example, they cannot be applied to test generation for FL, because during the evolution process in test generation, accurate oracles cannot be automatically generated ( Perez et al., 2017 ; Campos et al., 2013 ) . Moreover, all existing metrics are based on hand-crafted formulas and therefore use fixed heuristics on all subjects. However, the prior knowledge may be inaccurate and lacks general applicability for different subjects, making the proposed metrics inherently ineffective. For example, our experimental results on Defects4J show that the result-agnostic metric TfD ( Baudry et al., 2006 ) performs well on Chart and Lang , but performs badly on Time and Math . To solve this problem, we propose a Reinforcement Learning (RL) based approach to build a more accurate result-agnostic FDC metric, RLFDC. Note that, differing slightly from previous work ( An and Yoo, 2022 ) 1 1 1 Prior work ( An and Yoo, 2022 ) classify metrics as “result-aware” or “result-agnostic” based on whether they require test execution results. , we classify metrics as “result-aware” or “result-agnostic” based on whether they

Read the complete text