Revisiting “Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion”: A Critical Review and Implications on DNN Coverage Testing Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM

Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Shin Yoo, Paolo Tonella · 2026

Read Revisiting “Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion”: A Critical Review and Implications on DNN Coverage Testing Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM by Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Shin Yoo, Paolo Tonella (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08729.txt.

Introduction

Jinhan Kim and four co-authors take a close look at a test rule for deep neural nets. The rule is called Neural Coverage, or NLC for short. Yuan and his team put it out in 2023. They said it met eight design goals and did well in tests. Kim and his team ask if those claims hold up.

The team checks both the math claims and the test claims. They say some of the base claims need a fresh look. Their point is not just about one rule. How we test a net shapes how safe we think it is. A reader who builds or tests such nets gets a clear warn to check coverage rules with care.

Catalog record

CREATORJinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Shin Yoo, Paolo Tonella
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting “Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion”: A Critical Review and Implications on DNN Coverage Testing Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3773148 ISBN: 979-8-4007-2025-3/26/04 CCS: Software and its engineering Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Shin Yoo, Paolo Tonella Abstract . We present a critical review of Neural Coverage (NLC), a state-of-the-art DNN coverage criterion by Yuan et al. at ICSE 2023. While NLC proposes to satisfy eight design requirements and demonstrates strong empirical performance, we question some of their theoretical and empirical assumptions. We observe that NLC deviates from core principles of coverage criteria, such as monotonicity and test suite order independence, and could more fully account for key properties of the covariance matrix. Additionally, we note threats to the validity of the empirical study, related to the ground truth ordering of test suites. Through our empirical validation, we substantiate our claims and propose improvements for future DNN coverage metrics. Finally, we conclude by discussing the implications of these insights. † † cc-license: by 1. Introduction Testing of Deep Neural Networks (DNNs) has become a crucial aspect of their integration into various applications, sparking a surge in the development of coverage criteria. These criteria draw inspiration from traditional software testing methodologies, aiming to ensure the reliability and robustness of DNNs. A significant milestone in this field was the introduction of Neuron Coverage (NC) by DeepXplore in SOSP 2017 ( Pei et al. 2017 ) , marking the first coverage criterion specifically tailored for DNNs. This drew considerable interest within the Software Engineering (SE) community, leading to the creation of various coverage criteria. Such criteria typically analyse the activation space of inner neurons and quantify the extent to which this space has been covered ( Ma et al. 2018 ; Kim et al. 2019 ) . At ICSE 2023, a new DNN coverage criterion called Neural Coverage (NLC) was introduced ( Yuan et al. 2023 ) . NLC distinguishes itself from prior coverage criteria by proposing and satisfying eight design requirements, which a DNN coverage criterion is expected to satisfy. To date, NLC is the only criterion that fulfils all these requirements. As a hyperparameter-free metric, NLC characterises the correlation between neurons’ continuous outputs. Empirical studies conducted by the NLC proponents have demonstrated that NLC outperforms other coverage criteria, underscoring the strength of its novel design ( Yuan et al. 2023 ) . The implementation of NLC, along with that of existing coverage criteria, and all empirical results are publicly available, which has gained significant traction, as evidenced by its 250+ GitHub stars. 1 1 1 https://github.com/Yuanyuan-Yuan/NeuraL-Coverage , accessed on February 11, 2025. Despite the proponents’ claims and the empirical evidence, we question some of the theoretical and empirical assumptions behind the work by Yuan et al. ( Yuan et al. 2023 ) . 2 2 2 Throughout the paper, we refer to Yuan et al. ( Yuan et al. 2023 ) simply as Yuan et al. (without repeated citations). As suggested by Monperrus ( Monperrus 2014 ) , we believe that respectful critique and scholarly debate are essential to the scientific process. While such discussions often occur within technical contributions (e.g., in related work sections), they are frequently superficial and biased toward the new approach. Explicit discussion of limitations and open issues in a fully dedicated paper, on the other hand, can provide a clearer understanding of the area, pointing to better ways to advance the discipline. Our critical review is driven by two primary motivations: (1) concerns regarding the theoretical foundations of NLC, and (2) concerns about its empirical validation. First, we articulate our conceptual concerns by analysing NLC’s design in light of the eight proposed requirements. Specifically, we highlight three major shortcomings: the lack of monotonicity (i.e., adding more tests should not decrease coverage), the lack of order independence (i.e., the coverage value should remain unchanged under permutations of the test suite), and the absence of a bounded coverage measure (i.e., the coverage value should have a known upper bound to serve as a stopping criterion). These are essential properties for any reliable DNN coverage criterion. In detail, we explore why NLC fails to serve as a sound coverage criterion, discuss the trade-offs involved in satisfying the considered requirements, address potential misconceptions about NLC’s relationship with the covariance matrix, and propose improvements for future work. Second, we identify flaws in the empirical study design and offer recommendations for the rigorous evaluation of DNN coverage criteria. One major concern descends from their requirement R1, stating: ‘DNN coverage criterion should precisely describe the continuous output of a neuron.’ . The authors argue that most existing criteria fail to satisfy R1 because they rely on thresholding, which artificially divides the neuron output space and fails to capture its true shape. While this claim is valid, we contend that by working without any thresholding, NLC has turned into a non-monotonic, unbounded coverage criterion, whose value may decrease as new tests are added to a test suite and without any clearly defined maximum value. Correspondingly, it cannot function as a reliable coverage criterion. The authors partially recognised this issue and proposed a fix, which involves updating coverage only when it increases. However, this solution introduces a dependency on the order of test case evaluation, as the same test case may or may not be taken into account by NLC, depending on the sequence of evaluation. We argue that monotonicity and order independence should be intrinsic properties of the coverage criterion’s design. Furthermore, we empirically validate our argument regarding order dependency, demonstrating that it is a practically significant issue. Our experiments reveal that NLC coverage can decrease by up to 10.42% depending on the evaluation order. In addition to our theoretical review of NLC, we critically analyse the empirical study design, which significantly deviates from prior work ( Pei et al. 2017 ; Kim et al. 2019 ; Ji et al. 2023 ; Ma et al. 2018 ) . Their approach establishes a ground truth ordering of test suites, for instance, asserting that test suite T_{1} must have higher coverage than another suite T_{2} based on their posit that T_{1} exhibits greater diversity than T_{2} . This is used to evaluate and compare coverage criteria in the absence of definitive ground truth in the literature. We highlight substantial threats to the validity of their study. This is primarily because the assumed ground truth does not always hold, as coverage cannot be equated to diversity, rendering their evaluation unreliable. At the end of the paper, we revisit the critiques outlined earlier, provide additional insights, and suggest potential solutions to the issues identified in Yuan et al.’s work, concluding with key takeaways. In summary, our contributions are: • An analysis of how NLC misses important properties of coverage criteria. We examine the eight proposed requirements, the preservation of covariance matrix properties, and the proposed layer-wise aggregation. • Identification of critical threats to the validity in their empirical study design, particularly in establishing a ground truth ordering of test suites. • Empirical

Read the complete text