Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers
Thomas Klein, Sascha Meyen, Wieland Brendel, Felix A. Wichmann, Kristof Meding · 2025
Read Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers by Thomas Klein, Sascha Meyen, Wieland Brendel, Felix A. Wichmann, Kristof Meding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06645.txt.
Introduction
Thomas Klein and co-authors put error bars on a popular way of comparing how classifiers behave, in a 2025 paper. Error consistency checks whether two systems, human or machine, fail on the same items, giving a finer match score than accuracy alone. It ranks vision models against humans in a well-known benchmark. Yet scores were reported without any measure of their noise.
The team adds confidence intervals through resampling, turning the score into something tests can judge. They also model the score as the chance that one system copies the other, which tells researchers how many trials a solid test needs. Revisiting famous results, they find the broad human-machine gap holds, but many gaps between vision models are too small to trust.
Catalog record
| CREATOR | Thomas Klein, Sascha Meyen, Wieland Brendel, Felix A. Wichmann, Kristof Meding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers Thomas Klein, Sascha Meyen, Wieland Brendel, Felix A. Wichmann, Kristof Meding Abstract Benchmarking models is a key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking metrics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as e.g. accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behavioral consistency with humans. Previously, EC values have been reported without confidence intervals. However, empirically measured EC values are typically noisy—thus, without confidence intervals, valid benchmarking conclusions are problematic. Here we improve on standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing us to derive significance tests for EC. Second, we propose a new computational model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us to give practical guidance to scientists regarding the number of trials required for sufficiently powerful, conclusive experiments. Finally, we use our methodology to revisit popular NeuroAI-results. We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology enables researchers to design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment. 1 Introduction Consider the following problem: Given two classifiers operating on the same domain, how should we quantify their similarity? This abstract problem has become highly practically relevant in the context of cognitive science and beyond. In cognitive science, deep neural networks (DNNs) have been proposed as models of the human visual system ( Doerig et al. 2023 ; Kriegeskorte 2015 ; Cichy and Kaiser 2019 ; Kietzmann et al. 2017 ; Yamins et al. 2014 ) , and thus the question arises how similarly they behave to human observers. In other domains, e.g. law or medicine, it is also crucial to know whether DNNs interpret and judge information similarly to humans. Thus, a metric is needed that can reliably quantify the degree of behavioral similarity between DNNs and human decision makers. The standard method proposed for this purpose is error consistency (EC) ( Geirhos et al. 2020 ) , which has seen wide application in the context of human-machine comparisons, e.g. in Geirhos et al. 2021 ; Ollikka et al. 2024 ; Li et al. 2025 ; Parthasarathy et al. 2023 . Similarity metrics based on EC have also seen application in the behavioral similarity section of the vision-branch of Brain-Score ( Schrimpf et al. 2018 ) . More recently, a variation of EC—sharing many of the relevant properties, so that our considerations apply—has been proposed as a metric to compare Large Language Models (LLMs) ( Goel et al. 2025 ) . EC builds on Cohen’s \kappa ( Cohen 1960 ) , which considers the trial-level agreement observed for a pair of classifiers to compute a scalar similarity score, which is 1 if the two models are behaviorally indistinguishable and -1 if they are as different as they could possibly be. A value of 0 indicates independence. In practice, EC is reported as a point estimate of a unit-less scalar value, based on classification decisions obtained on a single set of test stimuli, like in Fig. 2 a. However, since EC relies on trial-by-trial similarity, the measure is inherently noisy, as individual trials can have a large impact on the final score, especially when the total number of trials is small—as is often the case when conducting experiments with humans. Figure 1: Schematic: Error Consistency is noisy. The gist of our findings is that by calculating confidence intervals for empirical EC values, we show that while the human-machine gap is probably real, the measure is not stable enough to resolve differences between models. Our contributions are as follows: • We explain how confidence intervals for empirical EC values can be estimated on any dataset on which EC may be calculated using bootstrapping ( Efron and Tibshirani 1994 ) . • We propose a novel method of modeling classifier consistency, allowing us to simulate classifiers with a specific consistency and thus to plan sufficiently powerful, conclusive experiments. • We apply our methods to two influential benchmarks (Model-VS-Human ( Geirhos et al. 2021 ) and Brain-Score ( Schrimpf et al. 2018 ) ), showing that the currently available amount of human reference data is not sufficient to draw conclusions about model differences in many cases, as illustrated in Fig. 1 . • We provide a python package containing utility functions for obtaining confidence intervals around error consistency values, calculating p-values, planning experiments, and testing results for significance. We will begin by providing a brief introduction to EC in Section 2 , to then explain how confidence intervals can be calculated in Section 3 . A detailed Related Work section is provided in Appendix A since we discuss related work on measuring behavioral alignment, Error Consistency, and benchmarking. 2 Error Consistency First, we provide a brief introduction to error consistency and introduce our notation; see Geirhos et al. 2020 for an in-depth description. EC is measured by applying Cohen’s \kappa to evaluate whether two classifiers’ responses are jointly correct or incorrect—that is, whether they are consistent about when they make errors. The experimental setting in which EC can be calculated is the following: Two classifiers categorize each of N samples (e.g., natural images) into one of K classes (e.g., the 1,000 ImageNet classes). Each sample, x_{i} , has a ground-truth label y_{i}\in\{1,...,K\} and and receives classifications, \hat{y}_{i}^{(j)}\in\{1,...,K\} , with j\in\{1,2\} . Whether these classifications are correct or not is coded as r_{i}^{(j)}\in\{0,1\} where r_{i}^{(j)}=1 if and only if the j -th classifier responded with the true label, \hat{y}_{i}^{(j)}=y_{i} . This renders EC agnostic to the number of classes of the classification task because it only considers agreement on which samples are jointly classified correctly and incorrectly. In this setting, EC is measuring the agreement between r^{(1)} and r^{(2)} using Cohen’s \kappa (which is typically applied to \hat{y}^{(1)} and \hat{y}^{(2)} ): \kappa=\frac{p_{obs}-p_{exp}}{1-p_{exp}} (1) where p_{obs} is the observed agreement, that is, the proportion of trials which are jointly classified correctly or incorrectly. This quantity is normalized by the amount of agreement expected from independent classifiers, p_{exp} . This normalization accounts for the fact that classifiers with high accuracies are a priori expected to agree more often in their classifications than classifiers with low accuracies. However, the normalization does not render EC fully orthogonal to the accuracies of the classifiers, because Cohen’s \kappa depends on the marginal distributions of classifier responses ( Falotico and Quatto 2014 ; Grant et al. 2016 ) . This is easy to see in the limiting case where two classifiers have perfect accuracies ( p_{1}=p_{2}=1 ): Division by zero renders EC undefined because, intuitively, error consistency can not be evaluated in the absence of errors. Moreover, if exactly one classifier has perfect