Practical estimation of the optimal classification error with soft labels and calibration

Ryota Ushio, Takashi Ishida, Masashi Sugiyama · 2025

Read Practical estimation of the optimal classification error with soft labels and calibration by Ryota Ushio, Takashi Ishida, Masashi Sugiyama (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20761.txt.

Introduction

Ryota Ushio, Takashi Ishida, and Masashi Sugiyama work out how to estimate the best possible error rate of a classifier from soft labels in a 2025 paper. Knowing the Bayes error tells developers when to stop polishing a model. Earlier work used soft labels for this, but two gaps remained: how fast the simple estimator's bias shrinks, and what to do when soft labels are corrupted.

The team shows the bias can decay much faster than previously proven, at a rate that adapts to class separation. For corrupted labels, they prove a warning: even perfectly calibrated soft labels can mislead badly. A particular calibration method rescues consistency under weaker assumptions than before. Their method needs no raw inputs, suiting privacy-sensitive settings. Experiments on synthetic and real data back the theory.

Catalog record

CREATORRyota Ushio, Takashi Ishida, Masashi Sugiyama
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Practical estimation of the optimal classification error with soft labels and calibration Ryota Ushio, Takashi Ishida, Masashi Sugiyama Abstract While the performance of machine learning systems has experienced significant improvement in recent years, relatively little attention has been paid to the fundamental question: to what extent can we improve our models? This paper provides a means of answering this question in the setting of binary classification, which is practical and theoretically supported. We extend a previous work that utilizes soft labels for estimating the Bayes error, the optimal error rate, in two important ways. First, we theoretically investigate the properties of the bias of the hard-label-based estimator discussed in the original work. We reveal that the decay rate of the bias is adaptive to how well the two class-conditional distributions are separated, and it can decay significantly faster than the previous result suggested as the number of hard labels per instance grows. Second, we tackle a more challenging problem setting: estimation with corrupted soft labels. One might be tempted to use calibrated soft labels instead of clean ones. However, we reveal that calibration guarantee is not enough , that is, even perfectly calibrated soft labels can result in a substantially inaccurate estimate. Then, we show that isotonic calibration can provide a statistically consistent estimator under an assumption weaker than that of the previous work. Our method is instance-free , i.e., we do not assume access to any input instances. This feature allows it to be adopted in practical scenarios where the instances are not available due to privacy issues. Experiments with synthetic and real-world datasets show the validity of our methods and theory. The code is available at https://github.com/RyotaUshio/bayes-error-estimation . 1 Introduction It is a common practice in the field of machine learning research to assess the performance of a newly proposed algorithm using one or more metrics and compare them to the previous state-of-the-art (SOTA) performance to show its effectiveness ( , 2024 , , 2025 , , 2025 ) . In classification, arguably the most common one is the error rate, i.e., the expected frequency of misclassification for future data. While the SOTA performance continues to improve for a wide range of benchmarks over time, there is a limit on the prediction performance that any machine learning model can achieve, which is determined by the underlying data distribution. It is important to know this limit, or the best achievable performance. For example, if the current SOTA performance is close enough to the limit, there is no point in seeking further improvement. It is not only wasteful in terms of time and financial resources but also harmful to the environment, since large-scale machine learning models are notorious for their high energy consumption ( Strubell et al., 2020 , Luccioni et al., 2023 ) . Knowing the best possible performance also provides a practical check for test-set overfitting ( Recht et al., 2018 , Ishida et al., 2023 ) : if a model’s score on the test set approaches or even exceeds the upper bound, it may be a signal of the model directly training on the test set. In classification, the best achievable error rate for a given data distribution is called the Bayes error , and the estimation of the Bayes error has a rich history of research ( Fukunaga and Hostetler, 1975 , Devijver, 1985 , Berisha et al., 2014 , Moon et al., 2018 , Noshad et al., 2019 , Theisen et al., 2021 , Ishida et al., 2023 , Jeong et al., 2023 ) . In the case of binary classification, the existing approaches can be roughly categorized into two groups: the majority of estimation from instance-label pairs (x_{1},y_{1}),\dots,(x_{n},y_{n})\in\mathcal{X}\times\left\{{0,1}\right\} ( Fukunaga and Hostetler, 1975 , Devijver, 1985 , Berisha et al., 2014 , Moon et al., 2018 , Noshad et al., 2019 , Theisen et al., 2021 ) , where \mathcal{X} is the space of instances, and the recently proposed methods of estimation from soft labels \eta_{1},\dots,\eta_{n}\in[0,1] ( Ishida et al., 2023 , Jeong et al., 2023 ) . A soft label is a special type of supervision that represents the posterior class probability \eta_{i}\coloneqq\mathbb{P}(y=1\mid x=x_{i}),\ i\in\left\{{1,\dots,n}\right\} , and it quantifies the uncertainty of class labels associated with each instance x_{i} . The strength of the methods proposed in ( Ishida et al., 2023 , Jeong et al., 2023 ) based on soft labels is that they are instance-free , i.e., they do not require access to the instances \left\{{x_{i}}\right\}_{i=1}^{n} . Since the instances themselves are not used for estimation, these methods do not suffer from the curse of dimensionality even when dealing with very high-dimensional data. Moreover, the instance-free property is practically valuable since it makes the methods easy to apply to real-world problems, such as medical diagnoses, where the instances are often inaccessible due to privacy concerns. However, these methods have a crucial limitation: they assume that we have direct access to clean soft labels \eta_{i}=\mathbb{P}(y=1\mid x=x_{i}) , which only an oracle would know. Ishida et al. (2023) also discussed a scenario where each soft label \eta_{i} is approximated by an average \widehat{\eta}_{i}=\frac{1}{m}\sum_{j=1}^{m}y_{i}^{(j)} of m hard labels y_{i}^{(1)},\dots,y_{i}^{(m)} per instance. They showed that, for a fixed number n of samples, the bias of the resulting estimator approaches zero as m tends to infinity. However, their bound on the bias is prohibitively large for practical values of m , and thus the theoretical guarantee for this estimator is weak. Figure 1: The Bayes error estimated with the method of Ishida et al. (2023) is larger than the test error of a Vision Transformer ( Dosovitskiy et al., 2021 ) . Another issue is labeling distribution shift . For example, whereas the images in the original CIFAR-10 dataset ( Krizhevsky, 2009 ) can be regarded as though they were annotated before they were downscaled, the images in the CIFAR-10H dataset ( Peterson et al., 2019 ) were annotated after downscaling, making the task more challenging and thus increasing label uncertainty. 1 1 1 For more information, CIFAR-10 was curated as follows. First, images for each class were collected by searching for the class label or its hyponym on the Internet. Second, the images were downscaled to 32\times 32 . Finally, human labelers were asked to filter out mislabeled images. On the other hand, CIFAR-10H is a soft-labeled version of CIFAR-10, i.e., it consists of 10,000 test images of the CIFAR-10 test set along with their soft labels. Each soft label was obtained as an average of 47–63 hard labels, which were collected by asking human labelers to answer which class the downscaled image belongs to. As a result, the labeling processes are significantly different between these two datasets. Given that the Bayes error can be interpreted as the average label uncertainty, we will get an unreasonably high estimate of the Bayes error if we just plug the soft labels in CIFAR-10H into their estimator, as shown in Figure 1 . In general, a similar issue can arise due to subjectivity of human soft labelers, or the bias of using large language models (LLMs) as annotators ( Gilardi et al., 2023 , Tjuatja et al., 2024 ) . Recent work has explored a range of techniques

Read the complete text