Effects of label noise on the classification of outlier observations

Matheus Vinícius Barreto de Farias, Mário de Castro · 2025

Read Effects of label noise on the classification of outlier observations by Matheus Vinícius Barreto de Farias, Mário de Castro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08808.txt.

Introduction

Matheus Vinícius Barreto de Farias and Mário de Castro probe how label noise in training data affects a classifier's treatment of observations from classes it has never seen. The tool under study is BCOPS, a conformal prediction method that returns sets of candidate classes with a guaranteed coverage probability.

An observation counts as an outlier when its true class is absent from the training set. The natural hope is that the method abstains or flags it. Experiments on synthetic and real datasets measure the abstention rate for such outliers and the model's overall robustness, a scenario the method's designers had not tested. The headline finding is sobering. Even small amounts of label noise change performance a lot. That matters for anyone deploying conformal prediction on poorly labeled data.

Catalog record

CREATORMatheus Vinícius Barreto de Farias, Mário de Castro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Effects of label noise on the classification of outlier observations Matheus Vinícius Barreto de Farias, Mário de Castro Abstract This study investigates the impact of adding noise to the training set classes in classification tasks using the BCOPS algorithm ( Balanced and Conformal Optimized Prediction Sets ), proposed by 5 . The BCOPS algorithm is an application of conformal prediction combined with a machine learning method to construct prediction sets such that the probability of the true class being included in the prediction set for a test observation meets a specified coverage guarantee. An observation is considered an outlier if its true class is not present in the training set. The study employs both synthetic and real datasets and conducts experiments to evaluate the prediction abstention rate for outlier observations and the model’s robustness in this previously untested scenario. The results indicate that the addition of noise, even in small amounts, can have a significant effect on model performance. Keywords: Noise addition, Outlier detection, BCOPS, Machine learning. 1 Introduction The following study presents results obtained from experiments in which, before training a classification model, we added noise to the labels of the training set, so that the information contained in this set is not entirely correct. In fact, most datasets encountered in practical situations contain some degree of noise, which highlights the importance of this type of study for new techniques before implementing them in real-world applications. In this case, we are interested in measuring the impact of noise addition on BCOPS ( 5 ) , a algorithm based on conformal prediction ( 11 ) which, when combined with other machine learning methods, allows the construction of prediction sets for the test set observations in classification tasks. Prediction sets are sets that contain the possible values (for regression tasks) or possible classes (for classification tasks) for new observations. These sets are constructed so that the probability of the true value or class being contained within them meets a coverage guarantee. In the work developed by 5 , the possibility of using these prediction sets to detect outlier observations – meaning, observations whose true class was not present during training – is emphasized. Thus, we aim to measure both the classification coverage and the abstention rate on outlier observations of the BCOPS algorithm under the addition of noise, considering some of the datasets and machine learning algorithms used by 5 . This paper is organized as follows: In Section 2 , we provide a brief review of the fundamental concepts necessary for understanding the algorithm; in Section 3 , we present the method used for adding noise; in Section 4 , we show the results obtained from the conducted experiments and raise related discussions; finally, Section 5 summarizes and discusses the work carried out throughout this study. The implementations of the experiments in the R language ( 10 ) can be found at https://github.com/mvbdf/bcops-exemplos . 2 Conformal Prediction and the BCOPS Algorithm The idea behind conformal prediction, first introduced by 4 and 11 , arises from the need to add a layer of reliability to the predictions made by standard predictive methods, providing a certain guarantee that the true value (for regression problems) or true class (for classification problems) of an observation will be included in a set of possible outcomes. Imagine we have a population divided into different classes, and we wish to predict the class of a new observation—let x denote the feature vector of this observation—based on the characteristics of other samples with known classes. A common approach would be to classify this new observation as belonging to the most probable class using statistical learning methods, such as clustering based on Euclidean distance ( 6 , for details on conventional machine learning methods, see) . In the conformal prediction paradigm, however, our goal is to construct a set, denoted C(x) , containing the possible classes to which x may belong, such that the probability of the true class y being included in C(x) is greater than or equal to a predefined coverage guarantee 1-\alpha , typically 95\% . The construction of these sets is carried out through a conformity score function, which measures how similar a new observation is to the observations of each class present in the training set. Thus, given a new observation x , for each class in the training set ( \left\{1,2,\dots,K\right\} ), we obtain a conformity score that determines whether that class should be included in C(x) —the higher the conformity score for a given class, the closer x is to the observations of that class. Among the measures used to quantify conformity, one may use the Euclidean distance itself or, in the case of BCOPS, a machine learning algorithm capable of separating the classes. Details on the different ways to construct the conformity score function can be found in 7 and 5 . Note that if the conformity score of an observation x is low for all classes present in the training set, no class will be included in C(x) , and x may be considered an outlier—thus, the algorithm is said to abstain from making a prediction. The abstention rate on outlier observations is one of the performance metrics used to compare results, as the test set includes observations that do not belong to any of the training classes. Further information on the conformal prediction framework can be found in 8 and 9 . 3 Perturbation by Noise A common problem in annotated datasets is the presence of errors in the labels of observations. To examine the behavior of the BCOPS algorithm in this scenario, we refer to the work of 2 , which presents empirical results supporting the robustness of conformal prediction under the addition of noise to the labels of the training set. Our concern here is not only to verify compliance with coverage guarantees but also to measure the behavior of the abstention rate in outlier observations, a characteristic that has not yet been assessed. Following Section 2.2 of 2 , we define a corruption function g\left(y\right)=\left\{\begin{array}[]{cl}y&,\ \text{with probability}\ 1-\phi,\\ Y^{\prime}&,\ \text{otherwise,}\end{array}\right. (1) which changes the label y of each observation in the training set with probability \phi , sampling Y^{\prime} from the set of K classes present in the training data, \left\{1,2,\dots,K\right\} , according to a uniform distribution. This means that after applying the corruption function to a dataset, we will have approximately 100\cdot\phi\% noise in the training labels. It is worth noting that our analysis focuses on the simplest case, where an observation is mislabeled in a purely random manner. Other noise patterns in labels can be found in specific applications, some of which are described in 2 . 4 Results We take the corruption function defined in 1 and, as we increase the noise level, we examine the evolution of the coverage rate for each class, defined as \frac{1}{n_{k}}\sum_{i=1}^{n_{k}}\mathbb{I}_{C\left(x_{ik}\right)}\left(y_{ik}\right),\quad k=1,\dots,K, (2) where n_{k} is the number of observations of class k in the test set, \left(x_{ik},y_{ik}\right) is the i -th observation belonging to class k in the test set, C\left(x_{ik}\right) is the prediction set for that observation, and \mathbb{I}_{A}\left(a\right) is the indicator function: \mathbb{I}_{A}\left(a\right)=\left\{\begin{array}[]{cl}1&,\ \text{if}\ a\in A,\\ 0&,\ \text{if}\ a\not\in A.\end{array}\right. (3) This value indicates

Read the complete text