Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile

Andrea Maurino · 2026

Read Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile by Andrea Maurino (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25765.txt.

Introduction

Maurino proposes the Error Sensitivity Profile, a measure of how sensitive a classification model's performance is to errors in one feature or in several features. Data quality is critical to model performance, yet cleaning efforts need guidance on which errors matter most. The profile lets teams prioritize cleaning by error type and by feature, based on likely impact. To compute it, the author builds an integrated tool suite called dirty. Tests on two standard datasets with 14 classification models show performance loss is not always predictable from simple correlations.

Catalog record

CREATORAndrea Maurino
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29703
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25765

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile Andrea Maurino Abstract The quality of training data is critical to the performance of machine learning models. In this paper, the Error Sensitivity Profile (ESP) is proposed. It quantifies the sensitivity of model performance to errors in a single feature or in multiple features. By leveraging ESP, data-cleaning efforts can be prioritized based on error types and features most likely to affect model performance. To support the computation of this metric, an integrated suite of tools, called Dirtify , is created. We conduct an extensive experimental study on two widely used datasets using 14 classification models, revealing that performance degradation is not always predictable from simple correlations with the target variable. 1 Introduction Multiple studies have explored the influence of data errors on the efficacy of machine learning (ML) models [ 10 , 13 , 9 ] . These studies suggest that, due to the pronounced nonlinearity of datasets and models, it is challenging to generalize about how data quality affects model performance. Consequently, a dataset with certain errors might perform well with one machine learning model but poorly with another. Similarly, two datasets with the same error types, when used to train the same ML model, may yield significantly different results. In this paper, we propose the Error Sensitivity Profile (ESP), a new method for assessing and comparing the impact of errors on ML performance. ESP is able to 1) detect for a given model trained on a dataset contaminated by a type of error applied to one or more features if there is a linear relationship between the percentage of errors and the performance degradation (Error Performance Correlation EPC), 2) calculate the performance that globally the model miss or gain with the increase of the error level (Area under Curve Error-Performance AEPC), and 3) the behavior of the model according to the different level of errors. With ESP, both practitioners and researchers can compare the effects of training data quality across models and datasets. To fully exploit the ESP, we developed Dirtify a suite of Python-based tools to aid specialists in examining the impact of specific dataset errors on the performance of specific Machine Learning models. This is facilitated through a Python library named PuckTrick [ 2 ] , which is capable of injecting predefined percentages of specific errors. The analysis of two widely used datasets across 14 ML models demonstrates the effectiveness of ESP as a tool for assessing the impact of data quality on machine learning models. The study confirms the Dirtify suite’s ability to support this type of research. This paper is structured as follows: Section 2 examines the current state of the art. Section 3 introduces a formal definition of ESP, while the tool suite and its key architectural components are outlined in Section 4 . Section 5 displays and analyzes the principal results from the empirical analysis. Lastly, Section 6 concludes the discussion and outlines potential future works 2 State of the Art Prior research has explored the impact of different types of dirty data on ML models, including missing values [ 10 , 1 ] , inconsistent data [ 10 ] , outliers [ 3 ] , label noise [ 7 ] , and categorical duplicates [ 13 ] . Frameworks for sensitivity analysis have been proposed in [ 6 ] , but are limited to a specific domain. While these works addressed data quality issues in various ways, their analyses focused on a single error type; additionally, none have developed a flexible tool for dataset contamination and assessment comparable to Dirtify . Recently, [ 9 ] investigated the impact of six data quality dimensions across multiple ML tasks, evaluating five classification algorithms (Logistic Regression, MLP, SVM, TabNet, and KNN) on a fixed collection of datasets. Although complementary in spirit, the two approaches differ along several dimensions. First, [ 9 ] operates at the dataset level, producing aggregate scalar indicators of quality impact, whereas ESP operates at the feature level and provides a multi-dimensional profile that jointly captures the direction, cumulative magnitude, and local structure of the error–performance relationship. Second, the quality dimensions adopted in [ 9 ] conflate structurally distinct error types, for instance, outliers and noisy values are subsumed under the single dimension feature accuracy, whereas ESP treats each error type as a separate, independently controllable corruption operator, enabling finer diagnostic resolution. Third, ESP is evaluated across 14 classification models, providing broader coverage of the algorithmic landscape. These differences make ESP and [ 9 ] complementary rather than competing: [ 9 ] offers a dataset-level view of quality impact across a curated benchmark, while ESP provides a fine-grained, model-specific sensitivity profile designed to guide feature-level data cleaning decisions. CleanML [ 8 ] evaluates the impact of cleaning algorithms applied to real-world dirty datasets, using repeated runs and FDR control to ensure statistical rigour. The ESP profile can also be computed on top of experimental protocols such as CleanML, provided that the underlying methodology generates error–performance curves at controlled corruption levels. Among tools for data corruption, BART [ 4 ] allows users to specify error types and distributions via denial constraints, but operates only on relational tables and does not evaluate ML tasks. Jenga [ 12 ] is a Python library that introduces data corruptions into datasets to evaluate model robustness, but tests a single model at a time and lacks a configuration interface for error strategies. The PuckTrick library [ 2 ] , on which Dirtify relies, provides a broader range of error types than Jenga, including label perturbations and duplicated data. 3 Error Sensitivity Profile Let D_{0} be a cleaned training dataset (our baseline) and let \mathcal{T}_{\theta}:D\rightarrow D be the family of corruption operators, where \theta is a vector of parameters (e.g., error rate, error type, affected features). For each error rate k=1,\ldots,K , we define D_{k}=\mathcal{T}_{\theta_{k}}(D_{0}) as a corrupted training dataset. Let f_{k}=A(D_{k};s_{k}) be a model based on algorithm A trained on D_{k} with random seed s_{k} , and let p_{k}=\mathrm{Perf}(f_{k},D_{\mathrm{test}}) be a performance metric evaluated on a clean test set D_{\mathrm{test}} . We define a scalar measure of the training data degradation induced by \mathcal{T}_{\theta_{k}} as e_{k}=\mathrm{Err}_{\tau}(D_{k},D_{0}) , where \tau denotes the error type (e.g., label noise, missing values, outliers), and \mathrm{Err}_{\tau}:D\times D\rightarrow\mathbb{R}^{+} is an error-specific degradation function that quantifies the severity of the corruption with respect to the clean baseline D_{0} . In the experimental evaluation presented in this paper, \mathrm{Err}_{\tau}(D_{k},D_{0}) is instantiated as the percentage of corrupted values injected into the training set, i.e. e_{k}\in\{0,20,40,60,80\}\% , which provides a transparent and reproducible operationalisation of the corruption severity. Given a learning algorithm A , a baseline training dataset D_{0} , a corruption type \tau , and an associated family of corrupted datasets \{D_{k}\}_{k=1}^{K} , we define the Error Sensitivity Profile of A with respect to \tau as the tuple: \mathrm{ESP}(A,D_{0},\tau)=\Bigl\langle\,\mathrm{EPC},\;\mathrm{AEPC},\;\{\hat{\beta}_{j}\}_{j=1}^{J}\,\Bigr\rangle, (1) where each component isolates a specific, complementary facet of the error–performance relationship, as detailed below. 3.1 Error Performance Correlation The Error Performance Correlation (EPC) provides a global, first-order summary of the relationship between data corruption and model performance. Specifically, EPC is defined as the negated Pearson correlation coefficient between the error severity sequence \{e_{k}\}

Read the complete text