Towards Modeling Data Quality and Machine Learning Model Performance
Usman Anjum, Chris Trentman, Elrod Caden, Justin Zhan · 2024
Read Towards Modeling Data Quality and Machine Learning Model Performance by Usman Anjum, Chris Trentman, Elrod Caden, Justin Zhan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05882.txt.
Introduction
Garbage in, garbage out is the oldest warning in machine learning, yet data quality is rarely given a number a modeler can use. Usman Anjum and co-authors propose such a number in this 2024 paper. Borrowing the signal-to-noise idea from engineering, they define a ratio that weighs the deterministic part of data against its noisy part.
On synthetic data, they show how a model's accuracy moves as that ratio changes. The resulting curves let a user read off expected performance for a given data quality, or the quality a target accuracy demands. The authors aim to make trust in a model something that can be modeled, not just felt, and to guide effort toward the data fixes that matter most.
Catalog record
| CREATOR | Usman Anjum, Chris Trentman, Elrod Caden, Justin Zhan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05882
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Modeling Data Quality and Machine Learning Model Performance Usman Anjum, Chris Trentman, Elrod Caden, Justin Zhan Abstract Understanding the effect of uncertainty and noise in data on machine learning models (MLM) is crucial in developing trust and measuring performance. In this paper, a new model is proposed to quantify uncertainties and noise in data on MLMs. Using the concept of signal-to-noise ratio (SNR), a new metric called deterministic-non-deterministic ratio (DDR) is proposed to formulate performance of a model. Using synthetic data in experiments, we show how accuracy can change with DDR and how we can use DDR-accuracy curves to determine performance of a model. 1 Introduction Machine learning-based models and algorithms are being used widely for many different tasks. They are increasingly being used in sensitive domains like healthcare, finance & banking, security, etc. to make critical decisions. In addition, most of the machine learning algorithms are "black-box" models which means the information of the internal workings is not known. As a result, we may not completely trust a machine-learning algorithm and its results Kaplan et al. [2023] and it becomes necessary to understand the strength and limitations of a model. Biases, strengths, and limitations of machine learning algorithms can also arise from the quality of the data and inductive bias of the algorithm itself. In the machine learning domain, biases have also been referred to as "fairness". Fairness is the absence of any prejudice or favoritism towards an individual or group based on their inherent or acquired characteristics Mehrabi et al. [2021] . An unfair algorithm and low-quality data can result in skewed results that can lead to incorrect conclusions Pagano et al. [2023] . A major reason for the data quality is the uncertainty and non-deterministic nature of data which is caused by the data source, and during data entry or acquisition. The non-deterministic nature of the data is usually considered as noise, but it may not necessarily be noise and there may be inherent properties of the data that are necessary for the analysis. For example, consider weather data that has some uncertainty due to patterns. Non-deterministic data would cause models to have difficulty in understanding and interpreting such data correctly. Most of the current research has focused on how to deal with the problem of non-deterministic nature of data sets. For example, filtering out noise Quinlan [1983] , Quinlan [1986] or enhance the quality by pre-processing by ’hand’ or automatically Zhu and Wu [2004] . There are numerous pre-processing methods to deal with the noise and non-deterministic nature of data, such as eliminating the noisy instances completely and interpolation to correct the non-deterministic data to match the deterministic data points. However, the pre-processing techniques are dependant on the task being performed. Hence, the non-deterministic nature of data can adversely affect the results of any data analysis and reduce accuracy of the model. Understanding the effect of noisy and nondeterministic data in the model can help understand the performance and limitations of a model under uncertain and noisy conditions. Uncertainty in data is an important aspect to consider when measuring model performance. Uncertainty can arise due to imperfect or unknown information Hariri et al. [2019] . Hence, if we can quantify and measure the uncertainty and noise, we could in fact get a true measure of the performance of a model. Current machine learning algorithms only focus on measuring the accuracy of the model, as to how well the predicted data match the actual data Flach [2019] . However, this information may not be enough, especially when we want to measure the performance of a model Brown et al. [2021] . In fact a research conducted by Wu and Keogh [2021] explored the quality of data in anomaly detection and showed that the benchmark dataset used for anomaly detection had very little uncertainty in the data and hence, even the most simple of algorithms performed very well. The results from the paper showed that data quality also needs to be considered when measuring the performance of a model. In this paper, we propose a framework that looks at quantifying data quality by as a function of uncertainty in the data set. In particular, we look at how uncertainty and non-deterministic data can effects accuracy in different model for different tasks and find the correlation between non-deterministic component in a data and the accuracy of the model. We focus on regression and classification tasks as non-determinism effects these tasks in different ways. We begin by defining any data consisting of deterministic and non-deterministic parts. The non-deterministic part of a data can be noise or be present due to stochastic nature of data. The deterministic part of the data is defined by a definitive function. Consequently, we propose a new metric called the Deterministic-Non-Deterministic Ratio (DDR). DDR is basically the ratio of “magnitude" of deterministic and non-deterministic portions of a data. The higher the DDR, the data have a higher deterministic part and consequently a model will be more accurate and able to make predictions. The lower the DDR, the data have a higher non-deterministic part and the model will make less accurate predictions. We prove this hypothesis by running experiments on different models and tasks using synthetic data. We look at the task of regression and classification. Through these experiments we draw DDR-accuracy plots. Using the DDR-accuracy plot, we can determine the performance of a model. The research questions that we aim to answer are: • How does the score of a metric appropriate for a given type of machine learning model and calculated from the target values of a data set appropriate for that type of model and the predictions of the model based on the matrix / vector of feature values of that data set depend on the feature (vector) estimated from the matrix / vector of feature values of that data set? • How does the performance of a machine learning model type in a data set appropriate for that type of model depend on both the feature (vector) of the matrix / vector of feature values of that data set and the value of a metric appropriate for that model type and calculated from the target values of that data set and the predictions of the model estimated from the matrix / vector of feature values of that data set? In summary, the main contributions of our paper are as follows: (1) we propose a new metric that can be used to quantify quality of data; (2) We formulate a model that explores the uncertainty and the determinism in data and its effect on accuracy; and (3) We use this model to measure a model’s performance and conduct different experiments on a variety of tasks and show how our model can be used to show the applicability of our model. 2 Related Works The need to develop a comprehensive technique to measure the performance of a model has been mentioned widely in previous literature Brown et al. [2021] . Currently, the modern way to measure performance is to measure the difference between predicted and actual values. But these methods of measuring performance do not include data quality and the uncertainty. One way of