FILM: Framework for Imbalanced Learning Machines based on a new unbiased performance measure and a new ensemble-based technique

Antonio Guillén-Teruel, Marcos Caracena, Jose A. Pardo, Fernando de-la-Gándara, José Palma, Juan A. Botía · 2025

Read FILM: Framework for Imbalanced Learning Machines based on a new unbiased performance measure and a new ensemble-based technique by Antonio Guillén-Teruel, Marcos Caracena, Jose A. Pardo, Fernando de-la-Gándara, José Palma, Juan A. Botía (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04370.txt.

Introduction

Guillén-Teruel and co-authors build a framework for learning from lopsided datasets, in a 2025 machine learning paper. When one class vastly outnumbers the other, standard scores flatter bad models. Testing seven datasets, the team found common metrics disagree about which model wins. Their new score merges existing metrics while penalizing the ones most skewed by imbalance, and it showed far less minority-class bias in tests.

A companion algorithm builds ensembles from identical data partitions tuned for imbalance problems. It beat rival imbalance-aware methods on three of seven datasets under the new score, using two standard classifier types. Both tools ship in an R package called FILM. The authors aim to make model choice on skewed data a fairer contest.

Catalog record

CREATORAntonio Guillén-Teruel, Marcos Caracena, Jose A. Pardo, Fernando de-la-Gándara, José Palma, Juan A. Botía
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FILM: Framework for Imbalanced Learning Machines based on a new unbiased performance measure and a new ensemble-based technique Antonio Guillén-Teruel, Marcos Caracena, Jose A. Pardo, Fernando de-la-Gándara, José Palma, Juan A. Botía Abstract This research addresses the challenges of handling unbalanced datasets for binary classification tasks. In such scenarios, standard evaluation metrics are often biased by the disproportionate representation of the minority class. Conducting experiments across seven datasets, we uncovered inconsistencies in evaluation metrics when determining the model that outperforms others for each binary classification problem. This justifies the need for a metric that provides a more consistent and unbiased evaluation across unbalanced datasets, thereby supporting robust model selection. To mitigate this problem, we propose a novel metric, the Unbiased Integration Coefficients (UIC), which exhibits significantly reduced bias ( p < 10 − 4 p<10^{-4} ) towards the minority class compared to conventional metrics. The UIC is constructed by aggregating existing metrics while penalising those more prone to imbalance. In addition, we introduce the Identical Partitions for Imbalance Problems (IPIP) algorithm for imbalanced ML problems, an ensemble-based approach. Our experimental results show that IPIP outperforms other baseline imbalance-aware approaches using Random Forest and Logistic Regression models in three out of seven datasets as assessed by the UIC metric, demonstrating its effectiveness in addressing imbalanced data challenges in binary classification tasks. This new framework for dealing with imbalanced datasets is materialized in the FILM (Framework for Imbalanced Learning Machines) R Package, accessible at https://github.com/antoniogt/FILM. keywords Machine Learning, Imbalanced problems, Ensemble-based methods, Classification,Performance metrics 1 Introduction: The class imbalance problem Machine learning is the combination of statistics, computer science and algorithms to learn patterns from data. How effective is this process is greatly influenced by imbalanced data. In these problems, some of the concepts we aim to learn from are underrepresented in comparison with the rest. In such cases, the resulting machine learning models tend to focus on the majority class, simultaneously forgetting the minority class. Imbalanced problems are frequent in real life machine learning problems. This is specially true for binary classification problems ( Chawla et al., (2004) ; García et al., (2010) ; Van Hulse et al., (2007) ; Y. Tang et al., (2009) ; Eitrich & Lang, (2006) ; Maalouf & Siddiqi, (2014) ) . In such case, the distribution of instances between the two classes is skewed, resulting in a disproportionate number of instances in one of the classes. All these usually leads to biased results and poor performance. There are also imbalanced regression problems. In regression tasks ( Branco et al., (2017) ; Song et al., (2022) ) , the issue of imbalanced datasets arises when the distribution of the target variable is significantly skewed. And again, the resulting machine learning model will show disproportionate better performance on the examples with better representation of the dependent variable. In definitive, it is essential to address this issue from the early stages of the data analysis tasks in any machine learning project. One of the domains that are critically affected by imbalances in the data is the medical domain. Examples of these are those relative to case/control cohorts of subjects of a particular disease, intervention or monitoring activity. This is usually due to the fact that the most important class is usually the minority class. In medical datasets, the consequences of biased results and poor model performance are significantly more severe ( Wardhani et al., (2019) ; Thölke et al., (2023) ) . For instance, consider the mammogram dataset example from ( He & Garcia, (2009) ) , with a distribution of 99\% negative cases and a little 1\% positive cases. A standard machine learning algorithm applied to this dataset could potentially yield an impressive accuracy of 99\% , merely by predicting all cases as negative. This approach, however, would fall short of the main goal: correctly identifying the much rarer, but crucial, positive cases while maintaining a good performance on the negative cases. These situations require of special approaches to be taken. On the one hand, it is necessary to appropriately measure performance with class-imbalance aware metrics ( Luque et al., (2019) ; Soleymani et al., (2020) ) . On the other hand, apart from a correct estimation of the performance, we also need class-imbalance aware data preparation techniques and machine learning algorithms ( Tarekegn et al., (2021) ) . One of the consequences of not using class-imbalance aware metrics is that those that are actually used become biased. Widely used performance metrics such as accuracy, precision, recall, area under the receiver operating characteristics (rocAUC) or F1-score can be heavily biased towards the majority class in imbalanced datasets ( Jeni et al., (2013) ) . We will demonstrate it through the paper but we can easily see this Fig. 1 in which we represent three of the most commont performance measures (accuracy, F1 and rocAUC) of a conventional random forest ( Breiman, (2001) ) with automated grid-based hyperparameter optimization, used on one of the datasets we will use for experimentation (SMS, see Table 1 and Sec. 3.1 ) in which we artificially vary the imbalance. There we see a statistically significant association between the proportion of the minority class (from now on p_{min} , at the x axis) and performance values, at the y axis. The metric rocAUC even shows opposite direction with the other two. This simple experiment points to two different issues. Firstly, there is a clear association between level of imbalance and inflation (or alternatively depletion) of performance estimates. Secondly, apart from the bias, we can see certain lack of agreement across metrics as shown in previous studies ( Gaudreault & Branco, (2024) ; Tian et al., (2024) ) . All this eventually leading to incorrect conclusions. Therefore, and following recommendations from previous recent work ( B. Liu & Tsoumakas, (2020) ; Lee & Seo, (2022) ) we propose using a set of complementary performance metrics through a new approach called UIC (Unbiased Integration Coefficients) whose aim is providing a single, p_{min} unbiased technique, which integrates the basic metrics through a weighted sum. In subsequent sections we will empirically test the possible association of the UIC measure to p_{min} Figure 1: Values of different metrics training Random Forest with upsample for SMS dataset. Examples of widely used imbalance-aware algorithms are SMOTE, ROSE, Upsample and Downsample. You can see the previous works for a detailed description but basically, undersampling the data may remove important examples from the dataset increasing noise whereas upsampling tends to produce overfitting and unnecessary data overlaps. To contribute to the arena of available algorithms, we include in this work a new algorithm based on resampling the original dataset ( Alamri & Ykhlef, (2022) ) , which avoids the drawbacks just mentioned. It is called IPIP (Identical Partitions for Imbalance Problems) and it creates ensembles of models trained on balanced subsets of the original. It is a new CPU-intensive approach but convenient in certain types of tasks (see Sec. 3.4 ). The rest of this paper is organized as follows. Sec. 2 surveys the diverse methodologies and methods currently being employed to address imbalance problems. Sec. 3 presents the datasets chosen for the study, the metrics chosen for the study along with the

Read the complete text