Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts

Taylor Maxson, Roberto Corizzo, Yaning Wu, Nathalie Japkowicz, Colin Bellinger · 2026

Read Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts by Taylor Maxson, Roberto Corizzo, Yaning Wu, Nathalie Japkowicz, Colin Bellinger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26024.txt.

Introduction

Maxson and four colleagues fix bias in imbalanced classification with minority subconcepts. Class scores hide gaps across subconcepts in one class. Earlier scores favor larger minority subconcepts; true-label weighting helps, but such labels rarely exist at test time. The authors use predicted posteriors from a multi-class subconcept model. Weights are mean utility under that posterior, giving the soft predicted-weighted balanced accuracy (pBA) metric. On tabular, imaging, and text data, plain scores mislead in mixed classes, while pBA gives stabler reads.

Catalog record

CREATORTaylor Maxson, Roberto Corizzo, Yaning Wu, Nathalie Japkowicz, Colin Bellinger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43549
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26024

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts Taylor Maxson, Roberto Corizzo, Yaning Wu, Nathalie Japkowicz, Colin Bellinger Abstract Class-level evaluation can conceal substantial performance disparities across subconcepts within the same class, causing models that perform well on average to fail on specific subpopulations. Prior work has shown that common evaluation measures for imbalanced classification are biased toward larger minority subconcepts and that utility-based reweighting using true subconcept labels can mitigate this bias; however, such labels are rarely available at test time. We introduce a practical utility-weighted evaluation that replaces unavailable subconcept labels with predicted posterior probabilities from a multiclass subconcept model. Evaluation weights are defined as the expected utility under this posterior, yielding a soft, uncertainty-aware metric we call predicted-weighted balanced accuracy (pBA). Experiments on tabular benchmarks as well as medical-imaging and text datasets show that unweighted scores can be misleading under within-class heterogeneity, while pBA provides more stable and interpretable assessments when subconcept distributions are uneven but not pathological. Our code is available at: https://anonymous.4open.science/r/correcting-bias-imbalance-9C6C/ Keywords: Class imbalance; minority subconcepts; performance evaluation; bias; group fairness. 1 Introduction The class imbalance problem has been studied for decades, yet the quality of an imbalanced classifier is still usually summarized with evaluation measures that operate at the class level [ 14 , 15 , 9 , 20 , 2 ] . This is often too coarse. In many real applications, a class is itself heterogeneous and contains multiple subconcepts with different sizes, geometries, and difficulty levels. A medical screening system, for example, may distinguish healthy from diseased patients with high average performance, while still performing poorly on rare disease subtypes. In such cases, a single class-level score may suggest reliable deployment even though some clinically meaningful subpopulations are underserved. This paper focuses on this particular issue: performance disparity within a class . As a running example, consider the common situation in which a practitioner reports a strong test score, say around 90%, and reasonably concludes that any future sample should also be handled well on average. Yet this interpretation can be misleading when the target class contains multiple subconcepts. Some subpopulations may receive substantially lower performance than the score suggests, and standard unweighted measures provide little warning that this is happening. A model can therefore look successful at the dataset level while remaining unreliable for specific groups embedded inside one class. Prior work [ 1 ] showed that this is not merely a fairness concern but also an evaluation problem: standard measures for imbalanced classification can be biased toward the largest minority subconcepts. That study further showed that instance weighting can reduce this bias when the true subconcept labels of the test instances are known. At the same time, it revealed several lessons that motivate the present study. First, poor minority-subconcept coverage is not the whole story: sometimes the majority class is itself highly complex, so the relevant source of difficulty is distributed across the full class geometry rather than concentrated only in rare groups. Second, size alone does not determine difficulty: some small subconcepts are actually very easy, meaning that blindly distrustful treatment of all rare subconcepts is also unwarranted. Third, visualization-based analyses such as t-SNE can be useful for diagnosing when score disparity is driven by genuine overlap, isolation, or heterogeneous local structure rather than by frequency alone. These observations suggest that an ideal performance measure should account for subconcept size without assuming that every rare subconcept is hard or that every large one is representative. In other words, the goal is not simply to replace one uniform score with another, but to construct an evaluation that reflects how performance is distributed over the latent subpopulations that make up a class. This is especially important in applications where deployment decisions are made from aggregate test scores and where underperformance on a small but important subgroup may carry disproportionate risk. This work argues that unweighted scores can be misleading in the presence of subconcept complexity and within-class heterogeneity. To correct this, we build on previous work to develop a utility weighting strategy that only assumes that subconcept labels are available in the training data. We propose a practical utility-weighted evaluation that replaces unavailable test-time subconcept labels with predicted posterior probabilities from a multiclass subconcept model. Evaluation weights are defined as the expected utility a_{x}=\sum_{s\in\mathcal{S}}q_{s}(x)\,u_{s} , yielding a soft, uncertainty-aware utility weighting measure that avoids brittle hard assignments while preserving sensitivity to performance on rare but important subconcepts. We evaluate the proposed method on real-world medical-image and multimodal hate-speech/text datasets as well as the Keel and PMLB benchmark datasets. Multiclass datasets are transformed into binary class-imbalance problems while preserving knowledge of the underlying subconcepts. Our method is compared to standard imbalanced measures and the utility weighted measure based on true subconcept labels used in the prior work. We study both whether utility weighting corrects the optimism of unweighted scores and when that correction should be expected to work. From this analysis, the following research questions are studied: - RQ1: Can utility weighting based only on training-time subconcept information reduce the bias of standard unweighted measures toward larger minority subconcepts? - RQ2: How does the accuracy of the subconcept predictor affect the reliability of the resulting predicted utility weights? - RQ3: How do subconcept geometry, subconcept-specific difficulty, and class complexity affect when weighted and unweighted measures agree or disagree? The remainder of the paper is organized as follows. Section 2 reviews related work, Section 4 describes the experimental methodology, Section 5 presents the empirical results and their implications for evaluation under within-class heterogeneity, and Section 6 concludes the paper. 2 Related Work This paper builds on imbalanced learning, where class skew, overlap, noise, low-density regions, and small disjuncts can all affect minority-class performance [ 7 , 2 , 17 , 13 , 4 ] . It is most directly connected to work on within-class imbalance and small disjuncts [ 8 , 10 ] , and extends [ 1 ] , which showed that standard imbalanced-classification measures can be biased toward larger minority subconcepts when test-time subconcept labels are known. Related concerns appear in hidden-stratification work in medical imaging [ 18 ] , long-tailed learning methods that address nonuniform class frequencies [ 5 , 3 , 21 , 11 ] , and group-robust or subpopulation-shift evaluation, where strong average performance can coexist with high error on particular groups [ 22 , 12 , 16 ] . The focus here is complementary: rather than training a more robust classifier directly, training-time subconcept information is used to obtain a more informative evaluation of a fixed classifier when testing subconcept labels are unavailable. 3 Preliminaries We consider a binary classification problem with labels y_{i}\in\{0,1\} , where class 1 denotes the minority class. Within each class, and in particular within the minority class, instances may belong to distinct subconcepts (e.g., semantic subgroups or latent modes) that are themselves unevenly represented. Our goal is to evaluate classifiers in a way that: (i) treats the two classes symmetrically despite class imbalance, and (ii) mitigates bias toward large subconcepts by increasing the influence of smaller subconcepts during evaluation. To this end, we define a utility-weighted imbalance evaluation measures. In this section, we demonstrate the utility-weighted balanced accuracy measure

Read the complete text