Beyond Synthetic Augmentation: Group - Aware Threshold Calibration for Robust Balanced Accuracy in Imbalanced Learning

Hunter Gittlin(✉) · 2025

Read Beyond Synthetic Augmentation: Group - Aware Threshold Calibration for Robust Balanced Accuracy in Imbalanced Learning by Hunter Gittlin(✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02592.txt.

Introduction

Hunter Gittlin wrote a 2025 solo machine learning paper. It argues that class imbalance is better handled by threshold choices than by fake data. Class imbalance means one outcome dwarfs the other in training data. Synthetic oversampling can create new problems of its own.

The proposed fix sets a distinct threshold for each demographic group. Across experiments, group-aware thresholds beat models trained with SMOTE or CT-GAN synthetic data. Balanced accuracy rose. Worst-group scores improved too. A single global threshold cannot trade these goals as finely. Adding group thresholds on top of synthetic data gained almost nothing. So the two fixes overlap. Results held across seven model families. They span linear models through boosting.

Catalog record

CREATORHunter Gittlin(✉)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Synthetic Augmentation: Group - Aware Threshold Calibration for Robust Balanced Accuracy in Imbalanced Learning Hunter Gittlin(✉) Abstract Class imbalance remains a fundamental challenge in machine learning, with traditional solutions often creating as many problems as they solve. We demonstrate that group-aware threshold calibration—setting different decision thresholds for different demographic groups—provides superior robustness compared to synthetic data generation methods. Through extensive experiments we show that group-specific thresholds achieve 1.5-4% higher balanced accuracy than SMOTE and CT-GAN augmented models while improving worst-group balanced accuracy. Unlike single-threshold approaches that apply one cutoff across all groups, our group-aware method optimizes the Pareto frontier between balanced accuracy and worst-group balanced accuracy, enabling fine-grained control over group-level performance. Critically, we find that applying group thresholds to synthetically augmented data yields minimal additional benefit, suggesting these approaches are fundamentally redundant. Our results span seven model families including linear, tree-based, instance-based, and boosting methods, confirming that group-aware threshold calibration offers a simpler, more interpretable, and more effective solution to class imbalance. 1 Introduction Figure 1: Left: Classification of male and female non-defaulters and defaulters in a single threshold setting. This single threshold performs poorly, misclassifying three out of seven individuals (roughly 57% accuracy). Right: Classification of male and female non-defaulters and defaulters in a group thresholding scenario, in which there is a separate threshold for males and females. In this case, the model correctly classifies all individuals as either defaulting or non-defaulting. Algorithmic scores increasingly gate access to credit, welfare, and jobs. When they underrate certain groups they block housing, entrepreneurship, and civic participation, breaching fairness mandates such as the EU AI Act and the U.S. Equal Credit Opportunity Act. A frequent technical culprit is class imbalance: if 99% of records are legitimate, a model that predicts no default everywhere enjoys 99% accuracy while failing exactly where oversight matters. We propose a practical alternative— group-aware threshold calibration —that assigns separate decision cut-offs to each protected group, outperforms heavyweight oversampling pipelines, and gives auditors an explicit lever over group-level error rates. Experiments on two financial benchmarks show that responsibly leveraging sensitive attributes in this way lifts both overall and worst-group balanced accuracy, reinforcing evidence that such attributes can be a powerful instrument for fairness. These results challenge blanket bans on their use, suggesting that tightly regulated access—rather than wholesale exclusion—may be essential for the equity goals those laws pursue, even as many practitioners still default to synthetic oversampling. Synthetic approaches like SMOTE [ 2 ] and CT-GAN [ 16 ] attempt to balance datasets by creating artificial minority class samples. However, recent evidence suggests these methods introduce problematic artifacts. As demonstrated across 71 datasets, oversampling methods often lead to overfitting and poor generalization, with the authors concluding that oversampling should be avoided in real-world applications [ 9 ] . As these synthetic samples often create overlapping class regions that confuse decision boundaries rather than clarifying them. We propose a fundamentally different approach: rather than manipulating training data and hoping for improved outcomes, we directly optimize for balanced accuracy through group-aware threshold calibration. Unlike single-threshold approaches that apply one decision boundary across all samples, group-aware thresholds recognize that different groups may require different decision criteria due to varying base rates or feature distributions [ 8 ] . Fig. 1 illustrates this concept concretely. 2 Background and Related Work 2.1 The Limitations of Accuracy in Imbalanced Settings Traditional accuracy fails catastrophically with class imbalance. Consider a dataset with 95% negative and 5% positive examples—a classifier predicting all negatives achieves 95% accuracy while completely failing to identify any positive cases. This paradox [ 7 ] has motivated alternative metrics that give equal importance to all classes. Balanced accuracy addresses this by averaging per-class accuracies: BA=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right)=\frac{TPR+TNR}{2} (1) This formulation ensures that a trivial all-negative classifier achieves only 50% balanced accuracy, properly reflecting its failure on the positive class. Recent work by [ 13 ] connects balanced accuracy optimization to distributional robustness, providing theoretical foundations for its use in fairness-critical applications. The connection between worst-group performance and distributional robustness has been further established by [ 11 ] , who show that optimizing for worst-group accuracy provides guarantees against distribution shift. 2.2 Synthetic Data Generation: Promise vs. Reality SMOTE generates synthetic minority samples by interpolating between existing instances and their k-nearest neighbors. While intuitive, this approach suffers from several fundamental limitations. [ 1 ] showed that SMOTE can increase classifier variance without improving minority class recognition in high-dimensional settings. More recently, the comprehensive GHOST study by [ 6 ] tested 138 datasets and found that threshold optimization significantly outperformed SMOTE for 75% of ML methods tested. A critical limitation identified by [ 10 ] is that SMOTE and similar oversampling methods lead to poorly calibrated probability estimates, with significantly worse Log-Loss and Brier scores compared to threshold-based approaches. This calibration degradation is particularly problematic for applications requiring meaningful confidence scores, such as medical diagnosis or financial risk assessment. CT-GAN attempts to address these issues through conditional generative adversarial networks, learning the underlying data distribution rather than simple interpolation. However, [ 5 ] found that even sophisticated generative models struggle with tabular data, often producing unrealistic feature combinations that don’t respect complex dependencies in real-world datasets. Our experiments confirm these findings, revealing that CT-GAN fails to improve upon simple threshold optimization despite its computational complexity. 2.3 Group-Aware Threshold Optimization Traditional threshold optimization applies a single cutoff across all predictions. However, when protected groups exhibit different base rates or feature distributions, this one-size-fits-all approach can perpetuate disparities. Group-aware threshold optimization addresses this by learning separate thresholds for each demographic group, a approach that [ 3 , 8 ] show can achieve optimal fairness-accuracy trade-offs under certain conditions. Formally, for groups g\in\{1,...,G\} and predicted probabilities p_{i} , we learn thresholds \tau_{g} such that: \hat{y}_{i}=\mathbb{1}[p_{i}\geq\tau_{g(i)}] (2) where g(i) denotes the group membership of instance i . This enables fine-grained control over group-level true positive and false positive rates. 3 Methods 3.1 Experimental Setup We evaluate our approach on two benchmark datasets with natural class imbalance and protected groups. The UCI Default of Credit Card Clients dataset contains 30,000 instances with 22.1% default rate using sex as the protected attribute. The Adult Income dataset includes 48,842 instances with 24.1% high-income rate, using race as the protected attribute. For each dataset, we implement four approaches. The baseline uses models trained on original imbalanced data with a single threshold. SMOTE applies group-aware oversampling to balance classes per demographic, implemented by separately generating synthetic samples for male and female subgroups to avoid information leakage between groups, following recommendations by [ 4 ] . CT-GAN performs conditional generation with demographic conditioning, where we train the generative model for 5 epochs and condition on the protected attribute to generate synthetic positive samples proportional to each group’s representation. Finally, we apply group-aware threshold optimization to all data variants. We employ 5-fold stratified cross-validation with 20% test sets. Within training data, we reserve 12.5% for threshold optimization using OxonFair’s grid search over group-specific thresholds, ensuring no data leakage [ 4 ] . The validation set provides unbiased estimates for threshold selection. 3.2 Model Families To ensure findings generalize across algorithmic paradigms, we test several diverse classifiers spanning linear methods (Logistic Regression, SGD Classifier), tree-based approaches (Random

Read the complete text