Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class
Annie D’souza, Swetha M, Sunita Sarawagi · 2024
Read Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class by Annie D’souza, Swetha M, Sunita Sarawagi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15657.txt.
Introduction
Annie D'souza and co-authors find a counterintuitive trick for teaching generative models to invent fair examples of rare classes. Classifiers starve when one category is scarce, and synthetic data is the usual supplement. But deep generators trained on lopsided data produce their worst fakes exactly for the minority that needs help most.
The authors add an explicit overlap class, examples crafted to sit where categories blur, and show this strangely improves minority generation quality. Rather than pretending classes are cleanly separate, the model learns the contested border on purpose. The trick speaks to fraud detection, medical screening, and every domain where the rare case is the one that matters.
Catalog record
| CREATOR | Annie D’souza, Swetha M, Sunita Sarawagi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class Annie D’souza, Swetha M, Sunita Sarawagi Abstract Handling imbalance in class distribution when building a classifier over tabular data has been a problem of long-standing interest. One popular approach is augmenting the training dataset with synthetically generated data. While classical augmentation techniques were limited to linear interpolation of existing minority class examples, recently higher capacity deep generative models are providing greater promise. However, handling of imbalance in class distribution when building a deep generative model is also a challenging problem, that has not been studied as extensively as imbalanced classifier model training. We show that state-of-the-art deep generative models yield significantly lower-quality minority examples than majority examples. We propose a novel technique of converting the binary class labels to ternary class labels by introducing a class for the region where minority and majority distributions overlap. We show that just this pre-processing of the training set, significantly improves the quality of data generated spanning several state-of-the-art diffusion and GAN-based models. While training the classifier using synthetic data, we remove the overlap class from the training data and justify the reasons behind the enhanced accuracy. We perform extensive experiments on four real-life datasets, five different classifiers, and five generative models demonstrating that our method enhances not only the synthesizer performance of state-of-the-art models but also the classifier performance. 0 0 footnotetext: [email protected], [email protected], [email protected] Code — https://github.com/Annie2603/ORD 1 Introduction The problem of imbalance in class distributions is encountered in several fields, including fraud detection in the banking sector, disease diagnosis in the medical sector, and anomaly detection in finance, cybersecurity, and manufacturing industries. Imbalanced data refers to the distribution of classes or categories being highly skewed or disproportionate in a dataset, i.e., one class or category is significantly more prevalent than the others. The presence of imbalance introduces several challenges in training classification models as they are unable to learn the distribution of the rare class. This has been a problem of long-standing interest and many different techniques have been proposed ranging from cost-sensitive loss functions ( Zhou and Liu 2005 ) , oversampling minority instances and/or under-sampling majority instances ( Li et al. 2021 ) , and augmentation with synthetic instances ( Johnson and Khoshgoftaar 2019 ) . In this paper, we focus on data augmentation-based techniques since these are particularly effective ( Chawla et al. 2002 ) . One popular classical data augmentation method, SMOTE, generates new minority instances using a convex combination of existing minority instances. However, recent deep data generators yield even higher accuracy due to the enhanced quality of generated examples. Our focus in this paper is tabular data, and recently many high-quality generators have been proposed for generating tabular data ranging from GAN-based models like CTabGAN ( Zhao et al. 2021 ) to diffusion-based models like TabSyn ( Zhang et al. 2024 ) and ForestFlow ( Jolicoeur-Martineau, Fatras, and Kachman 2024 ) Most of these have been evaluated after training on balanced datasets, where their generated data have been shown to be highly effective in training classifiers that match the accuracy of real data. We are not aware of any study that evaluates them in highly imbalanced input data. In this paper, we show that default training of generative models yields significantly worse quality minority instances than majority instances as measured by the label provided by an Oracle Bayesian model. Simple fixes to the problem by over and under-sampling initial real data distributions are not effective. We present a method called ORD 1 1 1 for Overlapped Region Detection that employs three key ideas to improve the training of imbalanced classifiers on tabular data using synthetic data generators: (1) We pre-process the available real data by identifying a small subset of the majority instances called the overlap set that lies on the boundary of majority and minority examples. (2) We modify the generator to be class conditional and generate three-way labeled instances corresponding to minority, overlapping majority, and clear majority. (3) We train the final classifier with a careful mix of real and synthetically generated instances comprising of balanced sample of synthetic data generated consisting of only the minority and clear majority classes, and real minority. Our method is orthogonal to the type of data generator and classifier, and we evaluate it on four different tabular data generator architectures and five different classification models. We perform a set of insightful ablation studies to tease apart the reason why our method outperforms existing methods. We conclude that our method yields improved accuracy both due to the generator providing higher quality synthetic minority examples, and the classifier improving due to selective under-sampling of overlapping majority examples. Figure 1: The method of ORD and its application to generate synthetic data. Synthetic data is then used to train a better classifier compared to when no ORD is used. Steps in the overall algorithm: 1. To use k-fold training to identify the overlap in the validation set. This uncertainty is labelled as a third class label D_{01} . 2. Generate synthetic data using the three class dataset instead of binary class. The synthetic data quality is much better as the distribution is better reproduced by Generative models. 3. Train the final classifier with equal proportions of majority and minority synthetic data while discarding the overlapped region D_{01} which makes learning the decision boundary easier. The main contributions of this paper are as follows: • We evaluate state-of-the-art deep generative models for tabular data on imbalanced training data, and show that the quality of minority examples they generate is much worse than majority examples. • We propose a new method called ORD that introduces a new class comprising of the subset of majority examples that lie in the boundary of the two classes, harnesses these to impose a ternary classification loss when training the generative model, and finally trains the classifier using generated examples that lie outside the overlap region. • We propose a new SOTA model CTabSyn , a variation of TabSyn ( Zhang et al. 2024 ) with the inclusion of ORD and conditional training and generation. • We provide extensive experimental evaluation and proof that ORD provides significant increase in accuracy. • We justify the enhanced accuracy by showing that our method both improves the quality of minority examples generated from our method, and the classifier gains from selective under-sampling of points in the overlap region. 2 Related Works Traditionally, for training classifiers on imbalanced data, either the minority class is oversampled ( Chawla et al. 2002 ) or majority class is undersampled. While efficient to some extent, majority under-sampling methods can discard potentially useful instances. Under sampling methods include cluster-based ( Zhang, Zhang, and Wang 2010 ) under-sampling and Neighbourhood ( Vuttipittayamongkol and Elyan 2020 ) based boundary undersampling. While over sampling methods include Tomek link based oversampling ( Leng et al. 2024 ) .Over-sampling can be ineffective when number of minority points is very small. In such cases, a more effective alternative is generating synthetic examples. A pioneering method in this category is SMOTE (for Synthetic Minority Oversampling Technique) ( Chawla et al. 2002 ) that interpolates between minority points to generate new