Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification

Samuel McDowell, Nathan Stromberg, and Lalitha Sankar · 2026

Read Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification by Samuel McDowell, Nathan Stromberg, and Lalitha Sankar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21742.txt.

Introduction

Samuel McDowell, Nathan Stromberg, and Lalitha Sankar tackle class imbalance in prior-data fitted networks, which classify tables from context instead of weight updates. Rare classes suffer most when classes are uneven, yet loss-based fixes do not fit these models' in-context learning. The authors adapt several classical remedies and test them on tabular tasks. Thresholding works exceptionally well, thanks to the models' calibration, while downsampling matches it and cuts inference cost. The findings give practitioners two proven, cheap remedies for skewed tabular data.

Catalog record

CREATORSamuel McDowell, Nathan Stromberg, and Lalitha Sankar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23097
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification Samuel McDowell, Nathan Stromberg, and Lalitha Sankar Abstract Prior-data fitted networks (PFNs) have achieved exceptional performance on tabular classification tasks. However, like other classifiers, their performance can suffer under the effect of class imbalance, resulting in poor performance for rare classes. Several techniques exist which attempt to mitigate the deleterious effect of class imbalance on classification performance, but the in-context learning (ICL) dynamic of PFNs means that loss-based strategies are impossible, and other techniques are unproven. We have adapted several classical techniques addressing class imbalance and analyzed their performance on PFN classification. We observe that thresholding performs exceptionally well because of the calibration characteristics of PFNs, and downsampling performs comparably because of PFNs exceptional limited-data performance, with the additional benefit of reduced computation cost for inference. I Introduction Prior-data fitted networks (PFNs) have become ubiquitous in critical areas such as tabular data inference [ 2 ] and timeseries forecasting [ 7 ] . Specifically, TabPFN (and its variants) has demonstrated extraordinary performance on tabular classification and regression tasks without updating any weights and with very little task-specific training data. PFNs achieve this outstanding efficiency by pretraining on copious amounts of synthetic data specifically for the task of in-context learning. By training on synthetic data from only highly structured causal models, PFNs implicitly learn to estimate the posterior predictive distribution (PPD) over a prior determined by the class of data generating models. The relationship between the pretraining model class and the final prediction is critical, as the predictions of the classifier will follow the structure of the data generating models used in training. In this way, controlling the pretraining prior controls the hypothesis space of the classification. Another key architectural aspect of PFNs is their reliance on in-context learning (ICL). ICL is a form of meta-learning where the PFN model is pretrained not to predict outputs for a fixed task, but to predict the relationship between a labeled context set and an unlabeled query. This results in a model whose weights are not updated at all for a new task; instead, the prediction relies on a relatively few labeled examples (context) given to the model. This is distinct from classical in-weight learning where the task-specific examples are used to update model weights. ICL is typically performed using transformer models, so the relationships between context and query is captured using the attention mechanism. While PFNs achieve state of the art performance on tabular classification, like most classification models, they perform poorly on class-imbalanced data, where some classes are vastly more prevalent than others. In fact, these models may achieve reasonable or even stellar average accuracy, but the performance on the minority classes can be significantly degraded due to limited samples. This, in turn, limits detection of rare classes (e.g., rare disease or cyberattack detection). Addressing the effects of class imbalance falls in three major categories: loss-, data-, or decision-based methods. Loss reweighting is a classical method that falls in the first category, and involves upweighting the loss on minority class samples such that they have the same influence on the model as the majority. Data-level methods include downsampling the majority size to match that of the minority as well as generating synthetic minority samples. Decision-level methods involve manipulating the output of the model, such as scaling/tilting the soft score from the classifier. All of these methods have merit; however, since the learning dynamics of ICL are distinct from in-weight learning, some of these techniques, in particular loss reweighting, cannot be applied to this setting. Recently, [ 4 ] addressed the performance of TabPFNs on rare classes by evaluating the efficacy of generating synthetic minority samples. This technique is limited by the effectiveness of the method used to generate the synthetic samples, as any distortion of the distribution of the synthetic samples will affect the downstream classification. It is also computationally expensive, since a sample must be generated for each additional sample in the overrepresented classes. We begin by investigating the unique calibration characteristics of PFNs. This motivates our empirical evaluation of theoretically-grounded and practically useful correction methods including thresholding, downsampling, oversampling, and synthetic upsampling (all defined formally in Section II-E ). For binary classification tasks, our theory-guided experimental results show that thresholding achieves the best performance, with a drastic increase in minority class performance with a minimal decrease in majority performance. Downsampling also performs well, achieving the highest worst-class accuracy with only a slight decrease in balanced performance and the additional benefit of decreasing inference computation cost by reducing the number of context samples. II Problem Setup II-A Prior-Data Fitted Networks (PFNs) PFNs are a model class trained on a Bayesian prior over supervised learning tasks, typically with large-scale synthetic datasets, so that they can use in-context learning (ICL) to predict the posterior predictive distribution (PPD) directly [ 5 ] . During training, these models are given a set of context points drawn from a distribution and trained to predict the masked label of a query point drawn from the same distribution. Minimizing a standard cross entropy loss over a variety of distributions leads the model to learn P(y\mid x,D) , the PPD of the context D=\{x_{c},y_{c}\}_{i=1}^{n} and query x . Formally, the PPD can be written as [ 3 ] : P(y|x,D)\propto\int_{\Phi}P(y|x,\phi)P(D|\phi)P(\phi)d\phi, (1) where y is the class label of data point x , and D is a labeled dataset drawn from the same distribution as x , and \Phi is the set of data generating functions. The set of data generating functions is a prior which defines the hypothesis space of the classifier. A common selection is the set of structural causal models (SCMs), where the causal relationships between features are represented by the edges of a directed acyclic graph [ 3 ] . This trains the model to estimate the PPD directly by implicitly predicting the SCM which best fits the data and using that model to predict \hat{y} . II-B In-Context Learning (ICL) During ICL, the pretrained model is given a set of n labeled context points, \{(x_{c},y_{c})\}_{i=1}^{n} , and a query, x_{q} , and is asked to predict \hat{y}_{q} , the label of the query point. Critically, this does not involve updating model weights and is thus distinct from standard in-weight learning, where labeled data is used to update model weights before passing in the query to predict a label. The idea of ICL originated with large languages models (LLMs), when their ability to perform task-agnostic few-shot classification [ 1 ] . However, since then, the idea of ICL has expanded, and models have been trained to perform ICL explicitly in a variety of settings, including images [ 8 ] and tabular data [ 3 ] . The learning of an ICL model takes place entirely within the latent representation of the context and query data rather than the weights of the model, which can make manipulating the model difficult. Without performing a computationally expensive fine-tuning operation on the model, the only ways to affect its performance are to manipulate the input data or the downstream prediction. II-C Receiver Operating Characteristic (ROC) For binary classification with classes ‘0’ and ‘1’ (minority class), the ROC curve defines the

Read the complete text