Robust Tabular Foundation Models
Matthew Peroni, Franck Le, Vadim Sheinin · 2025
Read Robust Tabular Foundation Models by Matthew Peroni, Franck Le, Vadim Sheinin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03307.txt.
Introduction
Models for table data are getting a foundation-model upgrade. Matthew Peroni, Franck Le, and Vadim Sheinin study robust tabular foundation models in this 2025 work. These models aim to beat classic methods on structured data like spreadsheets and records.
A key finding is that such models can be pretrained entirely on synthetic data. That opens room to design data generators that build in good traits, like robustness. Strong table models matter for business, health, and science, where most data sit in rows and columns, not text or images. Synthetic pretraining also lets builders stress-test models on rare table patterns safely.
Catalog record
| CREATOR | Matthew Peroni, Franck Le, Vadim Sheinin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Tabular Foundation Models Matthew Peroni, Franck Le, Vadim Sheinin Abstract The development of tabular foundation models (TFMs) has accelerated in recent years, showing strong potential to outperform traditional ML methods for structured data. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our insight is that parameterizing the generator distribution enables an adversarial robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model. We formalize this by introducing an optimality gap measure, given by the difference between TFM performance and the best achievable performance as estimated by strong baselines such as XGBoost, CatBoost, and Random Forests. Building on this idea, we propose Robust Tabular Foundation Models ( RTFM ), a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RTFM improves benchmark performance, with up to a 6% increase in mean normalized AUC over the original TabPFN and other baseline algorithms, while requiring less than 100k additional synthetic datasets. These results highlight a promising new direction for targeted adversarial training and fine-tuning of TFMs using synthetic data alone. Introduction Recent survey studies have challenged the deep learning community to improve upon boosted tree methods for learning from structured data, having found a significant gap in performance remained between the paradigms ( Grinsztajn, Oyallon, and Varoquaux 2022 ) . In response, substantial progress has been made to close this gap, resulting in a variety of strong deep learning approaches being proposed over the past few years ( Somepalli et al. 2021 ; Gorishniy et al. 2023 ; Hegselmann et al. 2023 ; Hollmann et al. 2023 ; Carballo et al. 2023 ; Yan et al. 2024 ; Zhang and Robinson 2025 ) . Among these new approaches, tabular foundation models (TFMs) which rely on in-context learning (ICL) ( Hollmann et al. 2023 ; Qu et al. 2025 ; Zhang and Robinson 2025 ) have emerged as a promising direction for classification and regression tasks with structured datasets. The benefits of this approach are two-fold. In many cases, the performance of the TFMs improves upon strong traditional baselines such as boosted trees ( Friedman 2001 ; Erickson et al. 2025 ) . Additionally, this performance is achieved in a zero-shot framework, enabling high-quality predictions on new datasets in milliseconds when GPU-accelerated. Further, as we will explore in this work, since these TFMs are pretrained using synthetic data, there is significant opportunity for improvement as the data generation process is expanded and improved. Training TFMs relies on generating a large amount of diverse synthetic datasets ( Hollmann et al. 2023 ) . As we will formally define in the following section, this generation process relies on constructing structural causal models (SCMs) from which datasets can be sampled ( Pearl 2009 ; Shanmugam 2018 ) . The structure of these SCMs is implicitly parameterized, giving significant control over the data generation process. All current publicly available, competitive TFMs ( Hollmann et al. 2023 ; Qu et al. 2025 ; Zhang and Robinson 2025 ) have been pretrained on datasets generated from a fixed prior distribution over these SCM parameters. However, fixed priors underrepresent certain regions of the parameter space, potentially degrading performance on real-world datasets with similar structure. This contributes to state-of-the-art TFMs still lagging behind tree-based methods on some benchmarks ( McElfresh et al. 2024 ; Ye, Liu, and Chao 2025 ) . In this work, we leverage the significant control provided by the data generation process to frame TFM training from an adversarial robustness perspective ( Madry et al. 2019 ) . Recent work by ( Wu and Bergman 2025 ) also explores a narrower version of this idea, focusing on adjusting the weights of a specific class of SCMs in GAN-style training of TFMs. In this work, we focus on a broad and highly flexible framework for adversarially robust training of TFMs. Our contributions can be summarized as: 1) We formalize adversarial training over the SCM parameter space, allowing the model to adapt to challenging regions (e.g., varying numbers of features, categorical feature ratios, nonlinearities). We introduce an optimality gap concept and use it to target regions where the TFM underperforms relative to the best achievable performance. 2) We propose an efficient, model-agnostic two-stage adversarial training algorithm for TFMs, which we call Robust Tabular Foundation Models ( RTFM ). 3) We apply RTFM to TabPFN V2 ( Hollmann et al. 2025 ) , showing that with only 90k additional training datasets, we can significantly improve the ranking of TabPFN on several real-world tabular benchmarks. Problem Definition Figure 1: Overview of Robust Tabular Foundation Models ( RTFM ). We begin by defining some notation, and then introduce our theoretical formulation for RTFM . Let \Phi represent the space of hypothesis functions which define the relationship between features x and outcome y . The outcome y can be discrete or continuous, but for this work, we focus on classification tasks. Then a hypothesis \phi\in\Phi is a specific function which we can treat as a data generating mechanism. In the case of SCMs, \Phi is the entire space of SCMs, and \phi\in\Phi is a specific instance of an SCM, from which a dataset D\sim p(\phi) can be generated. In this work, we implement SCMs as randomly initialized multi-layer perceptrons (MLPs), as introduced in ( Hollmann et al. 2023 ) . Full details of this process are provided in Appendix B . For example, we can sample the number of layers l\sim p(l) , the hidden size h\sim p(h) , the activation function a\sim p(a) , and the ratio of categorical features r_{cat}\sim p(r_{cat}) , among other dimensions. An SCM is generated by first sampling from the joint distribution over these hyperparameters, each of which is parameterized. For example, we might sample the ratio of categorical features r_{cat}\sim TruncNorm(\mu_{r_{cat}},a=0,b=1) from the truncated normal distribution with some mean \mu_{r_{cat}} which parametrizes the distribution. Then, assuming we have similar parameterizations for the distribution over each hyperparameter, we define \theta=(\mu_{l},\mu_{h},\mu_{r_{cat}},\dots) as the parameterization of the joint distribution over all the hyperparameters. We define \mathcal{P} as the space of the parameters, such that \theta\in\mathcal{P} . The generators are then sampled from \phi\sim p(\Phi;\theta) . We now define the original objective for the Prior-Fitted Network (PFN) learning task and provide our adversarially robust formulation. Let g_{\mathbf{W}} be our predictive model parameterized by weight matrix \mathbf{W} . As a PFN, g_{\mathbf{W}} takes as input a sequence of labeled training samples and unlabeled test samples, and predicts labels for the test samples. For a given data generator \phi\in\Phi , the PFN loss is defined as the cross entropy loss over the test samples, \displaystyle\mathcal{L}_{PFN}(\mathbf{W};\phi)= (1) \displaystyle\mathbb{E}_{(\{(x_{t},y_{t})\}\cup D)\sim p(\phi)}[-\log(g_{\mathbf{W}}(y_{t}|x_{t},D)]. Since the model will be trained by sampling many data generators from the distribution \phi\sim p(\Phi;\theta) , we can write the original optimization problem as follows, \min_{\mathbf{W}}\mathbb{E}_{\phi\sim p(\Phi;\theta)}[\mathcal{L}_{PFN}(\mathbf{W};\phi)]. (2) Since our goal is to learn a foundation model which performs well across all data generation schemes, we want to introduce an adversary who can shift the distribution over data generators to identify