Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models

Julius Vetter Manuel Gloeckler Daniel Gedon Jakob H. Macke[1pt] · 2025

Read Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models by Julius Vetter Manuel Gloeckler Daniel Gedon Jakob H. Macke[1pt] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17660.txt.

Introduction

Julius Vetter and three co-authors offer a shortcut for Bayesian inference from computer simulations, in a 2025 methods study. Many fields infer hidden values by running a costly model many times and training a network on the fake data. The cost soars when each run is slow. The team instead reuses a ready tabular foundation model, TabPFN, which was pre-trained on vast sets of tables.

Their method, called NPE-PFN, needs no fresh training, model choice, or tuning. On standard tasks and two hard science problems, it matched rival methods in accuracy. It often needed far fewer runs, at times orders of magnitude fewer. It also held up better when the model behind the data was a bit off. The authors present it as an easy, general tool for inverse problems.

Catalog record

CREATORJulius Vetter Manuel Gloeckler Daniel Gedon Jakob H. Macke[1pt]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17660

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models Julius Vetter Manuel Gloeckler Daniel Gedon Jakob H. Macke[1pt] Abstract Simulation-based inference (SBI) offers a flexible and general approach to performing Bayesian inference: In SBI, a neural network is trained on synthetic data simulated from a model and used to rapidly infer posterior distributions for observed data. A key goal for SBI is to achieve accurate inference with as few simulations as possible, especially for expensive simulators. In this work, we address this challenge by repurposing recent probabilistic foundation models for tabular data: We show how tabular foundation models—specifically TabPFN—can be used as pre-trained autoregressive conditional density estimators for SBI. We propose Neural Posterior Estimation with Prior-data Fitted Networks (NPE-PF) and show that it is competitive with current SBI approaches in terms of accuracy for both benchmark tasks and two complex scientific inverse problems. Crucially, it often substantially outperforms them in terms of simulation efficiency, sometimes requiring orders of magnitude fewer simulations. NPE-PF eliminates the need for inference network selection, training, and hyperparameter tuning. We also show that it exhibits superior robustness to model misspecification and can be scaled to simulation budgets that exceed the context size limit of TabPFN. NPE-PF provides a new direction for SBI, where training-free, general-purpose inference models offer efficient, easy-to-use, and flexible solutions for a wide range of stochastic inverse problems. † † * Equal contribution. † † \dagger Joint supervision. † † {firstname.lastname}@uni-tuebingen.de † † Code available at https://github.com/mackelab/npe-pf . 1 Introduction Simulation has long been a cornerstone of scientific inquiry [ 1 , 2 ] and is becoming increasingly relevant as researchers tackle ever more complex scientific questions and systems [ 3 ] . Simulators often depend on parameters that are challenging or impossible to measure experimentally. Bayesian inference provides a general framework for identifying such parameters by estimating posterior distributions over parameters. However, classical methods such as Markov chain Monte Carlo (MCMC) require evaluations of the associated model likelihoods, which can be computationally demanding or prohibitive for complex numerical simulators. The field of simulation-based inference (SBI), or likelihood-free inference, aims to address this challenge by enabling Bayesian inference without requiring access to likelihoods. The basic idea of SBI is to train a neural network on synthetic data simulated from the model such that it can approximate the posterior distributions. In Neural Posterior Estimation (NPE, [ 4 , 5 ] ), a conditional density neural network—often, a normalizing flow [ 6 , 7 ] or a diffusion model [ 8 , 9 , 10 ] —learns the conditional distribution of parameters given simulated data p({\bm{\theta}}\mid{\bm{x}}) . When evaluated on empirical observations {\bm{x}}_{o} , the network directly returns the posterior distribution p({\bm{\theta}}\mid{\bm{x}}_{o}) . Similarly, other SBI methods use neural networks to represent likelihoods [ 11 , 12 , 13 ] , likelihood ratios [ 14 , 15 , 16 , 17 ] , or target several properties at once [ 18 , 19 , 20 ] . Compared to classical Approximate Bayesian Computation (ABC, [ 21 , 22 ] ) or synthetic likelihoods methods [ 23 ] , neural-network-based SBI methods scale better to complex, high-dimensional simulators [ 24 ] . In addition, SBI methods often amortize the computational burden: Once trained, the network provides fast inference across multiple observations. Despite these advances, a key challenge for SBI methods remains: They typically require generating a large number of simulations as synthetic training data, rendering them impractical for expensive simulators. Recent work aims to increase simulation efficiency, e.g., by exploiting additional information or properties of the simulator [ 25 , 26 , 27 , 7 , 28 ] , utilizing low-fidelity simulations [ 29 , 30 ] , or using Bayesian neural networks [ 31 , 32 , 33 ] . So-called sequential approaches forego the amortization properties of the inference network and target simulations to particular observations [ 4 , 5 , 6 , 34 ] , but even then, simulation efficiency remains a challenge. In addition, users must select an appropriate SBI method and select and train an inference network with the associated hyperparameters—posing a barrier for inexperienced users. Figure 1 : Illustration of NPE and NPE-PF: Both approaches use simulations sampled from the prior and simulator. In (standard) NPE, a neural density estimator is trained to obtain the posterior. In NPE-PF, the posterior is evaluated by autoregressively passing the simulation dataset and observations to TabPFN. These limitations raise a key question: Can we eliminate the high cost associated with training and tuning, and alleviate the need to have a large number of simulations by leveraging recent advances in foundation models? Here, we answer this question by repurposing a tabular foundation model—specifically TabPFN v2 [ 35 ] —as an inference engine for SBI. TabPFN is trained to perform in-context learning on tabular data, returning a prediction when being prompted with a training dataset and a test point. For both regression and classification, TabPFN was shown to perform exceptionally well, in particular in low data regimes. Crucially, TabPFN can also serve as a density estimator by applying it in an autoregressive manner, though prior work has only explored this in limited settings [ 35 ] . Here, we show that using TabPFN autoregressively allows for the estimation of complex, high-dimensional conditional densities, making it suitable for SBI. Based on this observation, we introduce Neural Posterior Estimation with Prior-data Fitted Networks (NPE-PF), a variant of NPE that uses TabPFN as a pre-trained conditional density estimator, and therefore enables inference without any additional neural network training (Fig. 1 ). We show that, on benchmark tasks, NPE-PF performs competitively with other SBI approaches and often outperforms them substantially, especially for small simulation budgets. Furthermore, we leverage data-filtering schemes that enable NPE-PF to make use of large simulation budgets exceeding the context size of TabPFN and extend the approach to sequential inference settings. Finally, we show that NPE-PF remains empirically robust under model misspecification and achieves strong results on two challenging scientific applications, requiring orders of magnitude fewer simulations than standard SBI approaches. 2 Neural Posterior Estimation with Prior-data Fitted Networks 2.1 Background We consider a simulator with parameters {\bm{\theta}}\in\mathbb{R}^{d_{\theta}} that stochastically generates samples {\bm{x}}\in\mathbb{R}^{d_{x}} , thus implicitly defining a likelihood p({\bm{x}}\mid{\bm{\theta}}) . Given a prior distribution p({\bm{\theta}}) and observation {\bm{x}}_{o} , our aim is to infer the associated posterior distribution p({\bm{\theta}}\mid{\bm{x}}_{o})\propto p({\bm{x}}_{o}\mid{\bm{\theta}})p({\bm{\theta}}) . Simulation-based inference. We focus on Neural Posterior Estimation (NPE), a common approach to SBI: NPE approximates the target posterior distribution directly by training an inference network q_{\phi}({\bm{\theta}}\mid{\bm{x}}) on a dataset of parameter–data pairs \mathcal{D}=\{({\bm{\theta}}_{i},{\bm{x}}_{i})\}_{i=1}^{N} sampled from a prior p({\bm{\theta}}) and a simulator with implicit likelihood p({\bm{x}}\mid{\bm{\theta}}) . The inference network is trained to maximize the log-likelihood and learns to approximate the corresponding posterior distribution [ 4 , 36 ] . Flexible conditional neural density estimators, such as normalizing flows [ 37 , 38 , 39 ] , are typically used as inference networks. At inference time, NPE provides an amortized posterior approximation, allowing for rapid evaluations. Tabular foundation models. Recent foundation models for tabular data have shown strong performance on regression and classification tasks via in-context learning. Within this work, we focus on TabPFN v2 [ 40 , 35 ] , a state-of-the-art transformer-based foundation model

Read the complete text