Transformers Can Learn Posterior Predictive Distributions In-Context
Gyeonghun Kang, Changwoo J. Lee, Xiang Cheng · 2026
Read Transformers Can Learn Posterior Predictive Distributions In-Context by Gyeonghun Kang, Changwoo J. Lee, Xiang Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26713.txt.
Introduction
Kang, Lee, and Cheng prove, with an explicit step-by-step build, that transformers can carry out gradient descent aimed at the posterior predictive mean and variance for Gaussian process regression. The result is then turned into binned probabilities. They bound the error by attention depth and bin size, and show that normalization and depth control how far transformers reach past their training size range. Simulations back the theory and shed light on how prior-data fitted networks generalize.
Catalog record
| CREATOR | Gyeonghun Kang, Changwoo J. Lee, Xiang Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105759 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26713
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformers Can Learn Posterior Predictive Distributions In-Context Gyeonghun Kang, Changwoo J. Lee, Xiang Cheng Abstract Prior-data fitted networks (PFNs) have recently emerged as a powerful approach for Bayesian prediction tasks, approximating the posterior predictive distribution (PPD) through in-context learning. Despite their strong empirical performance and ability to go beyond point predictions, theoretical understandings of the algorithmic capability of transformers to learn distributions in context are still lacking. Focusing on Gaussian process regression problems, we show by construction that transformers can implement a gradient descent algorithm targeting the posterior predictive mean and variance, followed by nonlinear mappings that yield binned probabilities of PPD. We study the error bounds of the approximated PPD in terms of attention depth and bin resolution. Based on these results, we further demonstrate the key role of normalization and the choice of attention depth in enabling the extrapolation abilities of transformers beyond the pretraining sample size range. We conduct simulations that corroborate our findings, providing insight into the expressivity of PFNs targeting PPDs and how architectural choices may influence generalization capabilities. † † affiliationnotice: \dagger Equal advising. 1 Introduction Figure 1 : Comparison of PPDs produced by a transformer-based PFN and an empirical-Bayes Gaussian process (GP) on the Sacramento home price dataset ( 25 ) . The axes represent spatial coordinates (longitude and latitude), and the color scale represents price per square foot (V). The top row shows PFN posterior predictive summaries, and the bottom row shows the corresponding GP outputs. Columns correspond to the PPD mean (left), 5\% quantile (center), and 95\% quantile (right). Housing locations used as context are shown by blue \times marks in the left column. The PFN PPD tracks that of exact GP closely overall except in the regions with no nearby observations. See Section F for details. In-context learning (ICL) is the ability of a pre-trained model to adapt to new tasks at inference time by using examples provided in the input sequence without any parameter updates ( 5 ; 15 ) . This ability has been most prominently observed in transformer architectures, which have become the dominant modeling component in modern machine learning ( 41 ) . Formally, we refer to ICL as the ability to learn a mapping from a set of input-output pairs \mathcal{D}_{n}=\{(x_{i},y_{i})\}_{i=1}^{n} together with a new input x_{n+1} to a prediction of y_{n+1} ; see 11 and references therein for a review. When the goal is probabilistic prediction of y_{n+1} , rather than point prediction, prior-data fitted networks (PFNs) ( 30 ) have recently gained considerable attention as an ICL method for approximating the posterior predictive distribution (PPD) of y_{n+1} . By pre-training a transformer model on synthetic data from a joint prior predictive model of (x,y) , PFNs aim to learn a mapping from (\mathcal{D}_{n},x_{n+1}) to the posterior predictive distribution p(\cdot|x_{n+1},\mathcal{D}_{n}) . Compared to traditional approaches for computing PPDs such as Markov chain Monte Carlo (MCMC), PFNs offer substantial computational gains since model parameters are no longer updated after pre-training and also maintain robustness through pre-training on diverse data-generating scenarios. Most notably, PFNs for tabular data have demonstrated strong empirical performance and excellent scalability across a wide range of benchmarks ( 19 ; 20 ) . PFNs have also been adopted in a broad range of application domains, such as RNA biophysics ( 39 ) and genomic prediction ( 40 ) . We are particularly motivated by PFN’s ability to quantify posterior predictive uncertainty. For instance, under Gaussian process regression settings, 30 reports that 95% credible intervals of y_{n+1} produced by PFNs are virtually indistinguishable from the truth; see also 46 , where PFN-based PPD intervals closely match nominal coverage under interpolation settings. Also, PFNs have been shown to successfully serve as flexible surrogate models for Bayesian optimization ( 29 ; 45 ) , accurately computing common acquisition functions such as expected improvement, which depend on both posterior predictive means and variances. See Figure 1 for an illustration of PFN’s ability to accurately capture the lower 5% and upper 95% PPD quantiles; details are in Section F . Despite its increasing popularity, the theoretical understanding of PFNs remains largely unexplored, especially regarding the role of the transformer in the pre-training stage. An increasing body of work suggests that ICL can be interpreted as implicitly performing gradient-based optimization. For a linear regression task, 2 ; 43 ; 3 demonstrates that transformers and their recurrences are expressive enough to implement gradient descent updates in their forward pass; see also 14 ; 42 . Furthermore, 1 ; 47 ; 27 shows that the global optimum of a single layer transformer weights in the linear regression task implements a single iteration of preconditioned gradient descent. For PFNs, a natural research question concerns understanding how PFNs carry out in-context learning of PPD, and the mechanisms and conditions under which this capability arises. Taking a gradient-based optimization view of ICL, we are particularly interested in a principled understanding of how attention depth affects the approximation quality of PPDs. Recent state-of-the-art PFN models suggest that increasing network depth can substantially improve performance. For example, TabPFN-2.5 and TabPFN-3 ( 17 ; 18 ) report markedly improved accuracy by increasing attention depth to 18 or 24 layers, a 1.5- or 2-fold increase over TabPFNv2 ( 20 ) with 12 layers. Another important aspect of PFN that affects the approximation quality of PPDs is the discretization resolution used for PPDs over continuous domains, a practice that is widespread in existing PFN implementations ( 30 ; 20 ) but is seldom studied in the literature. Another interesting aspect of PFNs is their ability to generalize beyond the pre-training sample size. For instance, even though TabPFNv2 is pretrained on synthetic data only up to a sample size of 2048, it remains highly competitive for context size n up to 10,000 ( 20 ) . To reach even larger, recent works have adopted approaches such as fine-tuning ( 12 ) , boosting ( 44 ) , and two-stage architectures ( 34 ) . Adopting a gradient-based optimization view of ICL, we seek to provide a better understanding of when this generalization ability arises and how it can be further strengthened. 32 attempts to explain this capability asymptotically, based on sensitivity to individual training samples and localization for a fixed architecture; our focus is instead on the relationship between attention depth, normalization, and generalization ability. Contributions. Focusing on GP regression problems, we establish theoretical support for PFNs’ capability to approximate the PPD and identify the mechanisms driving the generalization across different context size n , referred to as n -generalization , whereby models retain reliable point predictions and uncertainty quantification beyond the pretraining sample size range. 1. We provide an explicit transformer construction that approximates the PPD in-context up to a prescribed tolerance: self-attention implements an iterative solver for the predictive mean and variance, and a shallow multilayer perceptron (MLP) maps these moments to a binned distribution. With binning and tail errors controlled, the principal remaining error arises in the attention block (Theorem 4.1 ), which inherits an exponential convergence rate in the number of layers L (Theorem 3.1 ). This extends earlier ICL works focused on point estimation to distributional approximation assessment. 2. We use our construction to