Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy · 2026
Read Universal priors: solving empirical Bayes via Bayesian inference and pretraining by Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15136.txt.
Description
Cannella and colleagues justify a finding: a transformer trained on synthetic data performs strongly on empirical Bayes problems. Rather than dissecting the model, they ask why a Bayes estimator, trained under one preset distribution, can adapt to arbitrary test cases. Focusing on Poisson Bayes problems, they identify universal priors under which training yields near-optimal regret across all test cases.
Their analysis leans on posterior contraction, a classical Bayesian effect, showing the trained model adapts to unknown test cases precisely through that effect. The same lens explains length generalization, where test runs exceed training runs, since the model performs Bayesian inference with a general posterior. The paper thus grounds a practical pretraining success in classical Bayesian theory.
Catalog record
| CREATOR | Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15136