Universal priors: solving empirical Bayes via Bayesian inference and pretraining

Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy · 2026

Read Universal priors: solving empirical Bayes via Bayesian inference and pretraining by Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15136.txt.

Description

Cannella and colleagues justify a finding: a transformer trained on synthetic data performs strongly on empirical Bayes problems. Rather than dissecting the model, they ask why a Bayes estimator, trained under one preset distribution, can adapt to arbitrary test cases. Focusing on Poisson Bayes problems, they identify universal priors under which training yields near-optimal regret across all test cases.

Their analysis leans on posterior contraction, a classical Bayesian effect, showing the trained model adapts to unknown test cases precisely through that effect. The same lens explains length generalization, where test runs exceed training runs, since the model performs Bayesian inference with a general posterior. The paper thus grounds a practical pretraining success in classical Bayesian theory.

Catalog record

CREATORNick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15136

Browse the full library · About the Tower of Babel