Solving Empirical Bayes via Transformers
Anzo Teh, Mark Jabbour, Yury Polyanskiy · 2025
Read Solving Empirical Bayes via Transformers by Anzo Teh, Mark Jabbour, Yury Polyanskiy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09844.txt.
Description
Anzo Teh, Mark Jabbour, and Yury Polyanskiy hand one of statistics' oldest problems to a transformer in this 2025 paper. Empirical Bayes asks for many hidden averages at once when only noisy counts are seen; the classic oracle knows the hidden distribution, and everyone else approximates it.
Their transformer is pretrained on synthetic pairs of counts and true means, so at test time it adapts to an unseen prior from context alone. Theory says a wide enough transformer can match the oracle as the problem grows. In practice even a tiny model of about a hundred thousand weights beat the best classical algorithm on speed and accuracy, on fresh synthetic data and on hockey, baseball, and text corpora. Probing inside showed the network's method resembles neither classic estimator, suggesting the machine found its own route.
Catalog record
| CREATOR | Anzo Teh, Mark Jabbour, Yury Polyanskiy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09844