An Old Look at Empirical Bayes

Nicholas G. Polson, Vadim O. Sokolov, Daniel Zantedeschi · 2026

Read An Old Look at Empirical Bayes by Nicholas G. Polson, Vadim O. Sokolov, Daniel Zantedeschi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21535.txt.

Introduction

Nicholas G. Polson, Vadim O. Sokolov, and Daniel Zantedeschi offer an old look at empirical Bayes. Lindley once quipped that only one thing is worse than a frequentist: an empirical Bayesian. The quip is technically serious. The method uses the same data twice and mixes levels. It produces belief-like summaries whose uncertainty measures differ from a full hierarchy model. David Blei's 2026 IMS Medallion Lecture brings back the program under three labels: mirror patterns, hidden odds via simulation methods, and calibration studies combining trial and observed data.

Catalog record

CREATORNicholas G. Polson, Vadim O. Sokolov, Daniel Zantedeschi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70562
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Old Look at Empirical Bayes Nicholas G. Polson, Vadim O. Sokolov, Daniel Zantedeschi Abstract Dennis Lindley once said that there is only one thing worse than a frequentist, and that is an empirical Bayesian. The quip has the air of caricature, but its technical content is serious: empirical Bayes uses the same data twice, conflates levels of a hierarchy, and produces posterior-shaped summaries whose uncertainty quantification differs from what a fully hierarchical model delivers. David Blei’s 2026 IMS Medallion Lecture, “A Fresh Look at Empirical Bayes,” revives the program under three new banners: empirical Bayes via probabilistic symmetries (rebranded “Bayesian empirical Bayes”), empirical Bayes with implicit likelihoods through simulation-based inference, and empirical Bayes for combining experimental and observational data through calibration studies. This is a continuation of Blei and Kucukelbir’s earlier “population empirical Bayes” (PopEB, 2015). We argue, in the spirit of Lindley, I. J. Good, William DuMouchel, Thomas Louis, and our own recent work with Datta, that Blei’s machinery targets inferential objects distinct from the posterior conditional on the realized data, and that the cost of maintaining the full hierarchical discipline has fallen low enough that the computational trade-off no longer favors the shortcut. The case study is the Tweedie formula. Efron’s f-modeling empirical Bayes plugs an estimated score function into a posterior-mean identity, but a smoothed score need not arise from any prior. The horseshoe Tweedie formula does. We conclude by recommending that the impressive computational machinery of modern empirical Bayes (variational inference, neural amortization, simulation-based inference) be redeployed in service of properly hierarchical Bayes. Keywords: empirical Bayes, hierarchical Bayes, Tweedie formula, horseshoe prior, coherence, likelihood principle, population posterior, calibration. “There is only one thing worse than a frequentist, and that is an empirical Bayesian.” Dennis V. Lindley 1. Introduction Dennis Lindley is reported to have said, more than once, that there is only one thing worse than a frequentist, and that is an empirical Bayesian. The remark is funny because it surprises: surely an empirical Bayesian, who at least tries to write down a prior, is closer to the truth than a strict frequentist. The remark is sharp because it is technically loaded. The strict frequentist commits the well-known sins of conditioning on the wrong thing and entertaining ancillary statistics ad hoc. The empirical Bayesian commits a different and, in Lindley’s view, deeper sin: he writes down a prior, looks at the data, picks the prior on the basis of the data, and then computes a posterior as if the prior had been chosen first. The same data is used twice; the hierarchy is collapsed; the posterior is no longer a coherent summary of belief under any joint probability assignment on parameters and data. The empirical Bayesian has all the formal apparatus of Bayes and none of its discipline. This paper revisits Lindley’s critique in light of David Blei’s 2026 IMS Medallion Lecture, titled “A Fresh Look at Empirical Bayes,” announced for delivery at JSM 2026 in Boston and not yet published in final form. Our critique is based on the announced description of the lecture and on closely related published work by Blei and collaborators (Kucukelbir and Blei 2015; McInerney, Ranganath, and Blei 2015). We recognize that the final lecture content may differ from the announced themes; where we impute specific technical content to the lecture, we note the basis for the imputation. The lecture, building on a decade of work by Blei and collaborators on population empirical Bayes, the population posterior, and probabilistic programming, sketches three new directions. First, empirical Bayes is extended beyond the i.i.d. compound decision problem of Robbins by way of probabilistic symmetries (exchangeability of arrays, shift invariance of spatial processes, conditional symmetry under covariates) with ergodic decompositions standing in for the unknown prior. Blei labels this “Bayesian empirical Bayes” (BEB; Wu, Weinstein, and Blei 2025). Second, the requirement of a tractable likelihood is relaxed: an EB method is constructed for the simulator-based or implicit-likelihood case, using neural amortization and the “population posterior” as an approximation target. Third, EB thinking is applied to the problem of combining experimental and observational data, with calibration studies (in which the causal effect is known a priori to be zero) used to identify the distribution of observational bias. There is much to admire in this program. The computational machinery is impressive; the choice of problems is timely; the language of probabilistic symmetries is appropriately general; the recognition that EB needs to be extended beyond the textbook normal–normal setting is overdue. Our purpose is not to deny any of this. We ask whether the renamings and computational extensions resolve, or merely paper over, the objections raised by Lindley, Good, DuMouchel, Efron, and others over six decades. We argue four things. First, the name “Bayesian empirical Bayes,” chosen by Blei in continuity with Deely and Lindley (1981), names two structurally different objects. The Deely–Lindley construction places a proper hyperprior on a finite-dimensional hyperparameter; Blei’s BEB fits the directing measure (the invariant parameter g ) by maximum marginal likelihood, using variational inference and neural networks as computational devices rather than placing a posterior over g . The two are not equivalent, and the MML estimate of g inherits data-dependences we associate with empirical Bayes. Second, the “population posterior” introduced by McInerney, Ranganath, and Blei (2015) is, on inspection, a frequentist object: the expectation of the standard posterior under replicate sampling from the population. It has a clean predictive interpretation, with connections to PAC-Bayesian generalization theory. Our quarrel is not with the predictive object but with its marketing as a Bayesian posterior; we propose that it be renamed the population predictive distribution . Third, the simulation-based empirical Bayes machinery sits alongside a large modern literature on simulation-based inference (neural posterior estimation, neural likelihood estimation, sequential neural ratio estimation, normalizing flows) and shares with that literature the difficulty that the trained network’s relationship to a genuine Bayes posterior is not transparent. Our generative Bayesian computation (GBC) framework targets the standard posterior by quantile regression under a proper scoring rule, an alternative we will compare carefully rather than promote unconditionally. Fourth, the workhorse of large-scale empirical Bayes, Efron’s Tweedie shrinkage, requires a sharper analysis than we originally provided. Efron’s f-modeling estimator fits the marginal score function directly by smoothing; the resulting \hat{m}(x) need not arise from any proper prior, and the resulting “posterior mean” is not \mathbb{E}[\theta\mid x] under any model. G-modeling and NPMLE, by contrast, estimate a valid mixing distribution and produce genuine Bayes rules, though for a prior with restricted support. The shape-constrained nonparametric MLE (Koenker and Mizera 2014; Saha and Guntuboyina 2020; Soloff, Guntuboyina, and Sen 2024) is by construction a convolution with a discrete prior, and the resulting posterior mean is a genuine Bayes rule, specifically for that discrete prior. The horseshoe Tweedie formula corrects both: the marginal is a half-Cauchy scale mixture of normals, the shrinkage profile is Bayes for a heavy-tailed continuous prior, and the estimator is minimax optimal in the sparse normal means problem. The thread running through these arguments is that empirical Bayes and full hierarchical Bayes target different inferential objects, and the differences matter for uncertainty quantification, coverage, and optimality. Where EB targets the posterior on the realized data, the guarantees are

Read the complete text