From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian Supported in part by NSF Grant DMS-2113364.

Ya’acov Ritov · 2025

Read From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian Supported in part by NSF Grant DMS-2113364. by Ya’acov Ritov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01642.txt.

Introduction

Ya'acov Ritov asks whether a statistician can truly be a subjective Bayesian in the age of big data. His 2025 paper argues that the classic Bayesian picture, a prior that holds the statistician's beliefs before data arrive, fails in ultra-high-dimensional models.

Natural priors that express honest prior beliefs, he claims, fail in unpredictable ways at parameter values that cannot be ignored. Bayesian procedures that do work in such settings were built with frequentist goals in mind. They depend on the loss function and the target of the analysis, so different priors would be needed for different analyses of the same data. Those, he argues, are frequentist procedures in Bayesian form.

Catalog record

CREATORYa’acov Ritov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian Supported in part by NSF Grant DMS-2113364. Ya’acov Ritov Abstract We argue that the Bayesian paradigm, of a prior which represents the beliefs of the statistician before observing the data, is not feasible in ultra-high dimensional models. We claim that natural priors that represent the a priori beliefs fail in unpredictable ways under values of the parameters that cannot be honestly ignored. We do not claim that the frequentist estimators we present cannot be mimicked by Bayesian procedures, but that these Bayesian procedures do not represent beliefs. They were created with the frequentist analysis in mind, and in most cases, they cannot represent a consistent set of beliefs about the parameters (for example, since they depend on the loss function, the particular functional of interest, and not only on the a priori knowledge, different priors should be used for different analyses of the same data set). In a way, these are frequentist procedures using Bayesian technique. The paper presents different examples where the subjective point of view fails. It is argued that the arguments based on Wald’s and Savage’s seminal works are not relevant to the validity of the subjective Bayesian paradigm. The discussion tries to deal with the fundamentals, but the argument is based on a firm mathematical proofs. 1 Introduction and summary It was used to be that the Bayesian analysis was restricted to simple models. Computational considerations were an issue, and practical Bayesian analysis was limited to the simple case of exponential families with conjugate models. Thus Savage wrote: ‘No attempt is made to include criteria like intellectual simplicity or facility of computation that depend not only on the estimate but also on the capabilities of the people who contemplate using it.” ( Savage 1972 , page 221) He continues: “no one really considers it a virtue in itself for an estimate to be a maximum-likelihood estimate …; rather, it is believed that such estimates do typically have real virtues.” In this note we try to apply these criteria to the Bayesian analysis itself. Thus, we try to consider the Bayesian approach not as a methodology to yield simple solutions, almost off-the-shelf, for complex problems. Solutions that can be approximated in practice. It is lowering the net, since other solutions can be approximated as well, and it does not make sense to save on the final computer time for the analysis of a data which cost millions to collect. We, of course, acknowledge that Bayesian solutions may be good. One may consider as a Bayesian estimator any procedure which is the minimizer of the a posteriori mean loss for some artificial a priori distribution, or even to the maximizer of the a posteriori density. We will, however, restrict the adjective ‘Bayesian’ to those who follow a Bayesian philosophy. Berger 2000 describes the subjective Bayesian analysis as the “soul” of Bayesian statistics, which probably means that it is not the mind, and certainly not the reality. We admit that. However, it is hard to justify “philosphically” a technique. There is no ideology in M-estimation or generalized methods of moments. They do not becomes an adjective like “Bayesian.” Nobody refers to himself as an M-estimatian. Frequentist techniques can be judged on how close to the best (in any sense) solution to the problem they are. They are not the goal by itself. The division of the community is to Bayesians vs. non-Bayesians. There is no parallel division to method of moments people to non-method of moments guys. Some may argue that, sometimes, the conclusions are not sensitive to the subjective probability, and therefore we can bypass the step of having a real subjective prior, and the philosophy stands with using practical priors. This would not be the case in the examples we are going to present. The results would heavily depend on the prior, and it would not be possible to have a good, approxiately subjective prior. Lindley 2001 outlined the Bayesian philosophy. For him uncertainty is always formalized as a probability distribution. A statistical problem has two uncertain components. Almost all statistical schools agree on the uncertainty, or randomness, of the observations given the parameters. However, statistical analysis is needed exactly since there is uncertainty about the values of the model parameters. The researcher start with his own world view or, at least, a small world view, Savage 1972 . As any other uncertainty, this is also described by the so call a priori distribution. The proper statistical analysis combines these two using the inverse probability formula, AKA, Bayes Theorem. Bernardo 2011 claims that this approach succeeds in what any other approach to statistics fails, namely to give a solid axiomatic foundations that guarantees a methodological consistency, no less than “a scientific revolution in Kuhn’s sense.” Except for the philosophical assertions there are the mathematical arguments. They are based on two results, attributed to two giants, Wald and Savage. As Barnard wrote in his comment on Lindley 1953 : “What Wald did from the practical point of view was to show that even if we disbelieve in the existence of a prior distribution, we should none the less behave as if it did exist.” Wald argued that any admissible statistical procedure is Bayesian (or, at least, a limit of Bayesian procedures). Savage argued that any utility maximizer behaves as if he has prior over the states of nature. We do not dispute these mathematical arguments (which does not mean we accept Savage’s axioms). However, we will argue that they cannot justify or even motivate the positions of Lindley or Bernardo. The mid-twenty century Bayesian approach described relatively simple models with only a few unknown parameters. We are now in the age of big data and deep networks. A simple current statistical model involves many unknown parameters. Regularly, there are much more unknown parameters than observations. This has a few major implications to the feasibility and success of the Bayesian approach. To begin with, it is difficult if not utterly impossible (in theory!) to have a world view about a set of thousand parameters. Suppose we, somehow, arrive to a prior which describes properly the unknown certainty of the a compound vector of parameters. If there is a consistent estimator then a Bayesian procedure is consistent under its prior, Ritov et al. 2014 . This is a strong statement for a procedure which is continuous in low dimension. It is a very weak statement in an ultra-high dimension world, where a set with high probability under one prior has almost zero measure by a slightly different prior. Naive assumptions about the proper world-view may bring a very misleading Bayesian procedure. Consider a simple case. Let \Theta_{0}\subset{\mathbb{R}} be a compact set. We can come with a reasonable prior describing our uncertainty on a parameter on \Theta_{0} . Suppose now that we consider \Theta_{0}^{p} , i.e., we consider a vector of p parameters: \vartheta_{i}\in\Theta_{0} , i=1,\dots,p . The \vartheta_{i} s are, to the best of our knowledge, similar and unrelated. Typically, our prior considers them independent, or at least, exchangeable. But this have a strong implications. For example, the means of any large subset of them, \vartheta_{i_{1}},\dots,\vartheta_{i_{q}}

Read the complete text