BB plot: A Tool for Accurate Model Selection Using Bayes factors

Ankur Barsode · 2026

Read BB plot: A Tool for Accurate Model Selection Using Bayes factors by Ankur Barsode (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10333.txt.

Introduction

Ankur Barsode presents a diagnostic tool for choosing between competing hypotheses in physics and astronomy. The usual approach computes the Bayes factor, the ratio of evidences for data under each hypothesis, and reads it as posterior odds or as a ranking statistic for a frequentist test. His 2026 paper describes the BB relationship, linking a Bayes factor to its distributions under both hypotheses, expressed as a plot.

Using gravitational-wave astronomy examples, With wave examples, Barsode shows BB plots checking Bayes-factor math, catching errors that rough methods might hide. The relationship may also estimate background distributions of the Bayes factor cheaply, even analytically in some cases. Applied to wave-optics lensing of gravitational waves, the technique extrapolates the background from the GWTC4 catalog to bound the significance of event GW231123 at roughly 4.1 sigma.

Catalog record

CREATORAnkur Barsode
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BB plot: A Tool for Accurate Model Selection Using Bayes factors Ankur Barsode Abstract A common task in physics and astronomy is studying which of the competing hypotheses the data prefer. This is usually done by computing the Bayes factor between the two hypotheses, and either interpreting it in terms of the posterior odds or as a ranking statistic for a frequentist p-value test. Here we describe a relationship between the Bayes factor and its distributions under the two competing hypotheses, called the Bayes factor-Bayes factor ( BB ) relationship, expressed as a diagnostic plot. Using examples from gravitational wave (GW) astronomy, we demonstrate how the BB plot can validate the accuracy of Bayes factor calculations. The BB relationship may also be useful for estimating background distributions of the Bayes factor at low computational cost, even analytically in some cases. We apply this technique in the context of wave-optics lensing of GWs, extrapolating the background distribution from GWTC4 to put a rough bound of ≲ 4.1 ​ σ \lesssim 4.1\sigma on the statistical significance of GW231123. I Introduction Model selection is a central task in many areas of physics and astronomy, where one seeks to determine which among the proposed hypotheses are consistent with the observations, and which must be rejected. It lies at the heart of most major scientific discoveries that confirmed or challenged prevailing scientific theories Abbott et al. 2016 ; Collaboration et al. 2012 ; Abbott et al. 2017 ; Yu and Allen 2025 ; Riess et al. 1998 ; Adame et al. 2025 . In modern applications, this task is typically statistical in nature and often involves comparing the relative plausibility of models given noisy observations. In the Bayesian approach, we compute the posterior odds in favor of one hypothesis over another. This is encoded by the Bayes factor, which is the ratio of evidences of data under each hypothesis, to be compared against an empirical scale (for eg, Jeffrey’s Jeffreys 1939 ) for interpretation. On the other hand, the frequentist approach seeks to reject a “null” hypothesis based on a test “statistic” whose value is compared with its “background” distribution under the null hypothesis. The statistical significance is then expressed using the false positive probability (FPP), often quoted in terms of Gaussian significance; with \geq 5\sigma usually considered a decisive detection. It is imperative that the statistic be appropriately chosen to maximize the efficiency of detecting the phenomenon of interest, while simultaneously minimizing false alarms. Such a statistic happens to be the Bayes factor itself Neyman and Pearson 1933 . Thus, accurate Bayes factors provide the basis for detection under both of these approaches. However, they may be challenging to calculate exactly due to the complexity of realistic models and limited computational resources. Approximations are often made, which raises a question of how correct they really are. Also, as human researchers, we must acknowledge the possibility that our calculations could have errors. If the (numerically) exact calculation of the Bayes factor using, for eg, nested sampling Skilling 2006 ; Sivia and Skilling 2006 , is possible, only computationally expensive, we may consider validating our approximation of the Bayes factor in a few, typical examples by comparing it against the exact answer. However, if such ground truth is too expensive to obtain, we must resort to internal consistency checks, such as those based on theorems by Alan Turing (the Good checks Good 1965 ; Sekulovski et al. 2024 ; Okada 2026 ) or other simulation based checks Modrák et al. 2023 ; Modrák et al. 2025 . In this paper, we present a discussion of a simulation based validation approach introduced in Barsode et al. 2026 . This approach, called the Bayes factor-Bayes factor or BB plot, can (i) provide a direct and internally consistent diagnostic for assessing the accuracy of approximate Bayes factor calculations without requiring access to ground-truth results, (ii) guide the systematic improvement of such approximations, and (iii) help construct computationally efficient estimates of frequentist background distributions, including semi-analytical extrapolations to regimes inaccessible to brute-force simulations. We illustrate these ideas using applications in gravitational wave (GW) astronomy, including waveform distortion searches and strong lensing, and apply them to place bounds on the statistical significance of GW231123 Abac et al. 2025a . This paper is structured as follows: in Sec. II we provide a primer on Bayesian analysis and the BB plot. In Sec. III , we demonstrate benchmarking using BB plot in two model selection problems in GW astronomy. In Sec. IV , we estimate backgrounds for gravitational lensing of GWs using the BB relationship and estimate an upper bound on the statistical significance of GW231123. Our conclusions are presented in Sec. V . II The Bayes factor-Bayes factor ( BB ) Plot At the heart of Bayesian model selection lies the Bayes theorem P(\vec{\theta}_{i}\mid d,\mathcal{H}_{i})=\dfrac{P(d\mid\vec{\theta}_{i},\mathcal{H}_{i})~P(\vec{\theta}_{i}\mid\mathcal{H}_{i})}{P(d\mid\mathcal{H}_{i})} (1) which, for a model hypothesis \mathcal{H}_{i} , expresses the posterior distribution of model parameters \vec{\theta}_{i} , given data d , in terms of the likelihood of getting the data given those parameters P(d\mid\vec{\theta}_{i},\mathcal{H}_{i}) , and their prior distribution P(\vec{\theta}_{i}\mid\mathcal{H}_{i}) . The denominator on the right is the Bayesian evidence that normalizes the posterior P(d\mid\mathcal{H}_{i})=\int\mathrm{d}\vec{\theta}_{i}~P(d\mid\vec{\theta}_{i},\mathcal{H}_{i})~P(\vec{\theta}_{i}\mid\mathcal{H}_{i}) (2) The Bayes factor is defined as the ratio of evidences under different model hypotheses, say, for i=1,2 , {\mathcal{B}^{1}_{2}}(d)=\dfrac{P(d\mid{\mathcal{H}_{1}})}{P(d\mid{\mathcal{H}_{2}})}. (3) It is often called the likelihood ratio of the competing models for producing the observed data. Note that the Bayes factor also implicitly depends on the prior choices under each hypothesis, apart from being an explicit function of the data. It is useful to clarify what is meant by “data” in this context. The raw numbers recorded by the instruments typically pass through a pipeline of processes that reduce, refine, and simplify them. For a given observation, the data product at the end of every intermediate process in this pipeline should, in principle, give the same Bayes factor. Ultimately, we might consider the calculation of the Bayes factor itself as a process that returns “data” (= the Bayes factor) using which we could compute the Bayes factor. Thus {\mathcal{B}^{1}_{2}}=\dfrac{P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{1}})}{P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{2}})}, (4) where the Bayes factor is being treated as a random variable, and the equality holds at the level of its probability densities under each hypothesis. More formally, we can derive the above relationship by noting that the probability density of the Bayes factor is given by marginalizing over all possible data that result in that Bayes factor. For the hypothesis {\mathcal{H}_{1}} , this is P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{1}})=\int\mathrm{d}d~P(d\mid{\mathcal{H}_{1}})~\delta\left({\mathcal{B}^{1}_{2}}-b(d)\right) (5) where \delta is the Dirac delta function, and we are temporarily denoting the Bayes factor function by b(d) to avoid confusion with the Bayes factor as a random variable {\mathcal{B}^{1}_{2}} . Using the definition in Eq. ( 3 ), we can substitute P(d\mid{\mathcal{H}_{1}}) by b(d)\cdot P(d\mid{\mathcal{H}_{2}}) P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{1}})=\int\mathrm{d}d~b(d)~P(d\mid{\mathcal{H}_{2}})~\delta\left({\mathcal{B}^{1}_{2}}-b(d)\right). (6) The Dirac delta function restricts the integral to those data realizations for which b(d)={\mathcal{B}^{1}_{2}} . On this support, b(d) can therefore be treated as a constant within the integral. This gives P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{1}})={\mathcal{B}^{1}_{2}}~\int\mathrm{d}d~P(d\mid{\mathcal{H}_{2}})~\delta\left({\mathcal{B}^{1}_{2}}-b(d)\right). (7) The integral is analogous to Eq. ( 5 ), only in this case it gives the distribution of Bayes factors under the hypothesis {\mathcal{H}_{2}} , resulting in P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{1}})={\mathcal{B}^{1}_{2}}~P({\mathcal{B}^{1}_{2}}\mid{\mathcal{H}_{2}}), (8) which is identical to Eq. ( 4 ). Thus, the ratio of the PDFs of

Read the complete text