Posterior SBC: Simulation-Based Calibration Checking Conditional on Data
Teemu Säilynoja, Marvin Schmitt, Paul-Christian Bürkner, Aki Vehtari · 2025
Read Posterior SBC: Simulation-Based Calibration Checking Conditional on Data by Teemu Säilynoja, Marvin Schmitt, Paul-Christian Bürkner, Aki Vehtari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03279.txt.
Introduction
Teemu Säilynoja and co-authors sharpen a standard check for Bayesian inference code in this 2025 paper. Simulation-based calibration tests a method by re-running it on fake data. The classic version draws fake settings from the prior, before seeing real data.
After real data arrive, a different question matters more. Does the method work on data like the ones actually observed? Their posterior version conditions the check on the observed dataset. That focuses testing power where the analyst actually stands. A check can pass globally yet fail near the real case. Statisticians gain a sharper tool for trusting their own pipelines. Software packages can adopt the sharper check directly.
Catalog record
| CREATOR | Teemu Säilynoja, Marvin Schmitt, Paul-Christian Bürkner, Aki Vehtari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03279
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Posterior SBC: Simulation-Based Calibration Checking Conditional on Data Teemu Säilynoja, Marvin Schmitt, Paul-Christian Bürkner, Aki Vehtari Abstract Simulation-based calibration checking (SBC) refers to the validation of an inference algorithm and model implementation through repeated inference on data simulated from a generative model. In the original and commonly used approach, the generative model uses parameters drawn from the prior, and thus the approach is testing whether the inference works for simulated data generated with parameter values plausible under that prior. This approach is natural and desirable when we want to test whether the inference works for a wide range of datasets we might observe. However, after observing data, we are interested in answering whether the inference works conditional on that particular data. In this paper, we propose posterior SBC and demonstrate how it can be used to validate the inference conditionally on observed data. We illustrate the utility of posterior SBC in three case studies: (1) A simple multilevel model; (2) a model that is governed by differential equations; and (3) a joint integrative neuroscience model which is approximated via amortized Bayesian inference with neural networks. 1 Introduction (a) Two cancelling biases (b) One problematic region Figure 1 : Two conceptual illustrations of a model parameter space, with gray area denoting the prior. In (a), the colored regions are potential posteriors with bias in opposite directions, while the inference is well calibrated for parameter values outside these regions. Prior SBC will not show calibration issues due to cancellation of biases, while posterior SBC would indicate issues for posteriors intersecting the colored regions. In (b), only one region is problematic. Now prior SBC would indicate calibration issues, while posterior SBC would show that the inference is calibrated outside the colored region. In both cases posterior SBC has the benefit of focusing on the region, which matters given the observed dataset. To ensure trust on the results of the inference, different forms of calibration checking play important roles in the Bayesian workflow for model building ( Gelman et al.,, 2020 ) . Simulation-based calibration checking ( Cook et al.,, 2006 ; Talts et al.,, 2020 ; Modrák et al.,, 2023 , SBC; ) is one of these methods and validates the chosen posterior inference algorithm as well as checks for inconsistencies between the model implementation and a possibly separate implementation of the data generating process. Probabilistic programming software ( Štrumbelj et al.,, 2024 ) provide probabilistic programming languages to make it easier to implement probabilistic models, and inference engines that implement various inference algorithms. As the probabilistic programming software and algorithms for probabilistic modelling advance, the threshold for specifying increasingly complicated models gets lowered. However, as the complexity of the models increases, so does the probability of human error when writing the code, which implements the model. In addition to such model code implementation mistakes, the inference algorithms used for obtaining posterior approximations—even when correctly coded—can work well for one posterior, but suffer from computational difficulties for another. By simply changing the observed data, one may obtain vastly different posterior geometries, favoring one implementation or algorithm over another. SBC aims to check that the inference is calibrated, but miscalibration can be caused either a mistake in the model implementation code, a mistake in inference algorithm implementation code, or significant bias in the inference algorithm. The algorithm is considered to be calibrated, if the probability integral transformation (PIT) of the parameters is uniformly distributed, when transformed with respect to the posterior conditional on data generated using those parameters. We review SBC and the definition of calibration in more detail in Section 2 . In this paper we assume that the both the code for implementing the model, and the inference algorithm implementation are correct and focus on examples were the inference algorithm may produce significantly biased results with finite computation time. Even when the inference algorithm is implemented correctly, it may produce significantly biased inference for some posteriors. For example, many Markov chain Monte Carlo (MCMC) and variational inference approaches have challenges with funnel shaped posteriors ( Neal,, 2003 ; Yao et al.,, 2018 , see, e.g.) , which are common in case of hierarchical models. The original SBC is limited to checking for calibration on the whole joint distribution of parameters and observations under the specified priors, and thus we call it prior SBC . Specifying prior distributions that accurately present the available prior information is often difficult. In some cases, the prior may have significant probability mass in parts of the parameter space that generate such data for which the inference algorithm produce significantly biased inference (in finite time). For example, the data generated from a hierarchical model with certain type of parameter values can lead to a funnel shaped posterior which has a challenging geometry for many inference algorithms to explore. Figure 1 illustrates two cases: a) in which the prior space includes two subspaces with canceling biases and SBC indicates that the calibration over the prior is good, and b) in which the prior space includes one subspace with biased inference causing SBC to indicate that the calibration over the prior is not good. However, after observing some data we are not interested in calibration over the prior, but calibration within the region of the posterior. In this paper, we introduce posterior SBC , a variant of SBC using the self-consistency of Bayesian posterior distribution to validate the model implementation and inference algorithm when conditioned on some fixed set of data. Figure 1 illustrates two cases where, if the posterior is much more concentrated than the prior, posterior SBC checks the calibration of the inference for those regions of the parameter space where it matters and does not care how well the inference algorithm works elsewhere. Prior SBC is useful for algorithm and software developers, while posterior SBC is useful for modellers. As the number of modellers is several orders of magnitude bigger than the number of developers, we expect the posterior SBC to become more popular than prior SBC. We next review the method of simulation-based calibration as it has traditionally been operationalized. In Section 3 , we introduce posterior SBC. In Section 4 we illustrate the differences between prior and posterior SBC through three case studies including a simple hierarchical model, Lotka-Volterra model, and a drift-diffusion model. In the first two examples we use MCMC and in the last example we use amortized Bayesian inference for which there are no similar convergence diagnostics as for MCMC. Finally, in Section 5 we summarize the contributions of the paper and provide an outlook for future research. 2 Simulation-based calibration checking Let us consider a Bayesian model of the joint distribution of the data y and parameters \theta , \pi(y,\theta)=\pi(y\,|\,\theta)\pi(\theta), where \pi(\theta) is the prior distribution of the parameters and \pi(y\,|\,\theta) the likelihood of the data. Cook et al., (2006) propose a simulation-based calibration checking method for validating Bayesian inference software designed to fit the model. The key to this method is the following self-consistency property of Bayesian posterior distributions. Let \theta^{\prime}\sim\pi(\theta) , and y^{\prime}\sim\pi(y\,|\,\theta^{\prime}) . Now the pair (y^{\prime},\theta^{\prime}) represents a draw from the joint distribution \pi(y,\theta) , and therefore \theta^{\prime} is a draw from the posterior distribution, \pi(\theta\,|\,y^{\prime}) .