Recommendations for visual predictive checks in Bayesian workflow
Teemu Säilynoja, Andrew R. Johnson, Osvaldo A. Martin, Aki Vehtari · 2025
Read Recommendations for visual predictive checks in Bayesian workflow by Teemu Säilynoja, Andrew R. Johnson, Osvaldo A. Martin, Aki Vehtari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01509.txt.
Introduction
Teemu Säilynoja and three co-authors give practical guidance for checking Bayesian models with pictures. Their work appeared in twenty twenty-five. Analysts plot model predictions against data to judge fit, but common plots can mislead when their hidden assumptions fail.
The authors treat each check itself as a small model that can fail. They offer rules for picking the right plot for continuous data, count data, and mixes of both. Their diagnostics flag when data are likely discrete, when values hit hidden bounds, and when smooth density curves fit poorly. The aim is fewer ad-hoc choices and stronger model criticism. The advice is meant for everyday modeling work, not just theory.
Catalog record
| CREATOR | Teemu Säilynoja, Andrew R. Johnson, Osvaldo A. Martin, Aki Vehtari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Recommendations for visual predictive checks in Bayesian workflow Teemu Säilynoja, Andrew R. Johnson, Osvaldo A. Martin, Aki Vehtari Abstract This article is intended to be viewed as an HTML, and its latest version is hosted online here . Under Review This paper is under review on the experimental track of the Journal of Visualization and Interaction . See the reviewing process . Abstract Introduction A key step in the Bayesian workflow for model building is the graphical assessment of model predictions, whether these are drawn from the prior or posterior predictive distribution. The goal of these assessments is to identify whether the model is a reasonable (and ideally accurate) representation of the domain knowledge and/or observed data. There are many commonly used visual predictive checks which can be misleading if their implicit assumptions do not match the reality. Thus, there is a need for more guidance for selecting, interpreting, and diagnosing appropriate visualizations. As a visual predictive check itself can be viewed as a model fit to data, assessing when this model fails to represent the data is important for drawing well-informed conclusions. Demonstration We present recommendations for appropriate visual predictive checks for observations that are: continuous, discrete, or a mixture of the two. We also discuss diagnostics to aid in the selection of visual methods. Specifically, in the detection of an incorrect assumption of continuously-distributed data: identifying when data is likely to be discrete or contain discrete components, detecting and estimating possible bounds in data, and a diagnostic of the goodness-of-fit to data for density plots made through kernel density estimates. Conclusion We offer recommendations and diagnostic tools to mitigate ad-hoc decision-making in visual predictive checks. These contributions aim to improve the robustness and interpretability of Bayesian model criticism practices. Materials, Authorship, License, Conflicts Research materials Source files of this article, as well as supplementary materials providing additional detail into the examples and case studies shown in this article, can be found at https://teemusailynoja.github.io/visual-predictive-checks . Authorship • Teemu Säilynoja : Conceptualization, Methodology, Software, Investigation, Writing - Review & Editing, Visualization • Andrew R. Johnson : Conceptualization, Writing - Review & Editing • Osvaldo A. Martin : Writing - Review & Editing, Visualization • Aki Vehtari : Conceptualization, Methodology, Writing - Review & Editing, Supervision, Funding acquisition. License This work is licensed under a Creative Commons Attribution 4.0 International License . Conflicts of interest The authors declare that there are no competing interests. 1 Introduction Assessing the sensibility of model predictions and their fit to observations is a key part of most model building workflows. These assessments may reveal that the model predictions poorly represent (or replicate) the observed data, prompting the modeller to either improve their model, or adjust their confidence in the predictions accordingly. In this paper, we focus on examples rising from Bayesian workflows ( [ ref-gabry_visualization_2019 ] ; [ ref-gelman_bayesian_2020 ] ), such as posterior predictive checking [PPC; Box ( [ ref-box_sampling_1980 ] );Rubin ( [ ref-rubin_bayesianly_1984 ] ); Gelman, Xiao-Li, and Stern ( [ ref-gelman_posterior_1996 ] ); Gelman et al. ( [ ref-gelman_bayesian_2013 ] )], and visualizations of data, posterior, and posterior predictive distributions. Visualization of data distribution is useful even if we do not do any modeling. In Bayesian inference, the posterior distribution presents the uncertainty in the parameter values after conditioning on data, and the posterior predictive distribution presents the uncertainty in the predictions for new observations. The posterior inference is often performed using Monte Carlo methods ( [ ref-strumbelj_past_2024 ] ), and posterior and posterior predictive distributions are then represented with draws from these distributions. In the same way as we visualize data distributions, we can visualize posterior and posterior predictive distributions. In posterior predictive checking, we compare the data distribution to the posterior predictive distribution. If these distributions are not similar, the model is misspecified and we should consider improving the model. These visualizations are also applicable to prior predictive checking, where one might compare prior predictive samples to some reference values rising from domain knowledge ( [ ref-wesnerChoosingPriorsBayesian2021 ] ), and to cross-validation predictive checking where data are compared to cross-validation predictive distributions to avoid double use of data. The purpose of this paper is to illustrate common issues in visual predictive checking, to provide better recommendations for different use cases, and to provide diagnostics to automatically warn if useless or misleading visualization is used. Together we have decades of experience in teaching visual predictive checking and helping modellers using popular visual predictive checking tools such as bayesplot (Gabry and Mahr ( [ ref-gabry_plotting_2022 ] )) and ArviZ (Kumar et al. ( [ ref-kumar_arviz_2019 ] )). We have seen many times where students and modellers use commonly recommended visualizations without realizing that in their context, the specific visualization is the wrong one and either useless or even misleading. Backed up with arguments from recent studies in uncertainty visualization, we provide a summary of methods we recommend. These methods are aimed to offer an informed basis for decision-making, be it for the modeller to improve on the model or for the end user to set expectations on the performance of the model. The visualizations we discuss should be considered as broadly-applicable tools aimed to give insight into the overall goodness of model fit to the observations, as well as to possible deviations in certain parts of the predictive density. Most modeling workflows benefit from additional visualizations tailored to their specific use cases, as there is no general visual or numeric check that would reveal every aspect of the predictive capabilities of any given model. Some commonly-used methods for visualizing model predictions have risen from comparisons of continuous variables, and as such have varying degrees of usability for assessing predictions with discrete values. Some may not work with all continuous cases, in case of sharp upper or lower bounds in the set of possible values the prediction can take. In Section 2 , we inspect the common use of kernel density estimates (KDE) in summarizing the predictive distribution and the observed data, and show comparisons between commonly used visualization approaches. We highlight common cases when these visualizations may hide useful information and propose an automated goodness-of-fit test to alert the modeller of the presence of conflict between the data and the visualization. In Section 3 , we discuss the use of visualizations assuming continuous data when the data and predictions are in reality discrete, but have a high number of unique values. We also discuss an alternate way of visualizing count data, when the number of cases is large, but still small enough for the modeller to be interested in attempting to assess the predictions as individual counts. In Section 4 , we focus on workflows involving binary predictions. We showcase tools that expand the visualizations beyond the typical bar graphs and binned calibration plots and instead allow for a more robust assessment of the calibration of the predictive probabilities. In Section 5 and Section 6 , we show how the visual predictive checks described for binary predictions can be extended to discrete predictions with a small or medium number of individual cases. 2 Visual predictive checks for continuous data In this section, we consider visualizations for observations from a continuous or