Reliability, bias, and computational cost of estimating the Bayes factor using bridge sampling and the Savage–Dickey density ratio Authors: Klaus Oberauer, Philipp Musfeld, Frederik Aust Published: 2026 ABSTRACT Bayes factors often require numerical estimation because closed-form solutions are unavailable. In six simulation studies, we explored the reliability, bias, and computational cost of two easy-to-use and broadly applicable methods: bridge sampling and the Savage–Dickey density ratios, based on Gaussian, logspline, and spline-smoothed kernel density approximations of the posterior distribution, as well as conditional marginal density estimation. In generalized linear mixed effect models for normally and binomially distributed data, we explore the effects of the (1) number of MCMC samples from the posterior, (2) size of effects or magnitude of the Bayes factor, (3) number of participants, and (4) number of model parameters. Our findings suggest that, with enough MCMC samples, both methods yield reliable and accurate estimates across a wide range of conditions. However, with many model parameters, bridge sampling becomes computationally expensive and can be unreliable. In contrast, the Savage–Dickey density ratio scales well, remaining computationally efficient and reliable, even with many model parameters. However, Savage–Dickey density ratio requires careful consideration of posterior density estimation to mitigate bias while limiting the variability of Bayes factor estimates. We provide practical recommendations to guide researchers in selecting the most suitable estimation method for their applications. Introduction Testing whether an effect of an independent variable on a dependent variable exists (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{H}}_{1}$$\end{document}H1) or not (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{H}}_{0}$$\end{document}H0) is the most frequent hypothesis test in psychology and other social sciences. In light of the conceptual problems of classical null-hypothesis significance testing (Wagenmakers, 2007), many researchers are turning to Bayesian hypothesis testing as the alternative with a more sound foundation in statistical theory. In the Bayesian framework, testing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{H}}_{1}$$\end{document}H1 against \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{H}}_{0}$$\end{document}H0 comes down to comparing two statistical models, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0, which differ only in that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1 includes a free parameter for the effect in question, whereas in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0 that parameter is fixed to a value reflecting no effect of the independent variable, such as chance performance or zero. The gold standard for Bayesian model comparison is to use the Bayes factor (BF) as a measure of the relative evidence in favor of one model over the other. The BF follows directly from Bayes’ theorem and is the ratio of the marginal likelihoods of the two models, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{B}\mathrm{F}}_{10}=p\left({\boldsymbol{y}} | {\mathcal{M}}_{1}\right)/p\left({\boldsymbol{y}} | {\mathcal{M}}_{0}\right)$$\end{document}BF10=py|M1/py|M0. The marginal likelihood of a model, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\left({\boldsymbol{y}} | {\mathcal{M}}_{i}\right)$$\end{document}py|Mi, is the average likelihood of the observed data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\boldsymbol{y}}$$\end{document}y across all possible model parameter values, weighted by their prior probability. In other words, the marginal likelihood is a measure of a model’s prior predictive accuracy. Hence, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{B}\mathrm{F}}_{10}$$\end{document}BF10 indicates whether \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1 or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0 more accurately anticipates the observed data, and by how much. For example, a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{B}\mathrm{F}}_{10}=3$$\end{document}BF10=3 means that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1 predicts the observed data 3 times better than \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0. This is why \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{B}\mathrm{F}}_{10}$$\end{document}BF10 is sometimes referred to as the evidence for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1 relative to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0. The direction of the comparison is indicated by the index, hence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${1/\mathrm{B}\mathrm{F}}_{10}={\mathrm{B}\mathrm{F}}_{01}$$\end{document}1/BF10=BF01 gives the evidence for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{0}$$\end{document}M0 relative to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{M}}_{1}$$\end{document}M1. The main limiting factor on using BF for hypothesis testing is practical: They are hard to compute. Analytical solutions exist only for relatively simple cases (Rouder et al., 2012; Rouder et al., 2009), so that researchers mostly have to resort to numeric estimation of BFs based on Markov chain Monte Carlo (MCMC) samples from the posterior distribution. Several estimation methods exist that differ in bias, reliability, generality, and ease of use, such as the Savage–Dickey density ratio (Dickey, 1971; Dickey & Lientz, 1970), Reversible Jump MCMC (Green, 1995), Chib’s method (Chib, 1995), bridge sampling (Meng & Wong, 1996), path sampling (Gelman & Meng, 1998), and others (for a review see Llorente et al., 2023). A few general considerations reduce the number of available estimation methods. First, we believe most researchers place a premium on ease of use. Second, in our experience, nested model comparisons and specifically comparisons that test individual parameters in (hierarchical) generalized linear models are the most common statistical hypothesis tests in psychology and other social sciences. Choosing among the methods that fit this description requires researchers to consider the reliability of BF estimates (i.e., how much the estimates vary from one estimation to the next on the same data), their bias, and their computation costs. Here we report six simulations designed to help practical researchers to choose an easy-to-use BF estimation method for nested model comparisons in (hierarchical) generalized linear models. We investigate two broadly applicable methods of estimating BFs1, bridge sampling (Gronau et al., 2020; Gronau et al., 2018) and the Savage–Dickey density ratio (Wagenmakers et al., 2010). Bridge sampling is a simulation-based method for approximating the marginal likelihood of a model—again, Bayes factors are simply the ratio of competing models’ marginal likelihoods. Bridge sampling is a general method in the sense that it is not restricted to comparisons of nested models, where the constrained model results from removing the parameter of interest θ (e.g., a condition difference) from the encompassing model, but is also applicable to non-nested model comparisons (Gronau et al., 2017; Meng & Wong, 1996). Unfortunately, this generality comes at a cost: Bridge sampling can be computationally expensive. Estimation of the marginal likelihood requires that one obtain a relatively large number of samples from the joint posterior distribution, which are then submitted to the bridge sampling algorithm. Both steps can be computationally demanding and are required for all competing models. The Savage–Dickey density ratio is a simpler but less flexible method to calculate the Bayes factor. Instead of estimating each models’ marginal likelihood separately, the Bayes Factor is calculated directly as the ratio of the prior and the posterior density of the parameter of interest θ (e.g., a condition difference) at a theoretically interesting value (e.g., chance performance or zero; Wagenmakers et al., 2010). Often this requires the estimation of the marginal posterior distribution of θ because it is not analytically available. The Savage–Dickey density ratio is computationally less expensive than bridge sampling because (1) only posterior samples from the encompassing model are required2 and (2) approximating the marginal posterior distribution for the parameter of interest θ is less computationally demanding than the bridge sampling algorithm. However, this efficiency again comes at a cost: The Savage–Dickey density ratio is limited to nested model comparisons (and prior distributions that assign a finite, non-zero density to the test value; see Heck (2019) for other important caveats). Moreover, it is difficult in practice to use the Savage–Dickey density ratio to compare models that differ in a more than one parameter. A common example are omnibus tests of the overall effect of a categorical predictor with more than two levels, where that effect is represented by n−1 parameters (for n levels) in the model.3 To the best of our knowledge, there is limited guidance on choosing between bridge sampling and the Savage–Dickey density ratio when both methods are applicable. However, several factors are known to affect the reliability and bias of these methods. By reliability, we mean the variance of repeated estimates of the BF with the same method using the same data. In several examples of generalized linear models, bridge sampling estimates of the BF have been found to be unbiased and reliable compared to reversible jump MCMC, importance sampling, and Chib’s method (Meng & Schilling, 1996; Sinharay & Stern, 2005). The reliability of the estimates depends on the number of MCMC samples (Gronau, Wagenmakers, Heck, & Matzke, 2019), but beyond rules of thumb (“about an order of magnitude more posterior samples than estimation”, p. 12, Gronau et al., 2018), the practical researcher is left to their own devices. In Savage–Dickey density ratios, the bias and variance of the estimates depends on the method used to approximate the posterior density and its ability to match the true distribution (Morey et al., 2011). Two popular methods are logspline density estimation (Makowski et al., 2019; Morey et al., 2011; Wagenmakers et al., 2010) and approximating the posterior by a normal distribution (Morey et al., 2011). The latter approach appeals to the Bayesian central limit theorem, which posits that asymptotically marginal posterior distributions will be normally distributed (Bernardo & Smith, 2000; cf. Bernstein-von Mises theorem). Morey et al. (2011) compared both methods in a simulation study for a simple t test. They found that assuming a normal distribution yields more reliable estimates than logspline density estimation, but reliabilities of both estimates can increase with the number of MCMC samples used for the density estimation. In finite samples, both methods yielded biased estimates—in particular when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}$$\end{document}BF10 was large—but for different reasons. Assuming a normal distribution yielded biased estimates because the normal distribution misfitted the true posterior distribution. With more data, the bias decreased, as predicted by the Bayesian central limit theorem; increasing the number of MCMC samples did not affect bias. In contrast, logspline density estimation yielded biased estimates because it is difficult to estimate the density in the tail of the distribution where few MCMC samples are available. The logspline method addresses this problem by assuming that the density decreases exponentially when few samples are available (Kooperberg & Stone, 1991). Hence, increasing the number of MCMC samples increases reliability and decreases bias of the BF estimates. However, compared to the normal approximation, logspline density estimates performed so poorly overall that Morey et al. (2011) did not consider them in subsequent simulations. Based on these findings, an informative comparison of bridge sampling and the Savage–Dickey density ratio should compare their bias and reliability and how they are affected by the number of MCMC samples, the magnitude of BF (e.g., effect size), and the sample size (which affects the normality of the posterior). In addition, the number of model parameters is also likely to be a relevant factor because it strongly influences the difficulty of estimating the posteriors. Morey et al. (2011) also considered a third method for computing Savage–Dickey density ratios, conditional marginal density estimation (Chib, 1995; Gelfand & Adrian, 1990), and show that it is a more reliable and less biased estimation method than the logspline and normal approximations of the posterior probability density. Unfortunately, this method is only applicable when the conditional posterior distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\left(\theta \right|\phi , {\boldsymbol{y}}, {\mathcal{M}}_{1})$$\end{document}pθϕ,y,M1) is known, which is usually not the case in hierarchical generalized linear models. Therefore, we do not consider conditional marginal density estimation in most of our simulations. We include it as a comparison standard in Simulation 6. The remainder of this paper is organized as follows. We present six simulation studies that compare Bayes factors estimated using the Savage–Dickey density ratio and the bridge sampling algorithm (for an overview, see Table 1). Simulations 1–4 explore bias and reliability of bridge sampling and the Savage–Dickey density ratio for hierarchical generalized linear models, which we apply to simulations of typical experimental designs in psychology. Simulation 1 varies the number of posterior samples and compares the estimation methods for normally and binomially distributed data. Simulation 2 varies the effect size and thereby the magnitude of the to-be-estimated BF. Simulations 3 and 4 explore the effect of model complexity. In Simulation 3, we increase the number of to-be-estimated parameters, while keeping the sample size rather small (N = 20). In Simulation 4, we additionally increased the sample size (N = 100), which further increases the number of to-be-estimated parameters due to the estimation of random participant effects. Simulations 5 and 6 then explore some noticeable findings of our first five simulations in more depth. Simulation 5 compares three density estimators, widely used to obtain Savage–Dickey density ratios, by varying the tail probability of the test value and the number of posterior samples. In a simplified model for which analytic BFs are available, Simulation 6 jointly varies the magnitude of the to-be-estimated BF and the number of model parameters. We compare bridge sampling to Savage–Dickey density ratios based on the three density estimators explored in Simulation 5 and assess their reliability, bias, and computational cost. Finally, we summarize our findings and discuss the factors that should be considered when choosing between the Savage–Dickey density ratio and bridge sampling. Table 1Overview of findingsSimulationQuestionConclusion1Effect of number of MCMC samplesReliability of bridge sampling and Savage–Dickey BF estimates increases with number of samples2Effect of the size of experimental effectsThe variability of BFs scales with the mean BF. It is tolerable for all effect sizes3More complex experimental designReliability of BF estimates is not compromised by more complex designs4Larger sample size, with more parameters to estimateReliability of bridge sampling BF estimates suffers, and becomes worryingly poor. Savage–Dickey estimates still have tolerable variability.5Comparison of different methods for computing the Savage–Dickey ratioFor normally distributed posteriors, the normal approximation is less variable and less biased than the logspline density estimation, which is less variable than spline-smoothed kernel density estimation. The normal approximation risks substantial bias when the posterior distributions deviate strongly from normality. Logspline density estimation can fail catastrophically when the number of MCMC samples is too high6Effect of magnitude of evidence for an effect, and of the number of model parametersVariability of the Savage–Dickey BF estimates increases only with the strength of evidence for an effect, whereas the variability of bridge-sampling estimates increases only with the number of model parameters. Bridge sampling and CMDE remain unbiased across all conditions, whereas bias of all other Savage–Dickey approximation methods increases with stronger evidence for an effect Table 1Overview of findingsSimulationQuestionConclusion1Effect of number of MCMC samplesReliability of bridge sampling and Savage–Dickey BF estimates increases with number of samples2Effect of the size of experimental effectsThe variability of BFs scales with the mean BF. It is tolerable for all effect sizes3More complex experimental designReliability of BF estimates is not compromised by more complex designs4Larger sample size, with more parameters to estimateReliability of bridge sampling BF estimates suffers, and becomes worryingly poor. Savage–Dickey estimates still have tolerable variability.5Comparison of different methods for computing the Savage–Dickey ratioFor normally distributed posteriors, the normal approximation is less variable and less biased than the logspline density estimation, which is less variable than spline-smoothed kernel density estimation. The normal approximation risks substantial bias when the posterior distributions deviate strongly from normality. Logspline density estimation can fail catastrophically when the number of MCMC samples is too high6Effect of magnitude of evidence for an effect, and of the number of model parametersVariability of the Savage–Dickey BF estimates increases only with the strength of evidence for an effect, whereas the variability of bridge-sampling estimates increases only with the number of model parameters. Bridge sampling and CMDE remain unbiased across all conditions, whereas bias of all other Savage–Dickey approximation methods increases with stronger evidence for an effect Overview of findings Simulation 1: Varying MCMC sample size For both bridge sampling and Savage–Dickey density ratio, reliability of BF estimates is limited by the number of MCMC samples (Gronau et al., 2019; Morey et al., 2011). Outside of very simple models and small data sets, however, the number of MCMC samples is limited by the available computational resources and computing time. Hence, researchers face a trade-off: While larger MCMC sample sizes yield more reliable BF estimates, they demand greater computational resources and time. This trade-off may be different for bridge sampling and the Savage–Dickey density ratio. To quantify these relationships, we conducted a simulation study examining how MCMC sample size affects BF estimate reliability in hierarchical generalized linear models for typical experimental designs. Simulation 1a: Normally distributed dependent variable We ran simulations for a typical experimental design in psychology, involving one between- and one within-subjects manipulation as independent variables (IV). We considered two kinds of dependent variables: (1) Normally distributed, because the assumption of normality is pervasive and underlies the classical frequentist ANOVA and the default Bayesian ANOVA (Rouder et al., 2012)4, and (2) binomially distributed, because dichotomous outcomes, like two-alternative forced-choice or responses coded as correct/incorrect, are also common in experimental psychology. The simulated experiment has two IVs. IV 1 is a continuous variable varied within-subjects with five levels; IV 2 is a categorical variable varied between subjects with two levels (i.e., two groups). Each IV had a true effect size of d = 0.25, but there was no interaction. Effect sizes were unstandardized and defined as the deviation of condition means from the grand mean, rather than as the difference between condition means. Data were simulated for N = 20 subjects per group, and n = 150 responses per design cell, according to the following model:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{array}{l}{y}_{ij}\sim N({m}_{i}+{d}_{1,i}{X}_{1,i,j}+{d}_{2}{X}_{2,i},\zeta )\\ {m}_{i}\sim N(0,{\sigma}_{m})\\ {d}_{1,i}\sim N({d}_{1},{\sigma}_{{d}_{1}})\end{array}$$\end{document}yij∼N(mi+d1,iX1,i,j+d2X2,i,ζ)mi∼N(0,σm)d1,i∼N(d1,σd1) Here, yij is the response of subject i in trial j. X1,i,j is the level of IV 1 of trial j for subject i; X2,i is the level of IV 2 of subject i; d1 and d2 are the fixed effect sizes of IV 1 and IV 2, respectively. IV 1 was z-standardized (resulting in levels – 1.265, – 0.632, 0, 0.632, 1.265); IV 2 was contrast-coded (– 1 vs. 1). Each subject has an individual mean value of the dependent variable, mi, and an individual effect size of IV 1, d1,i. These random effects are normally distributed with standard deviation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\sigma}_{m}={\sigma}_{{d}_{1}}=0.3$$\end{document}σm=σd1=0.3. Trial-by-trial variability was modeled by normally distributed error with standard deviation ξ = 1.0. We simulated ten data sets and estimated hierarchical linear models assuming normally distributed residuals with the brms package in R (Bürkner, 2025; R Core Team, 2024). The encompassing model matched the data-generating process, but additionally assumed an interaction effect between IV1 and IV2: y ~ iv1 * iv2 + (1 + iv1 || subj) (Oberauer, 2022).5 All estimated models also included random participant effects on all within-subject parameters.6 We used Cauchy priors with scale 0.5 for the fixed effects, and Gamma priors with shape 1 and rate 0.01 for the random effects. Each model was run with eight chains, and we varied the number of MCMC samples per chain (5000 vs. 10,000 samples per chain). The first 1000 samples of each chain were used as HMC warmup and discarded. This results in 32,000 or 72,000 samples from the posterior distribution. The number of samples we investigated far exceeds the default number of samples per chain in brms (4000) because whereas the default number is appropriate for parameter estimation, it is likely to be insufficient for BF estimation. For each simulated data set, we estimated the evidence for each effect (BF10) via bridge sampling (Gronau et al., 2020; Gronau et al., 2018) and the Savage–Dickey density ratio (Wagenmakers et al., 2010). For the bridge sampling method, we estimated BFs through a series of model comparisons. For each effect, we compared the full model to a model in which the fixed effect of interest – but not the corresponding random effect – was removed. BFs were estimated using the bayes_factor() function in brms, which in turn calls the bridge_sampler() function from the bridge-sampling R-package that estimates the marginal likelihoods of each model (Gronau et al., 2018). We used all default settings, which set the bridge sampling method to “normal” (as compared to “Warp-III”) and runs the bridge sampler for a maximum of 1000 iterations. For the Savage–Dickey density ratio, we determined the prior density at zero from the analytical density of the Cauchy prior. For the corresponding posterior density, we assumed the posterior distribution to be approximately normally distributed (Bayesian central limit theorem; Bernardo & Smith, 2000) and therefore we evaluated the density of a normal distribution at zero with mean and standard deviation equal to the posterior mean and the standard deviation for the effect in question (e.g., the main effect of IV 1). Hence, the estimated BF10 in favor of the effect Xi is\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{array}{cl}B{F}_{10}\left({X}_{i}\right)=\frac{Cauchy\left(0;\mathrm{0,0.5}\right)}{Normal\left(0;\mu \left({d}_{i}\right),\sigma \left({d}_{i}\right)\right)}\\ \mu \left({d}_{i}\right)=\frac{1}{m}\sum\limits_{k=1}^{m}{d}_{i,k};\\ \sigma \left({d}_{i}\right)=\sqrt{\frac{1}{m-1}{\sum\limits_{k=1}^{m}\left({d}_{i,k}-\mu \left({d}_{i}\right)\right)}^{2}}\end{array}$$\end{document}BF10Xi=Cauchy0;0,0.5Normal0;μdi,σdiμdi=1m∑k=1mdi,k;σdi=1m-1∑k=1mdi,k-μdi2where di,k is the kth sample from the MCMC sample of effect size di. To gauge the variance of BF estimates when we repeat the full data analysis on the same data, we repeated the procedure—sampling the posterior distribution and estimating the BFs—ten times for each simulated data set. For all analyses, we used logarithm of basis 10 of the BF10 (log10-BF10) as the variable of interest because raw BF10 values grow to astronomical numbers with larger effect and sample sizes, and their distributions are substantially skewed. The log-transformation results in approximately normal distributions (see Fig. 1 for example distributions).Fig. 1 Distributions of Bayes factors (top panels) and log10(BF) (bottom panels) from Simulation 1a, 72,000 samples), for the main effect of IV 1. To remove variance between data samples, we subtracted the mean within each run from all BF estimates and added the grand mean across all runs at the end. This resulted in some negative BF values for the Savage–Dickey method because some runs had much larger means than the grand mean Distributions of Bayes factors (top panels) and log10(BF) (bottom panels) from Simulation 1a, 72,000 samples), for the main effect of IV 1. To remove variance between data samples, we subtracted the mean within each run from all BF estimates and added the grand mean across all runs at the end. This resulted in some negative BF values for the Savage–Dickey method because some runs had much larger means than the grand mean Table 2 provides a summary of the results: For each level of the number of MCMC samples, the table shows the mean log10-BF across the ten simulations and the ten model-fitting runs per simulation. Positive log10-BF values reflect evidence in favor of an effect; negative values show evidence against the effect. A log10-BF value > 1 implies a BF > 10, which is usually regarded as strong evidence (Jeffreys, 1961; Lee & Wagenmakers, 2014; Wasserman, 2000); a log10-BF > 2 implies a BF > 100, and so on. Table 2 also presents the average standard deviation of log10-BF within each simulation.7 The SD within each simulation reflects the variance of BF estimates across the ten repeated analyses of the same data and thereby informs about the reliability of the BF estimation. Because the standard deviations are calculated on the logarithmic scale, they express the variance of BF estimates as proportions of their means on the original scale. For instance, a standard deviation of log10-BF of 0.02 means that the BF estimates have a dispersion of about 2% of the mean BF value. Table 2Log10 Bayes factors from Simulation 1a and 1bNormal DVBinomial DVMeanStandard deviationMeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-within32,000 Samples3.6715.5840.0130.1784.2326.0620.0090.13872,000 Samples3.6725.5590.0080.1384.2336.0380.0050.095152,000 Samples4.2336.0170.0040.060IV2-between32,000 Samples3.7825.5040.0120.2123.7575.1830.0090.11972,000 Samples3.7845.5270.0070.1323.7575.1780.0050.088152,000 Samples3.7575.1740.0040.063Interaction 1 x 232,000 Samples– 1.184– 1.1750.0100.005– 1.001– 0.9910.0080.00472,000 Samples– 1.183– 1.1740.0070.004– 0.999– 0.9910.0050.003152,000 Samples– 0.998– 0.9910.0030.002 Table 2Log10 Bayes factors from Simulation 1a and 1bNormal DVBinomial DVMeanStandard deviationMeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-within32,000 Samples3.6715.5840.0130.1784.2326.0620.0090.13872,000 Samples3.6725.5590.0080.1384.2336.0380.0050.095152,000 Samples4.2336.0170.0040.060IV2-between32,000 Samples3.7825.5040.0120.2123.7575.1830.0090.11972,000 Samples3.7845.5270.0070.1323.7575.1780.0050.088152,000 Samples3.7575.1740.0040.063Interaction 1 x 232,000 Samples– 1.184– 1.1750.0100.005– 1.001– 0.9910.0080.00472,000 Samples– 1.183– 1.1740.0070.004– 0.999– 0.9910.0050.003152,000 Samples– 0.998– 0.9910.0030.002 Log10 Bayes factors from Simulation 1a and 1b Simulation 1b: Binary dependent variable Often behavioral responses are measured on binary scales, such as whether a person gave a correct or incorrect response to a test question. In that case, the data distribution is binomial rather than normal, and the linear model must be related to the data through a non-linear, typically logistic, link function. Simulation 1b uses the same design as Simulation 1a, except that the dependent variable was generated by a logistic model:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{array}{l}{y}_{ij}\sim Binom({\lambda}_{ij})\\ {\lambda}_{ij}=Logistic\left({m}_{i}+{d}_{1,i}{X}_{1,i,j}+{d}_{2}{X}_{2,i}\right)\end{array}$$\end{document}yij∼Binom(λij)λij=Logisticmi+d1,iX1,i,j+d2X2,i We modeled the data with a hierarchical generalized linear model using the brms package (Bürkner, 2025). The model formula was the same as described for Simulation 1a, but we used a binomial distribution family with a logit link function on the linear model term. For the fixed effects, we used a Cauchy prior with scale = 0.35 because it generates a distribution of effect sizes that, when transformed to the probability scale, is widely spread without a strong concentration of density at the extremes. We again simulated ten data sets and repeated the analytic procedure ten times for each data set. This time, we used 5000, 10,000, or 20,000 MCMC samples per chain. The first 1000 samples of each chain were used as HMC warmup and discarded. This results in 32,000, 72,000, or 152,000 samples from the posterior distribution. Table 2 shows the results. What is the risk of drawing erroneous conclusions about a hypothesis due to the variability of BF estimates? To assess this risk, we applied a very lenient conclusion criterion to the BF estimates, concluding in favor of an effect when BF > 3, and in favor of the null hypothesis when BF < 1/3. For each simulation (i.e., each simulated data set), we calculated the proportion of agreement in these decisions across the ten model-fitting runs for each hypothesis, separately for the BF estimates from the bridge-sampling and the Savage–Dickey method. When individual BF estimates were in the ambiguous range (1/3 < BF < 3), they were omitted because they do not support any decision. We found that the agreement was 100% for all hypothesis tests. Discussion Both simulations show similar patterns of results: For the BFs obtained from bridge sampling, the SD (on the logarithmic scale) was consistently lower than 0.1. We find this tolerable because it will hardly lead to erroneous conclusions. If the true BF is actually 10% higher or lower than the estimate, our assessment of the strength of evidence would not change substantially. For the Savage–Dickey estimates of BF, the SD for small BFs that support the null hypothesis—as in the case of the interaction—were even narrower than for the bridge sampling BFs. By contrast, the SDs were much larger for large BFs, as obtained here for the two main effects. As these BFs were so large that they would unambiguously support the existence of the main effects even if they were an order of magnitude smaller, we could tolerate even this large degree of variability.8 A large degree of variability, however, would be problematic for BFs that provide weaker support for an effect. We designed Simulation 2 to explore BFs in favor of effects in a smaller range. The variation of the number of MCMC samples had the expected effect: More samples resulted in more stable BF estimates. That said, even the smallest sample size (i.e., 32,000) was sufficient to obtain reasonably stable estimates in these simulations. The results for the normal and binomial data were comparable. Simulation 2: Varying effect sizes In Simulation 2, we generated data from the same logistic model as for Simulation 1b. Here, we held the number of post-warmup MCMC samples constant at 72,000 (10,000 per chain, including 1000 warmup samples), and instead varied the effect size of IV 1 and IV 2 (0.1, 0.15, 0.2, 0.25). The effect sizes for both IVs were always the same. Table 3 shows the results. We again calculated the agreement proportion for decisions and found it to be 100%. Table 3 Log10 Bayes factors from Simulation 2MeanSDParametersBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-WithinEffect d = 0.1– 0.036– 0.0160.0060.022Effect d = 0.150.7150.9330.0050.038Effect d = 0.21.2751.5150.0050.035Effect d = 0.252.7673.5860.0050.07IV2-BetweenEffect d = 0.1– 0.31– 0.3190.0050.013Effect d = 0.151.5491.8350.0050.053Effect d = 0.22.3883.1570.0050.058Effect d = 0.253.4634.6430.0050.066Interaction 1 x 2Effect d = 0.1– 0.985– 0.9780.0050.003Effect d = 0.15– 0.973– 0.9650.0050.003Effect d = 0.2– 1.02– 1.0120.0050.003Effect d = 0.25– 0.94– 0.9330.0040.003 Table 3 Log10 Bayes factors from Simulation 2MeanSDParametersBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-WithinEffect d = 0.1– 0.036– 0.0160.0060.022Effect d = 0.150.7150.9330.0050.038Effect d = 0.21.2751.5150.0050.035Effect d = 0.252.7673.5860.0050.07IV2-BetweenEffect d = 0.1– 0.31– 0.3190.0050.013Effect d = 0.151.5491.8350.0050.053Effect d = 0.22.3883.1570.0050.058Effect d = 0.253.4634.6430.0050.066Interaction 1 x 2Effect d = 0.1– 0.985– 0.9780.0050.003Effect d = 0.15– 0.973– 0.9650.0050.003Effect d = 0.2– 1.02– 1.0120.0050.003Effect d = 0.25– 0.94– 0.9330.0040.003 Log10 Bayes factors from Simulation 2 The variability of BF from bridge sampling was again tolerably low throughout. This was the case regardless of the effect size. For the Savage–Dickey density ratio, the variability was larger whenever there was a true effect—as for the two main effects in the present simulation—but somewhat smaller when there was none—as for the interaction. Still, the standard deviations of the Savage–Dickey estimates are far from alarming. The BF values on the original scale increase with the size of the effect, and, as a result, the variability of BF estimates increases. However, the ratio of variability to the size of the BF values, which is reflected in the standard deviations on the logarithmic scale, remain approximately constant. In other words, when the Bayes factors are small, and estimation uncertainty could potentially lead to incorrect conclusions, the variability is low. Simulation 3: Increasing model complexity In Simulation 3, we explored whether the variability of BF estimates increases with more complex models than those we studied so far. We added a third IV to the experimental design, which had two levels varying within subjects. This substantially increases the number of model parameters.9 The data-generating process included linear main effects of all three IVs (with effect sizes d1 = 0.25, d2 = 0.25, and d3 = 0.05) and an interaction between IVs 1 and 3 (d13 = 0.25). We reduced the number of trials per design cell to 50 so that the overall amount of data is still in the typical range for behavioral lab experiments. To reduce computational costs, in Simulations 3 and 4, we followed a different model-comparison strategy for estimating BFs through bridge sampling than in the preceding simulations. We tested each effect by comparing models that omitted all higher-order effects. For example, to test the main effect of IV1, we compared a reference model including main effects for all IVs (but no interactions) to a restricted model that additionally omitted the main effect of IV1. These model comparisons adhere to the principle of marginality.10 We ran this simulation in two versions, one using a linear model with a normal distribution for data generation and as a statistical model (Simulation 3a; aggregating data within design cells to accelerate computation), and one using a logistic model with a binomial distribution (Simulation 3b). The model was applied to the data with 32,000, 72,000, and 152,000 MCMC samples. Table 4 summarizes the results. Table 4Log10 Bayes factors from Simulation 3a and 3bSimulation 3a: Normal DVSimulation 3b: Binomial DVMeanStandard deviationMeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-Within32,000 Samples3.9363.1520.0100.0912.4726.9700.0100.14072,000 Samples3.9363.1530.0060.0512.4726.9810.0070.093152,000 Samples3.9353.1640.0030.0382.4736.9690.0040.067IV2-Between32,000 Samples3.1985.0770.0100.0953.6483.9210.0110.1072,000 Samples3.1975.0720.0050.073.653.9160.0050.058152,000 Samples3.1975.0860.0030.0383.6493.9200.0040.038IV3-Within32,000 Samples– 0.393– 0.5520.0090.013– 0.1920.1300.0100.02572,000 Samples– 0.393– 0.5530.0060.009– 0.1910.1330.0060.016152,000 Samples– 0.393– 0.5520.0040.004– 0.1920.1320.0040.011Interaction 1 x 232,000 Samples– 1.198– 1.1230.0100.003– 0.943– 0.9820.0110.00372,000 Samples– 1.198– 1.1230.0060.002– 0.943– 0.9820.0060.003152,000 Samples– 1.198– 1.1230.0040.001– 0.943– 0.9820.0040.002Interaction 1 x 332,000 Samples31.94332.9990.0090.40840.28735.8110.0100.41672,000 Samples31.94432.9420.0060.32240.28835.8770.0060.325152,000 Samples31.94432.9700.0040.20540.28835.8350.0040.198Interaction 2 x 332,000 Samples– 1.092– 1.1090.0110.004– 0.930– 0.9400.0100.00472,000 Samples– 1.091– 1.1090.0050.002– 0.930– 0.9400.0060.002152,000 Samples– 1.091– 1.1090.0040.002– 0.93– 0.9390.0040.0023-Way Interaction32,000 Samples– 1.517– 1.5310.0100.003– 1.365– 1.2630.0090.00472,000 Samples– 1.517– 1.5310.0060.001– 1.366– 1.2620.0060.002152,000 Samples– 1.516– 1.5310.0040.001– 1.367– 1.2630.0040.002 Table 4Log10 Bayes factors from Simulation 3a and 3bSimulation 3a: Normal DVSimulation 3b: Binomial DVMeanStandard deviationMeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-Within32,000 Samples3.9363.1520.0100.0912.4726.9700.0100.14072,000 Samples3.9363.1530.0060.0512.4726.9810.0070.093152,000 Samples3.9353.1640.0030.0382.4736.9690.0040.067IV2-Between32,000 Samples3.1985.0770.0100.0953.6483.9210.0110.1072,000 Samples3.1975.0720.0050.073.653.9160.0050.058152,000 Samples3.1975.0860.0030.0383.6493.9200.0040.038IV3-Within32,000 Samples– 0.393– 0.5520.0090.013– 0.1920.1300.0100.02572,000 Samples– 0.393– 0.5530.0060.009– 0.1910.1330.0060.016152,000 Samples– 0.393– 0.5520.0040.004– 0.1920.1320.0040.011Interaction 1 x 232,000 Samples– 1.198– 1.1230.0100.003– 0.943– 0.9820.0110.00372,000 Samples– 1.198– 1.1230.0060.002– 0.943– 0.9820.0060.003152,000 Samples– 1.198– 1.1230.0040.001– 0.943– 0.9820.0040.002Interaction 1 x 332,000 Samples31.94332.9990.0090.40840.28735.8110.0100.41672,000 Samples31.94432.9420.0060.32240.28835.8770.0060.325152,000 Samples31.94432.9700.0040.20540.28835.8350.0040.198Interaction 2 x 332,000 Samples– 1.092– 1.1090.0110.004– 0.930– 0.9400.0100.00472,000 Samples– 1.091– 1.1090.0050.002– 0.930– 0.9400.0060.002152,000 Samples– 1.091– 1.1090.0040.002– 0.93– 0.9390.0040.0023-Way Interaction32,000 Samples– 1.517– 1.5310.0100.003– 1.365– 1.2630.0090.00472,000 Samples– 1.517– 1.5310.0060.001– 1.366– 1.2620.0060.002152,000 Samples– 1.516– 1.5310.0040.001– 1.367– 1.2630.0040.002 Log10 Bayes factors from Simulation 3a and 3b For the more complex experimental design explored here, the SD estimates were comparable to those of the simpler designs in the preceding simulations. Again, the variability of BF estimates from bridge sampling was tolerable throughout. For BFs in support of an existing effect, the variability of Savage–Dickey estimates was substantially larger than that from bridge sampling, but never anywhere near as large as to risk drawing incorrect conclusions. Therefore, we consider this level of variability tolerable. This is supported by the observation that the proportion of decision agreements was again 100%. Once again, increasing the number of MCMC samples had the expected effect of reducing the variability of BF estimates across both estimation methods. For the non-zero effects—the main effects of IV1 and IV2, and the IV1 x IV3 interaction—the Savage–Dickey density ratio not only led to more variable BF estimates than the bridge sampler but also to larger BFs on average. This was also observed in all previous simulations and seems to reflect a general tendency of Savage–Dickey BF estimates to be overestimated when the true BF is large. Simulation 4: Increasing the number of parameters For this simulation, we used the relatively complex design of Simulation 3 and increased the number of to-be-estimated parameters further by simulating data from a larger subject sample (Nsubj = 100 per experimental group). This increases the dimensionality of the parameter space for which the MCMC sampler needs to explore the likelihood by a factor of about 5, and thereby challenges the estimation of BFs even more than in the preceding simulations.11 We ran only the logistic model because the computation time for the linear model is substantially longer. To save computation time, we ran only simulations with 5000 samples per chain (retaining 32,000 samples across eight chains), the smallest number investigated in previous simulations, which should be most diagnostic for estimation problems. We set the effect sizes of all the true effects—the three main effects and the interaction of IV 1 with IV 3—to d = 0.05 with the aim to obtain BFs closer to the ambiguous range, for which precise BF estimates are particularly important. We created 23 simulated data sets, each of which was analyzed ten times. Table 5 shows the results. Table 5 Log10 Bayes factors from Simulation 4MeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-Within0.01100.3480.027IV2-Between– 0.404– 0.4030.340.02IV3-Within0.2870.2990.3640.036Interaction 1 x 2– 1.322– 1.3110.4280.004Interaction 1 x 30.1820.2460.430.034Interaction 2 x 3– 1.328– 1.2980.370.0043-way interaction– 1.399– 1.3040.4860.004 Table 5 Log10 Bayes factors from Simulation 4MeanStandard deviationTermBridge samplingSavage–DickeyBridge samplingSavage–DickeyIV1-Within0.01100.3480.027IV2-Between– 0.404– 0.4030.340.02IV3-Within0.2870.2990.3640.036Interaction 1 x 2– 1.322– 1.3110.4280.004Interaction 1 x 30.1820.2460.430.034Interaction 2 x 3– 1.328– 1.2980.370.0043-way interaction– 1.399– 1.3040.4860.004 Log10 Bayes factors from Simulation 4 Compared to Simulation 3b, the results of Simulation 4 with bridge sampling are much more variable. The BF estimates from the Savage–Dickey estimation were more robust against the increase in subject sample size. Their variability was, if anything, even smaller than in Simulation 3b. When the BF estimates are used as the basis for a decision about a hypothesis, the variability of the bridge-sampling BF estimates in Simulation 4 can influence such decisions: For four effects (the three main effects and the interaction of IV1 and IV3), we found agreement proportions between 90 and 100%. For the Savage–Dickey based BF estimations, the decision agreement was 100% throughout. The opposite effects of the changes we made between Simulations 3b and 4 on the two BF estimation methods show how their variances are sensitive to different simulation settings: The estimates from bridge sampling are less reliable in Simulation 4 because of the increased number of free parameters that the larger number of subjects entails. The estimates from the Savage–Dickey method are more reliable because the data-generating model of Simulation 4 included only small effects and only modest evidence for the presence of effects. Compared to Simulation 3b, the average Savage–Dickey estimates for BFs reflecting true effects are considerably smaller and in better agreement with the bridge sampling estimates. This is likely the result of two differences between these simulations. One is that the true effects were smaller in Simulation 4, yielding smaller BFs by both estimation methods. This implies that zero—the point at which the posterior density is evaluated—never lies in the far tails of the posterior distributions, at which their density is difficult to estimate. We explore this point further in Simulation 5. The other difference between simulations 3b and 4 is that we increased the sample size compared to the previous simulation. According to Bayesian central limit theorem (Bernardo & Smith, 2000), in parametric models, the multivariate posterior distributions will asymptotically be normally distributed. Hence, increasing the sample size will improve the normal approximation to the marginal posterior distribution and reduce potential bias in the estimation of the BF (cf., Morey et al. 2011)—in addition to increasing model complexity. This suggests that the average Savage–Dickey estimates in previous simulations were larger than bridge sampling estimates because the marginal posterior distributions deviated from normality. We explore these results further in Simulation 5. Discussion of Simulations 1–4 Simulations 1–4 have produced several results that will be useful to the practical researcher trying to decide what BF estimation method to choose and how to use it. First, the variability of Savage–Dickey density-ratio estimates increased with the strength of the evidence for an effect. In contrast, the variability of bridge sampling estimates appeared to be constant across a wide range of BF. This result is evident across all simulations, but is clearest in Simulation 2. As a result, Savage–Dickey estimates were more variable than the bridge sampling estimates when there was strong evidence for an effect (Simulations 1, 2, and 3) but comparable (Simulations 1–4) and sometimes less variable when there was weak evidence for, or clear evidence against an effect (Simulations 3 and 4). Critically, the variance of Savage–Dickey estimates was relatively small when the evidence in favor of an effect was modest. Hence, there was little risk of drawing incorrect conclusions due to the variance of the estimates. As expected, the variance of both estimation methods decreased when we increased the MCMC sample size (Simulations 1 and 3), but even modest sample sizes (32,000) yielded surprisingly stable estimates (Simulations 1, 3, and 4). In most simulations, we found that the Savage–Dickey density ratios yielded larger BF estimates than bridge sampling when the evidence for an effect was strong (Simulation 1, 2, and 3), but comparable estimates when there was evidence against an effect (Simulations 1–4). Our findings do not seem to depend on the distribution of the dependent variable or the link function of the hierarchical model (Simulations 1 and 3). Simulation 4 produced two additional results that warrant further investigation. We observed that increasing the number of participants had two effects: The average BF estimates of bridge sampling and of Savage–Dickey density ratio were in better agreement, even when the variability of bridge sampling estimates was increased compared to Simulation 3. We designed Simulation 5 and 6 to better understand these results. We suspect that the average Savage–Dickey estimates in previous simulations were larger than bridge sampling estimates because the marginal posterior distributions deviated from normality and hence yielded biased posterior density estimates. In practice, the risk of biased estimates can be mitigated by visually inspecting the fit of the estimated probability density function to the posterior samples or by using a less assumptive density estimation approach. However, as shown by Morey et al. (2011), the resulting estimates can be more variable and may still be biased when the evidence for an effect is large. We explored the relationship between bias and variance for three commonly used density estimators in Simulation 5. Simulation 5: Density approximation methods for the Savage-Dickey ratio In all previous simulations, we approximated the posterior density at the test value using a normal distribution (Bernardo & Smith, 2000). General-purpose implementations of the Savage–Dickey density ratio in the R packages bayestestr (Makowski et al., 2019) and brms12 (Bürkner, 2025) use more flexible approximations—logspline density (LS) estimates and spline-smoothed kernel (SSK) density estimates (Wetzels et al., 2009). If the posterior distribution is decidedly non-normal, a normal approximation can result in biased BF estimates. The logspline and spline-smoothed kernel density estimates are more flexible to approximate non-normal posterior distributions and can reduce bias in BF estimates. This flexibility, however, comes at the cost of higher variance (i.e., lower reliability). What is more, Morey et al. (2011) report that in approximately normal posterior distributions, the logspline density method can be more biased than the normal approximation—in particular when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}$$\end{document}BF10 is large. This is because the more flexible methods must estimate the shape of the tails of the distribution where the number of MCMC samples decreases rapidly and simplifying assumption must be made (e.g., Kooperberg & Stone, 1991). Hence, in Simulation 5, we compared bias and reliability of the normal approximation to logspline density (LS) and the spline-smoothed kernel (SSK) density estimates at varying tail probabilities of the posterior distribution and MCMC samples sizes. In our previous simulations, we have explored the number of MCMC samples necessary to obtain reliable BF estimates. Owing to the computational resources required by the bridge sampling algorithm, we varied the MCMC sample size in only three steps and across a limited range (32,000–152,000). By focusing on the Savage–Dickey density ratio, in Simulation 5 we were able to examine the effect of the MCMC sample size on the bias and reliability of BF estimates more closely. We varied the MCMC sample size across a larger range (500–500,000) and in ten steps. The potential for bias in the normal approximation to the posterior distribution is arbitrarily large as it is a direct consequence of the discrepancy in the shape of the posterior distribution. It is easy to create conditions under which this method performs poorly. Our goal for this simulation was to further gauge the performance of the more flexible methods and use the normal approximation as a benchmark. We therefore simulated posterior samples from normal distributions, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\left(\mu ;\sigma =1\right)$$\end{document}Nμ;σ=1. As we expected the differences between estimation methods to be particularly noticeable in the tails of the distribution, we varied the mean μμ of the posterior distribution from 0 to increasing positive values, corresponding to increasingly larger true effects and smaller tail probabilities at the test value 0 (i.e., smaller cumulative probability densities for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta \le 0$$\end{document}θ≤0). To put the estimated posterior densities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{{f}_{X}}\left(x=0\right)$$\end{document}fX^x=0 on a scale that is more readily interpretable and consistent with previous simulations, we report Savage–Dickey density ratios assuming a prior density that implies a fair amount of updating: We assumed a rather uninformative normal prior with μ = 0 and σ = 5. To compute the Savage–Dickey density ratio, we divide the prior density at 0 by the posterior density at 0. Thereby, the density of the prior at 0, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f}_{X}\left(\mathrm{x}=0; \upmu =0, \upsigma =5\right)=0.798$$\end{document}fXx=0;μ=0,σ=5=0.798, effectively serves as a scaling constant for the inverse posterior density at 0. Since we chose the prior arbitrarily to scale the posterior densities, the absolute magnitudes of the Bayes factors in Fig. 2 are also arbitrary. We chose posterior means that yield BFs that correspond to boundaries of intervals from popular interpretation guidelines, that is, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}\in \left[{~}^{1}\!\left/ \!{~}_{3}\right., 1, 3, 10, 30, 100, 1000\right]$$\end{document}BF10∈13,1,3,10,30,100,1000. We estimated 1000 probability densities in each condition of the simulation.Fig. 2 Bias and variability of estimated probability densities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{{f}_{\Theta }}\left(\theta =0\right)$$\end{document}fΘ^θ=0 as a function of MCMC sample size and three posterior density estimation methods: Spline-smoothed kernel density, logspline density, and best-fitting normal probability density. Probability densities were estimated for seven normal distributions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\left(\mu ;\sigma =1\right)$$\end{document}Nμ;σ=1 with cumulative probability densities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0;\mu , \sigma \right)$$\end{document}FΘθ=0;μ,σ, where μμ was chosen to yield typical BF category borders when assuming a prior distribution implying a reasonable amount of belief updating (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\left(\mu =0;\sigma =5\right)$$\end{document}Nμ=0;σ=5). The grey curve in the inset represents the prior distribution; the colored curves represent the posterior distributions with shifting means. Violins and boxplots show the distributions of estimates for each MCMC sample size (labeled on the X-axis), and each true probability density (horizontal lines). Axes show log10-scale, i.e., orders of magnitude Bias and variability of estimated probability densities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{{f}_{\Theta }}\left(\theta =0\right)$$\end{document}fΘ^θ=0 as a function of MCMC sample size and three posterior density estimation methods: Spline-smoothed kernel density, logspline density, and best-fitting normal probability density. Probability densities were estimated for seven normal distributions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\left(\mu ;\sigma =1\right)$$\end{document}Nμ;σ=1 with cumulative probability densities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0;\mu , \sigma \right)$$\end{document}FΘθ=0;μ,σ, where μμ was chosen to yield typical BF category borders when assuming a prior distribution implying a reasonable amount of belief updating (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\left(\mu =0;\sigma =5\right)$$\end{document}Nμ=0;σ=5). The grey curve in the inset represents the prior distribution; the colored curves represent the posterior distributions with shifting means. Violins and boxplots show the distributions of estimates for each MCMC sample size (labeled on the X-axis), and each true probability density (horizontal lines). Axes show log10-scale, i.e., orders of magnitude Before we discuss the results, we reiterate that in all simulations Savage–Dickey density ratios were estimated using the analytic prior density, and the posterior density was estimated using MCMC samples from the posterior. In some applications, however, the prior density may not be available analytically, for example, because the test-relevant parameter is calculated from other model parameters. In such instances, the density estimation procedures must also be used to estimate the prior density from MCMC samples. This approach is the default in some general-purpose implementations of the Savage–Dickey density ratio (e.g., Bürkner, 2025; Makowski et al., 2019). The resulting estimator of the BF compounds uncertainty about the prior and posterior densities and will likely be less reliable than our results suggest. Although most of the prior density is typically centered around 0 and estimation of the prior density at 0 should be achievable with relatively high precision, we recommend using the analytical prior whenever available. Results The normal approximation of the posterior distributions produced unbiased and reliable density estimates even for modest MCMC sample sizes, Fig. 2. As expected, the variability of the estimates increased as the posterior tail probability of the test value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0;\mu , \sigma =1\right)$$\end{document}FΘθ=0;μ,σ=1 decreased. Yet, even for our smallest MCMC sample size of 5000 the density estimates were so reliable that all interpretation categories on the BF scale could be distinguished unambiguously. As the MCMC sample size approached 40,000 there was no noteworthy uncertainty in the estimated BFs. As expected, the more flexible density estimation methods produced more variable but also more biased estimates. In both methods, the variance of estimates increased as the posterior tail probability decreased (i.e., the posterior moved farther away from 0) and decreased as MCMC sample size increased. The pattern of bias was more complex. The logspline approximation exhibited an upward bias for moderate posterior tail probabilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)>.010$$\end{document}FΘθ=0>.010, which was negligible across all MCMC sample sizes. For small posterior tail probabilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)<.003$$\end{document}FΘθ=0<.003, estimates were biased downwards. When the number of MCMC samples was small, this bias and the variability of the estimates made it difficult to distinguish between BF interpretation categories. For the most extreme posterior tail probability, a noticeable downward bias persisted even for 300,000 MCMC samples. In our view, bias and reliability of the estimates reached acceptable levels at an MCMC sample size of approximately 40,000 in all conditions. If feasible, we recommend samples sizes of 65,000 or more. For 500,000 MCMC samples, the logspline estimation procedure failed catastrophically. In two simulation runs, logspline density estimates were negative or infinite, and across all runs, the estimates were strongly biased and highly variable. In our experience, it is unusual to draw as many as 500,000 posterior samples. Hence, we do not consider this algorithmic failure a fatal flaw of the method—it performed well for more typical sample sizes. Yet, users of the logspline method should be wary of such algorithmic failures when using many MCMC samples. The spline-smoothed kernel approximation exhibited a downward bias for moderate to small posterior tail probabilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)>.001$$\end{document}FΘθ=0>.001, which was more pronounced than in the logspline method but relatively small across all MCMC sample sizes. For very small posterior tail probabilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)<.0002$$\end{document}FΘθ=0<.0002, estimates were increasingly biased upwards. Compared to logspline estimates, the bias decreased more quickly with the number of MCMC samples.13 Across all conditions, the spline-smoothed kernel estimates were more variable than the logspline estimates. When the number of MCMC samples was small, this bias and variability made distinguishing between BF interpretation categories difficult. In our view, bias and variability of the estimates reached acceptable levels at an MCMC sample size of 65,000 for all conditions. If feasible, we recommend sample sizes of approximately 100,000 or more—approximately 50% more than for the logspline approximation. For moderate posterior tail probabilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)>.01$$\end{document}FΘθ=0>.01, all methods produced relatively reliable estimates with no or negligible bias—even for MCMC sample sizes as small as 5000–10,000. Discussion Our results highlight how parametric assumptions about the posterior distribution—here, the assumption that it is approximately normally distributed—can yield very efficient estimates of posterior densities. This high reliability comes with the risk of a potentially large bias when applied blindly, even though the parametric assumption is inappropriate. Consider the example of a posterior distribution that follows a non-central t distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\left(\mu =2, \nu =3\right)$$\end{document}tμ=2,ν=3—skewed and heavy-tailed compared to a normal distribution. Figure 3 shows that the normal approximation can underestimate the true probability density by a factor of 2.5 or severely overestimate the probability density by a factor of 20 or more. In such situations, more flexible density approximations promise to be less biased at the cost of being more variable. However, our simulation shows that these methods can be very biased when the to-be-estimated density is deep in the tails of the posterior distribution, and the number of MCMC samples is modest. In such cases, spline-smoothed kernel estimates are substantially more variable than logspline estimates, but they can be less biased than logspline estimates when the number of MCMC samples is large. Overall, our results suggest that logspline estimates may be a better default for general-purpose implementations of the Savage–Dickey ratio, because the estimates are more reliable and what bias occurs in the tails of the distribution appears to bias the estimates towards lower BFs—that is, it errs on the side of under- rather than overstating the evidence. However, firm conclusions require additional simulations for a wider range of posterior distributions. We leave these simulations for future research.Fig. 3 Illustration of bias resulting from inappropriately approximating a posterior distribution with a normal distribution. A Probability density distributions of a non-central t distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\left(\mu =2, \nu =3\right)$$\end{document}tμ=2,ν=3 and the normal distribution with the same mean and variance. B Factor (on log-scale) by which the probability density of the normal approximation overestimates the true probability density Illustration of bias resulting from inappropriately approximating a posterior distribution with a normal distribution. A Probability density distributions of a non-central t distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\left(\mu =2, \nu =3\right)$$\end{document}tμ=2,ν=3 and the normal distribution with the same mean and variance. B Factor (on log-scale) by which the probability density of the normal approximation overestimates the true probability density Simulation 6: Varying strength of evidence and model complexity The results of Simulations 2, 3, and 4 suggest that the variance of BF estimates obtained by bridge sampling and by the Savage–Dickey density ratio may be differently affected by the magnitude of the true BF and the number of model parameters: Whereas the Savage–Dickey density ratio is unaffected by the number of model parameters, its variance increases with the evidence for an effect—that is, as the posterior tail probability of the test value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${F}_{\Theta }\left(\theta =0\right)$$\end{document}FΘθ=0 decreases. Conversely, the variability of bridge sampling estimates increases with the number of model parameters, but it is unaffected by the magnitude of the true BF. To confirm these results, we conducted a final simulation, in which we varied these factors directly and independently. In this simulation, we added one more method for computing the Savage–Dickey density ratio: conditional marginal density estimation (CMDE). This method requires that the conditional posterior distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\left(\theta \right|\phi , {\boldsymbol{y}}, {\mathcal{M}}_{1})$$\end{document}pθϕ,y,M1) is known. When this is the case, the estimator can be efficiently sampled during the MCMC simulation and, due to Rao-Blackwellization, is very reliable. It can be considered a best-case scenario for the Savage–Dickey density ratio. Morey et al. (2011) showed that this method is substantially more reliable and less biased than the logspline and normal approximations of the posterior probability density. However, note that the conditional posterior distribution differs across models and is often unknown for even moderately complex models, such as hierarchical generalized linear models. Hence, it is not one of the generally applicable, easy-to-use methods explored in our previous simulations. We include it here for comparison. We also explored whether the reliability of bridge-sampling estimates from the bridge-sampling R package (Gronau et al., 2018) differs across parameter types. Specifically, we tested whether the estimation may be less reliable for models with large correlation matrices, such as hierarchical models with more random effects. We therefore compared the effects of the number of independent, unbounded continuous parameters (e.g., independent means) and of a single large Cholesky-factorized correlation matrix (p. 273, Stan-Development-Team, 2024). For this simulation, we manipulated the true BF by simulating data for a one-sample t test, for which the true BF is available analytically. We simulated six datasets, consisting of 100 observations each, that yield Bayes factors of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}\in \left[{~}^{1}\!\left/ \!{~}_{9}\right.,{~}^{1}\!\left/ \!{~}_{3}\right., 1, 3, 10, 30, 100\right]$$\end{document}BF10∈19,13,1,3,10,30,100 in default JZS t tests (Rouder et al., 2009). The unconstrained model has two parameters: the mean and the standard deviation of the data-generating normal distribution. To explore the effect of the number of model parameters on the BF estimation methods, we introduced additional parameters that contributed nothing to the model predictions; that is, they did not affect the models’ marginal likelihood. These parameters were unrelated to the data that was the subject of the t test, were thus irrelevant to the BF, and were not updated by data—they were sampled from their assigned prior distributions. To achieve this artificial inflation of the dimensionality of the joint parameter distribution, we implemented the null and alternative models in the probabilistic programming language Stan (Stan-Development-Team, 2024) and added sampling statements for parameters that did not factor in the models’ likelihood. Specifically, we sampled values for 3, 435, or 780 parameters from either independent standard normal distributions or a Cholesky-factorized LKJ distribution with η =2η=2 degrees of freedom. The purpose of these irrelevant parameters was to increase the number of model parameters, while holding all other variables constant, such as the true BF, the data informing the model comparison, or the number of true effects in the data-generating process. To keep this simulation at a manageable scale, we opted to use a fixed number of MCMC samples. The results of Simulation 5 suggest that 100,000 samples yield relatively unbiased and stable estimates for all density estimation methods. Following 2000 warm-up samples, we drew 102,000 posterior samples across six chains. We repeated this procedure 30 times for each dataset. We monitored the sampler for divergent transitions, as well as the Bayesian fraction of missing information, whether the sampling algorithm reached the maximum tree depth, and the maximum \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{R}$$\end{document}R^ across all parameters. According to these diagnostics, the sampler converged to the stationary distribution and explored the joint posterior efficiently.14 Different from Simulations 1–4, we used the Warp-III method of the bridge sampler. One attractive property of Warp-III bridge sampling is that it makes more lenient assumptions about the shape of the full posterior distribution. Warp-III bridge sampling is, therefore, an alternative to spline-smoothed kernel and logspline density estimation when the posterior distribution is non-normal. The Warp-III bridge sampling algorithm converged within 3–9 iterations.15 We repeatedly estimated the marginal likelihood 30 times for each of the 30 sets of posterior samples. We compared bias and variance of individual estimates to the average of the 30 repeated estimates. Results Figure 4 shows the distributions of BF-estimates from our simulation. To explore bias and reliability (i.e., variance) separately, we estimated each method’s bias as the ratio of estimated and true Bayes factor (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathrm{B}\mathrm{F}}}_{10}/{\mathrm{BF}}_{10}$$\end{document}BF^10/BF10), and each method’s variance as the standard deviation of estimated log Bayes factors, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{S}\mathrm{D}\left\{\mathrm{log}\left({\widehat{\mathrm{B}\mathrm{F}}}_{10}\right)\right\}$$\end{document}SDlogBF^10 fitting separate (hierarchical) linear models to the estimated log Bayes factors of each condition using brms with default priors suitable for estimation. Figure 5 highlights each method’s bias; Fig. 6 highlights each method’s variance. Overall, the results confirm our conclusions from previous simulations. The different types of added parameters yielded consistent results.Fig. 4 Bayes factors estimated using the Warp-III bridge sampling algorithm in two variants (BS: W3 = 30 individual estimates for each of 30 sets of posterior samples; M(BS: W3) = means across the 30 individual estimates for each of 30 sets) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Violins and boxplots show the distributions of Bayes factor estimates on log10-scale, i.e., orders of magnitudeFig. 5 Proportional bias of Bayes factors estimated using the Warp-III bridge sampling algorithm (BS: W3) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Points represent posterior medians, and error bars 95% credible intervals. Grey points show results for different numbers of added parameters for comparisonFig. 6 Standard deviations (on logarithmic scale) of log Bayes factors estimated using the Warp-III bridge sampling algorithm (BS: W3) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Points represent posterior medians and error bars 95% credible intervals. Grey points show results for different numbers of added parameters for comparison Bayes factors estimated using the Warp-III bridge sampling algorithm in two variants (BS: W3 = 30 individual estimates for each of 30 sets of posterior samples; M(BS: W3) = means across the 30 individual estimates for each of 30 sets) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Violins and boxplots show the distributions of Bayes factor estimates on log10-scale, i.e., orders of magnitude Proportional bias of Bayes factors estimated using the Warp-III bridge sampling algorithm (BS: W3) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Points represent posterior medians, and error bars 95% credible intervals. Grey points show results for different numbers of added parameters for comparison Standard deviations (on logarithmic scale) of log Bayes factors estimated using the Warp-III bridge sampling algorithm (BS: W3) or Savage–Dickey density ratio (S-D) based on four different posterior density estimates: Spline-smoothed kernel density (SSK), logspline density (LS), normal probability density (N), and conditional marginal density estimation (CMDE). Bayes factors were estimated for six datasets with analytic BF varying from 1/9 to 100. To vary the dimensionality of the posterior, we added 3, 435, or 780 parameters that were unrelated to the model likelihood. Points represent posterior medians and error bars 95% credible intervals. Grey points show results for different numbers of added parameters for comparison Bias As shown in Fig. 5, the results for the Savage–Dickey density ratios are generally in line with our findings from Simulation 5. With more than 100,000 samples from the posterior, the bias of all methods was mild. Logspline estimates exhibited the largest bias, underestimating the true Bayes factor of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}=100$$\end{document}BF10=100 by approximately 10%. Estimates from the normal approximation increasingly underestimated the true BF as the evidence for an effect increased. At a true \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}=100$$\end{document}BF10=100, bias was approximately 5%. As in Simulation 5, the pattern of biases was more complex for the more flexible density estimation methods. Logspline estimates were unbiased when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{BF}}_{10}<1$$\end{document}BF10<1, slightly overestimated moderate Bayes factors of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1<{\mathrm{BF}}_{10}<10$$\end{document}1