Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions
2026
Read Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27791.txt.
Introduction
Don van den Bergh and three colleagues present singular distribution mixtures for Bayesian variable selection, holding rival models in one space. The formulation reproduces the spike-and-slab reading and shares the Metropolis-Hastings acceptance odds of reversible jump Monte Carlo. On a ten-predictor benchmark, both methods recover inclusion odds matching enumeration, while mixtures achieve comparable sample size per second. They show the approach in a sleep-and-memory regression and in factor selection for a partial credit model.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91341 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions Abstract Bayesian variable selection requires sampling from a posterior distribution that combines discrete model indicators with continuously varying parameters, a challenge often addressed through reversible jump Markov chain Monte Carlo (RJMCMC). Despite its generality, RJMCMC is widely regarded as difficult to design and implement correctly. We present mixtures of mutually singular (MoMS) distributions as a transparent alternative in which competing models are represented within a single fixed-dimensional parameter space partitioned into mutually singular subspaces. We show that this formulation reproduces the exact spike-and-slab interpretation of Bayesian variable selection and that, under appropriate constructions, MoMS and RJMCMC share the same Metropolis–Hastings acceptance probability. On a benchmark dataset with ten predictors, both methods recover posterior inclusion probabilities that match full enumeration, while MoMS achieves comparable or superior effective sample size per second relative to a carefully engineered RJMCMC scheme. We further illustrate the approach in a mixed-effects logistic regression for a sleep-and-memory experiment and in factor-loading selection for a multidimensional generalized partial credit model. Together, these results show that Bayesian variable selection can be carried out within standard fixed-dimensional Markov chain Monte Carlo methodology—without regret. \authorsnames [1,2,3,1]D. van den Bergh, M. A. Clyde, A. E. Raftery, M. Marsman \authorsaffiliations Department of Psychology, University of Amsterdam, Department of Statistical Science, Duke University, Department of Statistics, University of Washington \authornote Correspondence concerning this article should be addressed to: Don van den Bergh University of Amsterdam, Psychological Methods Nieuwe Achtergracht 129B PO Box 15906, 1001 NK Amsterdam, The Netherlands E-mail may be sent to [email protected]. Reversible Jump With No Regrets keywords Bayesian variable selection, mixtures of mutually singular distributions, reversible jump 1 Introduction Bayesian statistics is becoming increasingly popular in psychometrics ( zagaria2024bayesian ; PfadtEtAl2025 ) . This popularity stems from the suitability of Bayesian methods for complex psychometric models and their coherent treatment of inferential uncertainty. Uncertainty in psychometric models concerns both the values of model parameters and the structure of the model ( Kaplan_2021 ; ClydeGeorge_2004 ) . For example, researchers may be uncertain about which predictor variables to include in a regression model, which item-factor loadings to include in a factor model, or which edges to include in a graphical model. In each of these cases, every specific selection or configuration of effects corresponds to a distinct statistical model. Model uncertainty especially impacts inference of complex psychometric models, such as multidimensional latent variable models, and graphical models. For example, a Bayesian analysis by HuthEtAl_inpress_evidence of psychometric networks from 126 published studies found substantial uncertainty about which edges to include or exclude. Two common Bayesian approaches to dealing with this type of uncertainty are model comparison, which evaluates the relative plausibility of competing models, and variable selection, which aims to identify effects that have high predictive value. In Bayesian model comparison, model uncertainty is addressed by considering a finite set of competing models. Suppose we consider M\in\mathbb{N} models and are interested in the relative plausibility of a given model \mathcal{M}_{m} ( m=1,\dots,M ) given the observed data y and the prior model probabilities. This relative plausibility is quantified by the posterior model probability: p(\mathcal{M}_{m}\mid y)=\frac{p(y\mid\mathcal{M}_{m})\,p(\mathcal{M}_{m})}{\sum_{m^{\prime}=1}^{M}\,p(y\mid\mathcal{M}_{m^{\prime}})\,p(\mathcal{M}_{m^{\prime}})}. In practice, calculating posterior model probabilities is subject to two limitations. First, model comparison requires evaluation of the marginal likelihood for each model, p(y\mid\mathcal{M}_{m})=\int p(y\mid\theta,\mathcal{M}_{m})\,p(\theta\mid\mathcal{M}_{m})\,\text{d}\theta, where \theta denotes the model-specific parameters. Closed-form expressions for the marginal likelihood are available only in restrictive cases, and for more complex models it must be approximated using numerical methods such as Laplace approximations ( TierneyEtAl_1989 ) , importance sampling ( kass1995bayes ) , bridge sampling ( gronau2017tutorial ; meng1996simulating ) , reciprocal importance sampling ( MetodievEtal2025 ) , or thermodynamic integration ( lartillot2006computing ) ; see llorente2023marginal for an extensive review. Because many of these approaches are computationally intensive, they are feasible only when the number of competing models M is relatively small. Second, even when marginal likelihoods can be computed or approximated, the number of competing models can be prohibitively large. For example, a graphical model with p=10 variables already admits more than 35 trillion distinct network structures, making exhaustive evaluation infeasible even under idealized computational assumptions. In Bayesian variable selection, we consider E\in\mathbb{N} possible effects and define an indicator vector \bm{\gamma}_{m}\in\{0,1\}^{E} of length E , where the e -th element indicates whether effect e is included in model m (i.e., \gamma_{me}=1 ) or excluded (i.e., \gamma_{me}=0 ). In this setting, models \mathcal{M}_{m} are implicitly defined by different configurations of included and excluded effects \bm{\gamma}_{m} ( m=1,\dots,M=2^{E} ). The posterior inclusion probability for an effect e is obtained by aggregating posterior model probabilities, p(\gamma_{e}=1\mid y)=\sum_{m=1}^{M}\,\mathcal{I}(\gamma_{me}=1)\,p(\mathcal{M}_{m}\mid y). Direct evaluation of posterior inclusion probabilities is hindered by the need to compute marginal likelihoods and to evaluate them for all possible models. Bayesian variable selection addresses these limitations by treating the vector of effect indicators \bm{\gamma}_{m} as a parameter of the model and jointly sampling the indicators and model parameters using Markov chain Monte Carlo (MCMC). Rather than enumerating all models and evaluating their marginal likelihoods, posterior model- and inclusion probabilities can be approximated using Monte Carlo averages over the sampled indicator configurations. A major complication in using MCMC to simulate from the posterior distribution of effect indicators and model parameters is that the dimension of the model parameters may change as one moves between models. Standard MCMC algorithms assume that the dimension of the posterior distribution is fixed. Transdimensional MCMC relaxes the fixed dimensionality assumption and can simulate from the posterior distribution of effect indicators and model parameters. By far the most popular such approach is reversible jump MCMC (RJMCMC) proposed by Green_1995 . Green generalized the Metropolis-Hastings algorithm to allow for between-model “jumps” while mapping the parameters of these models onto each other and ensuring that detailed balance is achieved, i.e., the between-model jump is reversible. While RJMCMC allows for changes in parameter dimensions, it is so flexible that its design is considered very complicated. Folklore has it that no one has ever designed and implemented RJMCMC and not regretted it. The practical difficulty of using RJMCMC for variable selection has motivated the development of simpler alternatives that avoid transdimensional moves. One particularly influential alternative is the continuous spike-and-slab prior, originally proposed by GeorgeMcCulloch_1993 . Rather than allowing parameters to enter or leave the model, the original continuous spike-and-slab approach represents each effect as arising from a mixture of two Gaussian distributions: a “slab” distribution with relatively large variance that permits substantial effects, and a “spike” distribution with very small variance that concentrates mass near zero. Because effects are never exactly zero under a continuous spike-and-slab prior, the dimension of the parameter space remains fixed across all configurations, so standard MCMC algorithms can be used. This computational convenience has led to widespread adoption of the approach, including for predictor selection in linear regression ( GeorgeMcCulloch_1993 ) , item–factor selection ( MavridisNtzoufras_2014 ) , and edge selection in both continuous and discrete graphical models ( Wang_2015 ; ParkJinSchweinberger_2022 ; MarsmanEtAl_2022_rbinnet ) . However, this convenience comes at a cost. The performance of continuous spike-and-slab models depends critically on the specification of the spike and slab variances. When these variances are fixed, Gaussian spike-and-slab priors are not model-selection consistent: