Horseshoe Mixtures-of-Experts (HS-MoE)

Nick Polson, Vadim Sokolov · 2026

Read Horseshoe Mixtures-of-Experts (HS-MoE) by Nick Polson, Vadim Sokolov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09043.txt.

Introduction

Nick Polson and Vadim Sokolov give a Bayes frame for nets that pick one expert from a pool for each input. Such nets are called mixtures of experts. One expert may suit math asks, a next may suit code. The pool can grow vast, so most experts should sit out most asks. A firm prior can force that thrift in a smart way.

They use the horseshoe prior, which shrinks weak signs hard but lets strong signs stand. Gates that pick experts learn from data as they go. Their learn rule tracks just key sums as new data flows in. A reader who builds large nets gets a sparse pick plan with doubt built in.

Catalog record

CREATORNick Polson, Vadim Sokolov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-09043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Horseshoe Mixtures-of-Experts (HS-MoE) Nick Polson, Vadim Sokolov Abstract Horseshoe mixtures-of-experts (HS-MoE) models provide a Bayesian framework for sparse expert selection in mixture-of-experts architectures. We combine the horseshoe prior’s adaptive global-local shrinkage with input-dependent gating, yielding data-adaptive sparsity in expert usage. Our primary methodological contribution is a particle learning algorithm for sequential inference, in which the filter is propagated forward in time while tracking only sufficient statistics. We also discuss how HS-MoE relates to modern mixture-of-experts layers in large language models, which are deployed under extreme sparsity constraints (e.g., activating a small number of experts per token out of a large pool). Keywords: Mixture-of-Experts, Horseshoe Prior, Particle Learning, Sparse LLMs, Bayesian Inference 1 Introduction Mixture-of-Experts (MoE) models provide a powerful framework for combining multiple specialized models through input-dependent gating [ Jacobs et al. 1991 , Jordan and Jacobs 1994 ] . The foundational model represents the predictive distribution as: p(Y\mid X,\phi)=\sum_{k=1}^{K}g_{k}(X;\phi)\,f_{k}(Y\mid X;\theta_{k}), (1) where g_{k}(X;\phi) is the gating function assigning input-dependent weights to expert k , and \theta_{k} parameterizes the k -th expert. Unlike standard finite mixtures with fixed weights, the input-dependent gating enables adaptive partitioning of the input space [ Titterington et al. 1985 ] . Jiang and Tanner 1999 established theoretical foundations for hierarchical MoE, including conditions for identifiability and convergence rates. Bayesian approaches to MoE inference were developed by Peng et al. 1996 , who introduced MCMC methods for posterior computation in MoE and hierarchical MoE models. Recent work has revived MoE in large language models. Sparsely gated MoE layers [ Shazeer et al. 2017 ] scale model capacity while keeping per-token computation tractable. The Switch Transformer [ Fedus et al. 2022 ] simplified routing to a single expert per token. Modern architectures such as Mixtral [ Jiang et al. 2024 ] and DeepSeek-V3 [ Liu et al. 2024 ] operate at high sparsity, activating only a small number of experts per token from a large pool. 1.1 Connections to Previous Work Bayesian treatments of mixtures-of-experts date back to Peng et al. 1996 , who developed MCMC methods enabling uncertainty quantification over expert allocations, while Jiang and Tanner 1999 established identifiability conditions and convergence rates for hierarchical MoE. The connection between MoE-like switching structures and state-space modeling was explored by Stroud et al. 2003 through MCMC algorithms for nonlinear models with state-dependent variances. Our sparsity mechanism builds on the horseshoe prior [ Carvalho et al. 2010b ] and its global-local shrinkage theory [ Polson and Scott 2012b ] , with posterior concentration results for nearly-black signals [ van der Pas et al. 2014 ] and horseshoe-like extensions for subset selection [ Bhadra et al. 2021 ] . For inference, we build on particle learning for general mixtures [ Carvalho et al. 2010a ] and interacting particle system theory [ Johannes et al. 2008 ] . From the perspective of modern large language models, sparse MoE layers rely on deterministic top- k routing [ Shazeer et al. 2017 , Fedus et al. 2022 ] with additional objectives for load balancing and diversity. Our contribution is a Bayesian routing alternative that induces sparsity through shrinkage and provides uncertainty quantification and online updating. Finally, approximation theory [ Zeevi et al. 1998 ] and recent generalization bounds for sparse MoE [ Zhao et al. 2024 ] provide statistical context for why sparsity can improve performance when k\ll K . 1.2 Contributions This paper introduces Horseshoe Mixture-of-Experts (HS-MoE), combining horseshoe priors [ Carvalho et al. 2010b ] with MoE for adaptive sparse expert selection. Our contributions are a hierarchical Bayesian formulation integrating horseshoe priors with MoE, a particle learning algorithm for sequential inference based on sufficient statistics, theoretical connections to approximation and generalization bounds, and an interpretation of HS-MoE as a Bayesian router compatible with modern transformer MoE layers. The rest of the paper is outlined as follows. Section 2 presents the HS-MoE model formulation and the horseshoe prior. Section 3 describes the particle learning algorithm for sequential inference. Section 4 details applications to transformer architectures and streaming data. Section 5 provides theoretical results on approximation and generalization. Section 6 concludes with a discussion and comparison of routing approaches. 2 Horseshoe Mixture of Experts Let X\in\mathbb{R}^{d} denote the input and y the response. The MoE model with K experts is: p(Y\mid X,\phi)=\sum_{k=1}^{K}g_{k}(X;\phi)\,f_{k}(Y\mid X;\theta_{k}), (2) where \Theta=\{\phi,\theta_{1},\ldots,\theta_{K}\} collects all parameters. The gating function uses the softmax form: g_{k}(X;\phi)=\frac{\exp(\phi_{k}^{\top}X)}{\sum_{l=1}^{K}\exp(\phi_{l}^{\top}X)}. (3) We introduce latent allocations z_{i}\in\{1,\ldots,K\} so that the model becomes conditionally independent given z_{1:n} : \displaystyle y_{i}\mid X_{i},z_{i}=k,\Theta \displaystyle\sim f_{k}(\,\cdot\mid X_{i};\theta_{k}), (4) \displaystyle P(z_{i}=k\mid X_{i},\phi) \displaystyle=g_{k}(X_{i};\phi), (5) where z_{i}\in\{1,\ldots,K\} is the latent expert assignment. To make the paper self-contained and implementable, we focus on the canonical MoE regression setting with Gaussian linear experts: y_{i}\mid z_{i}=k,X_{i},\beta_{k},\sigma_{k}^{2}\sim\mathcal{N}(X_{i}^{\top}\beta_{k},\sigma_{k}^{2}),\qquad k=1,\ldots,K, (6) where \beta_{k}\in\mathbb{R}^{d} and \sigma_{k}^{2}>0 . Extensions to generalized linear experts can be handled by the same variance-mean mixture machinery (Section 3 , § 3.8 ). For each expert we use the Normal-inverse-gamma prior, \displaystyle\beta_{k}\mid\sigma_{k}^{2} \displaystyle\sim\mathcal{N}(m_{0},\sigma_{k}^{2}V_{0}), (7) \displaystyle\sigma_{k}^{2} \displaystyle\sim\text{IG}(a_{0},b_{0}), (8) with V_{0}\in\mathbb{R}^{d\times d} positive definite. Throughout, \text{IG}(a,b) denotes the inverse-gamma distribution with density p(x)\propto x^{-(a+1)}\exp(-b/x) on x>0 (shape a and scale b for 1/x ). This yields closed-form posterior and posterior-predictive updates required by particle learning. The softmax gate in ( 3 ) is over-parameterized since adding a common offset to all \phi_{k} leaves g_{k} unchanged; a standard identifiability constraint is \phi_{K}=0 (baseline class). In the inference section we instead adopt a stick-breaking logistic gate (Section 3.4 ), which is a convenient alternative categorical parameterization that admits conditionally Gaussian updates under Pólya–Gamma augmentation. 2.1 Horseshoe Prior for Sparse Expert Selection The horseshoe prior [ Carvalho et al. 2010b , Polson and Scott 2012b ] induces sparsity through local-global shrinkage. For each gating coefficient \phi_{k,j} (expert k , feature j ) we set \displaystyle\phi_{k,j}\mid\lambda_{k,j},\tau \displaystyle\sim\mathcal{N}(0,\tau^{2}\lambda_{k,j}^{2}),\quad k=1,\ldots,K-1,\ \ j=1,\ldots,d, (9) \displaystyle\lambda_{k,j} \displaystyle\sim\mathcal{C}^{+}(0,1), (10) \displaystyle\tau \displaystyle\sim\mathcal{C}^{+}(0,1), (11) where \mathcal{C}^{+}(0,1) is the half-Cauchy distribution. The local shrinkage \lambda_{k} allows individual coefficients to escape shrinkage when supported by data, while global shrinkage \tau adapts to overall sparsity. For computational tractability, we use a scale-mixture representation. One convenient form is the inverse-gamma mixture (written here for the local scales): \lambda_{k,j}^{2}\mid\nu_{k,j}\sim\text{IG}\!\left(\tfrac{1}{2},\tfrac{1}{\nu_{k,j}}\right),\quad\nu_{k,j}\sim\text{IG}\!\left(\tfrac{1}{2},1\right), (12) and similarly for \tau^{2} (with its own auxiliary variable). Conditional on (\tau^{2},\{\lambda_{k,j}^{2}\}) , the prior on \phi_{k} is Gaussian with diagonal covariance \tau^{2}\operatorname{diag}(\lambda_{k,1}^{2},\ldots,\lambda_{k,d}^{2}) . 3 Inference via Particle Learning We develop particle learning (PL) for sequential Bayesian inference in HS-MoE, following Carvalho et al. 2010a . This approach is distinguished by its efficiency in tracking only the essential state vector and by resampling according to predictive probability. This section proceeds from the general resample–propagate–allocate structure for mixture models (Section 3.2) to the conjugate ingredients required for a practical implementation. We provide closed-form predictive weights and recursive sufficient-statistic updates for Gaussian linear experts (Section 3.3 ), and then introduce a categorical gating parameterization (stick-breaking) whose conditional posteriors are Gaussian under Pólya–Gamma augmentation (Section 3.4 ). These components yield a complete particle-learning recursion (Section 3.5 ). Finally, we indicate how Polson and Scott 2012a extends the approach beyond Gaussian experts via variance–mean mixture augmentation (Section 3.8 ). 3.1 Particle Filtering as Interacting Particle System A particle filter is a stochastic process run forward in time with marginals equal to the sequence of conditional state posteriors. Let

Read the complete text