Repulsive mixtures via the sparsity-inducing partition prior

Alexander Mozdzen, Timothy Wertz, Maria De Iorio, Andrea Cremaschi, Gregor Kastner, Johan Eriksson · 2025

Read Repulsive mixtures via the sparsity-inducing partition prior by Alexander Mozdzen, Timothy Wertz, Maria De Iorio, Andrea Cremaschi, Gregor Kastner, Johan Eriksson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25860.txt.

Introduction

Alexander Mozdzen and co-authors craft a prior that keeps mixture model parts from crowding each other. Mixture models split data into hidden groups. Standard priors let groups pile up with near-equal weights, which blurs the split. Their 2025 paper builds on a broadened Dirichlet law called the Selberg Dirichlet. A push-apart term in it favors a few heavy groups.

The team names the build the sparsity-inducing partition prior. They trace its traits and set it against the standard prior. Tests run on fake data and on a health set covering children's body mass and eating habits. A custom sampling method keeps the math fast. The prior suits cases where a lean set of real groups is the goal.

Catalog record

CREATORAlexander Mozdzen, Timothy Wertz, Maria De Iorio, Andrea Cremaschi, Gregor Kastner, Johan Eriksson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25860

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Repulsive mixtures via the sparsity-inducing partition prior Alexander Mozdzen, Timothy Wertz, Maria De Iorio, Andrea Cremaschi, Gregor Kastner, Johan Eriksson Abstract We introduce a novel prior distribution for modelling the weights in mixture models based on a generalisation of the Dirichlet distribution, the Selberg Dirichlet distribution. This distribution contains a repulsive term, which naturally penalises values that lie close to each other on the simplex, thus encouraging few dominating clusters. The repulsive behaviour induces additional sparsity on the number of components. We refer to this construction as sparsity-inducing partition (SIP) prior. By highlighting differences with the conventional Dirichlet distribution, we present relevant properties of the SIP prior and demonstrate their implications across a variety of mixture models, including finite mixtures with a fixed or random number of components, as well as repulsive mixtures. We propose an efficient posterior sampling algorithm and validate our model through an extensive simulation study as well as an application to a biomedical dataset describing children’s Body Mass Index and eating behaviour. Keywords: Bayesian nonparametrics, hierarchical modelling, overfitted mixtures, model-based clustering, mixture models, repulsive prior 1 Introduction Clustering is a fundamental statistical technique used to uncover underlying structures within datasets. Among the various approaches, mixture models have gained particular prominence owing to their flexibility in representing data as originating from multiple latent sub-populations ( Banfield and Raftery 1993 ; Bensmail et al. 1997 ; McLachlan and Peel 2000 ; Frühwirth-Schnatter 2006 , see, for example,) . In a mixture model, observations are assumed to arise from one of M groups, each group being typically modelled by a distribution from a parametric family. The contribution of each group is referred to as a component of the mixture, and is weighted by the relative frequency of the group in the population. This provides a conceptually simple way of relaxing distributional assumptions and flexibly approximating distributions poorly captured by standard parametric families. Moreover, it provides a framework by which observations may be clustered together for purposes of discrimination or classification. The distribution of a set of N observations \bm{y}=(\bm{y}_{1},\dots,\bm{y}_{N}) in a finite mixture model with M components is: \displaystyle f_{y}(\bm{y}\mid M,\bm{w},\bm{\theta})=\prod_{i=1}^{N}\sum_{m=1}^{M}w_{m}f\left(\bm{y}_{i}\mid\bm{\theta}_{m}\right) (1) where \bm{w}=(w_{1},\dots,w_{M}) denotes the weights, with \sum_{m=1}^{M}w_{m}=1 , 0\leq w_{m}\leq 1 , while \bm{\theta}=\left(\bm{\theta}_{1},\dots,\bm{\theta}_{M}\right) denotes the array of component-specific parameter vectors. In representing underlying clusters as components of the mixture, information about cluster sizes, shapes and locations can be determined through the estimation of \bm{w} and \bm{\theta} . In this work, we focus on the prior distribution for the mixture weights exploring an extension of the Dirichlet distribution, the Selberg Dirichlet distribution. This prior influences the distribution of the number of clusters in the sample by incorporating a repulsive term to encourage dissimilarity among the weights, thereby inducing sparsity. We refer to this as the sparsity-inducing partition (SIP) prior. While a larger number of mixture components can potentially improve density estimation, uncovering underlying clusters is a nuanced problem without a ready answer ( Fraley and Raftery 1998 ) . Different approaches exist, from fixing M to a pre-specified value ( Frühwirth-Schnatter 2006 ; Bensmail et al. 1997 ) , placing a prior on it ( Miller and Harrison 2018 ; Nobile and Fearnside 2007 ; Richardson and Green 1997 ) , or setting the value deliberately high and letting small concentration parameters in the Dirichlet prior empty redundant components ( Rousseau and Mengersen 2011 ; Malsiner-Walli et al. 2016 ) . In the case of a random M , the novel method leads to a shrinkage of the number of clusters, eliminating redundant clusters. 2 Mixtures and repulsive mixtures In the context of mixture modelling, it is important to highlight the distinction between the number of components and the number of clusters ( Argiento and De Iorio 2022 ) . The number of components M refers to the number of possible clusters and corresponds to the data generating process, while the number of clusters is the number of allocated components, i.e., components to which at least an observation has been assigned. Such distinction is important, in particular when devising computational strategies. The mixture model in ( 1 ) can be expressed hierarchically through the allocation vector \bm{c}=(c_{1},\dots,c_{N}) , which indicates the component assignment for each observation: \displaystyle\bm{y}_{i}\mid c_{i},\bm{\theta}_{c_{i}} \displaystyle\stackrel{{\scriptstyle\rm ind}}{{\sim}}f\left(\bm{y}_{i}\mid\bm{\theta}_{c_{i}}\right),\quad i=1,\ldots,N (2) \displaystyle\bm{\theta}_{m} \displaystyle\stackrel{{\scriptstyle\rm iid}}{{\sim}}P_{\theta},\quad m=1,\ldots,M \displaystyle c_{1},\dots,c_{N}\mid\bm{w} \displaystyle\stackrel{{\scriptstyle\rm iid}}{{\sim}}\mathcal{C}\left(1,\bm{w}\right) \displaystyle\bm{w} \displaystyle\sim Dir(\alpha) \displaystyle M \displaystyle\sim p_{M} where f is usually a parametric distribution, P_{\theta} a prior measure for the location parameters \bm{\theta}_{m} on \Theta\in\mathcal{R}^{p} , \mathcal{C}\left(1,\bm{w}\right) the categorical distribution with probability vector \bm{w} , Dir(\alpha) is the symmetric Dirichlet distribution with parameter \alpha>0 , and p_{M} a probability mass function on \{1,2,...\} . The Bayesian literature suggests various approaches for dealing with the number of components M . The three main approaches are: (i) setting M=\infty leading to a nonparametric mixture, (ii) keeping M fixed and choose the number of components according to a model choice criterion, (iii) treating M as random and an object of posterior inference. Here we follow the last approach, allowing for a fully Bayesian treatment. Possible choices for p_{M} include an improper uniform prior ( Richardson and Green 1997 ) , or a beta-negative-binomial distribution introduced by Frühwirth-Schnatter et al. 2021 . Historically, sampling from the posterior distribution of such models requires the implementation of labour-intensive reversible jump Markov chain Monte Carlo (RJMCMC) methods ( Green 1995 ; Richardson and Green 1997 ) , which adaptively alter the dimensionality of the parameter space to account for different values of M . Recent developments bridge the trans-dimensional gap using an augmented scheme involving unnormalised weights ( Argiento and De Iorio 2022 ) , or a birth-and-death step ( Cremaschi et al. 2024 ) , which we employ for posterior inference (see Section 3 .) Due to its support on the simplex and conjugacy in a Bayesian framework, the Dirichlet distribution has been the standard choice as a prior for the component weights \bm{w} . It is parametrised by a vector of concentration parameters, with small values shifting the probability mass to the boundaries of the simplex and larger values favouring similar values for the weights by concentrating the probability mass to the centre. Argiento and De Iorio 2022 extend the class of possible prior distributions by introducing a constructive definition of the mixture weights through the normalisation of a finite point process. Recently, Page et al. 2023 proposed a novel asymmetric Dirichlet prior, which allows the introduction of prior information on the number of clusters in a direct way. An important aspect of the model specification in ( 3.4 ) is the assumption of independent and identically distributed (i.i.d.) component parameters \bm{\theta}_{m} , motivated also by computational convenience. Such a prior poses no restrictions on the proximity of the cluster centres, which can lead to the creation of multiple overlapping clusters, possibly introducing redundancy. If the primary objective of the analysis is density estimation, such behaviour can be desirable and beneficial. If, on the other hand, the aim is to investigate the underlying cluster structure, the presence of redundant or overlapping clusters can hinder meaningful interpretation. To mitigate this issue, repulsive mixtures have been proposed with the goal of favouring well-separated component locations. In the Bayesian paradigm, this can be achieved by

Read the complete text