Modifications of the BIC for order selection in finite mixture models
Hien Duy Nguyen, TrungTin Nguyen · 2025
Read Modifications of the BIC for order selection in finite mixture models by Hien Duy Nguyen, TrungTin Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20124.txt.
Introduction
Blend models, which cast data as a mix of hidden groups, force a basic choice: how many groups to use. Hien Nguyen and TrungTin Nguyen tweak a classic score for that choice in 2025. The score was long known to pick the right count, but only under stiff demands on the data, such as smooth shapes and rich spread.
Their two new scores weigh the classic penalty by a tiny log factor that barely moves it in practice. That small shift buys the same right-pick promise under far weaker demands. When the true model lies outside the set on offer, the scores still settle on the best near pick. The pair also show two hard limits: no smallest penalty works for all, and right picks can clash with other goals.
Catalog record
| CREATOR | Hien Duy Nguyen, TrungTin Nguyen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20124
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modifications of the BIC for order selection in finite mixture models Hien Duy Nguyen, TrungTin Nguyen Abstract Finite mixture models are ubiquitous tools in modern statistical modeling, and a frequently encountered problem that arises in their implementation is the choice of model order. In Keribin, 2000 , the frequently used Bayesian information criterion (BIC) was proved to provide consistent order estimation in the mixture model setting. However, the result requires particularly strong model regularity, including the existence of higher moments and higher derivatives of the component density function. We introduce the ν \nu -BIC and ϵ \epsilon -BIC, which modifies the BIC by weighting the penalty by a negligibly small logarithmic factors that are immaterial in practice. We prove that the minor modification enables consistency guarantees under weaker conditions, particularly without differentiability and with minimal moment assumptions. We demonstrate how our theory apply to obtaining order selection consistency for Gaussian mixtures, non-differentiable Laplace mixtures, and mixtures of regression models. 1 Department of Mathematics and Physical Science, La Trobe University, Melbourne, Australia. 2 Institute of Mathematics for Industry, Kyushu University, Fukuoka, Japan. 3 School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia. Keywords: Bayesian information criterion; finite mixture models; model selection; order selection 1 Introduction Finite mixtures are a natural and ubiquitous class of models for modeling heterogeneous populations consisting of distinct subpopulations. Multiple volumes have been written regarding the application, analysis, and implementation of such models, including the volumes of Titterington et al., 1985 , Lindsay, 1995 , Peel & McLachlan, 2000a , Fruhwirth-Schnatter et al., 2019 , Ng et al., 2019 , Chen, 2023 , and Yao & Xiang, 2024 , among many others. When conducting mixture modeling, a frequently required task is that of order selection. In this text, we will take an information criteria (IC) approach following the tradition of Akaike, 1974 as espoused in the texts of Anderson & Burnham, 2004 , Claeskens & Hjort, 2008 , and Konishi & Kitagawa, 2008 . We characterize the task of order selection and the IC method as follows. Let \left(\Omega,{\cal A},\text{P}\right) be a probability space with expectation operator E and suppose that X:\Omega\rightarrow\mathbb{X} is a random map with density function f_{0}:\mathbb{X}\to\mathbb{R}_{>0} with respect to dominant measure \mathfrak{m} , in the sense that \text{P}\left(X\left(\omega\right)\in\mathbb{A}\right)=\int_{\mathbb{A}}f_{0}\text{d}\mathfrak{m} , for each \mathbb{A}\in{\cal B}\left(\mathbb{X}\right) (the Borel \sigma -algebra of \mathbb{X} ). Let \phi:\mathbb{X}\times\mathbb{T}\to\mathbb{R}_{>0} be a probability density function (PDF) dependent on parameter \theta=\left(\vartheta_{1},\dots,\vartheta_{m}\right)\in\mathbb{T}\subset\mathbb{R}^{m} in the sense that \int_{\mathbb{X}}\phi\left(x;\theta\right)\mathfrak{m}\left(\text{d}x\right)=1 , for each \theta , and define the k -component mixture of \phi -densities as {\cal M}_{k}^{\phi}=\left\{\mathbb{X}\ni x\mapsto f_{k}\left(x;\psi_{k}\right)=\sum_{z=1}^{k}\pi_{z}\phi\left(x;\theta_{z}\right):\theta_{z}\in\mathbb{T},\pi_{z}\in\left[0,1\right],\sum_{z=1}^{k}\pi_{z}=1,z\in\left[k\right]\right\}\text{,} where \left[k\right]=\left\{1,\dots,k\right\} and \psi_{k}=\left(\pi_{1},\dots,\pi_{k},\theta_{1},\dots,\theta_{k}\right)\in\mathbb{S}_{k}\subset\mathbb{R}^{\left(1+m\right)k} (the parameter space of densities in {\cal M}_{k}^{\phi} ). Here, \phi is referred to as the component PDF. We suppose that f_{0}\in{\cal M}_{k_{0}}^{\phi} but f_{0}\notin{\cal M}_{k_{0}-1}^{\phi} for some k_{0}\in\left[\bar{k}\right] , and some sufficiently large \bar{k}\in\mathbb{N} . Here, the extra stipulation that f_{0}\notin{\cal M}_{k_{0}-1}^{\phi} is required because \left({\cal M}_{k}^{\phi}\right)_{k\in\left[\bar{k}\right]} is a nested sequence in the sense that {\cal M}_{k}^{\phi}\subset{\cal M}_{k+1}^{\phi} , and thus if f_{0}\in{\cal M}_{k_{0}}^{\phi} , it also holds that f_{0}\in{\cal M}_{k_{0}+1}^{\phi} , therefore the condition alone does not uniquely define k_{0} , which we refer to as the true order of f_{0} . We may interpret k_{0} as the order of the smallest (or most parsimonious) model {\cal M}_{k}^{\phi} that contains f_{0} . Let \ell_{k}\left(\psi_{k}\right)=-\text{E}\left\{\log f_{k}\left(X;\psi_{k}\right)\right\} denote the average negative log-density of f_{k}\left(\cdot;\psi_{k}\right)\in{\cal M}_{k}^{\phi} , for each k\in\left[\bar{k}\right] . Then, by virtue of the divergence property of the Kullback–Leibler divergence (cf. e.g, Csiszár, 1995 ), it also holds that k_{0}=\min_{k}\underset{k\in\left[\bar{k}\right]}{\arg\min}\left\{\min_{\psi_{k}\in\mathbb{S}_{k}}\ell_{k}\left(\psi_{k}\right)\right\}\text{,} assuming that the minimum is well-defined in each case, which is a useful characterization that we will exploit in the sequel. Suppose that we observe a data set of n independent and identically distributed (IID) replicates of X : \mathbf{X}_{n}=\left(X_{1},\dots,X_{n}\right)\subset\mathbb{X} . Then, the problem of order selection can be characterized as the construction of an estimator \hat{k}_{n}=\hat{k}\left(\mathbf{X}_{n}\right) of k_{0} with desirable properties. In particular, we want \hat{k}_{n} to be consistent in the sense that \lim_{n\to\infty}\text{P}\left(\hat{k}_{n}=k_{0}\right)=1\text{.} If we let \ell_{k,n}\left(\psi_{k}\right)=-n^{-1}\sum_{i=1}^{n}\log f_{k}\left(X_{i};\psi_{k}\right) denote the empirical average negative log-likelihood of f_{k}\left(\cdot;\psi_{k}\right) , the natural estimator of \ell_{k}\left(\psi_{k}\right) , then the method of IC suggests that we construct the estimator of k_{0} in the form \hat{k}_{n}=\min_{k}\underset{k\in\left[\bar{k}\right]}{\arg\min}\left\{\min_{\psi_{k}\in\mathbb{S}_{k}}\ell_{k,n}\left(\psi_{k}\right)+\text{pen}_{k,n}\right\}\text{,} for some appropriately chosen penalties \text{pen}_{k,n} . Here, we may recognize \min_{\psi_{k}\in\mathbb{S}_{k}}\ell_{k,n}\left(\psi_{k}\right) as the negative maximum likelihood for the mixture model of order k . We can easily identify this approach with some traditional method such as the Akaike information criterion (AIC; Akaike, 1974 ) and the Bayesian information criterion (BIC; Schwarz, 1978 ) by taking \text{pen}_{k,n}^{\text{AIC}}=\text{dim}\left(\mathbb{S}_{k}\right)n^{-1} and \text{pen}_{k,n}=\text{dim}\left(\mathbb{S}_{k}\right)\left(2n\right)^{-1}\log n , respectively, where \text{dim}\left(\mathbb{S}\right) is the vector space dimension of \mathbb{S} . Let \text{Ln}\left(n\right)=\log\left(e\vee n\right) the logarithm function truncated to 1 , from below, where a\vee b=\max\left\{a,b\right\} , and denote its \nu -fold composition by \text{Ln}^{\circ\nu}\left(n\right) (e.g., \text{Ln}^{\circ 3}\left(n\right)=\text{Ln}\circ\text{Ln}\circ\text{Ln}\left(n\right) ). In this work, we introduce the \nu -BIC and \epsilon -BIC families of penalties of the forms \text{pen}_{k,n}^{\nu}=\alpha\left(k\right)n^{-1}\text{Ln}^{\circ\nu}\left(n\right)\log n and \text{pen}_{k,n}^{\epsilon}=\alpha\left(k\right)n^{-1}\left(\log n\right)^{1+\epsilon} , for any choice of \epsilon>0 and \nu\in\mathbb{N} , and for \alpha:\mathbb{N}\to\mathbb{R}_{>0} strictly increasing, where the choice of \alpha\left(k\right)=\text{dim}\left(\mathbb{S}_{k}\right)=\left(m+1\right)k/2 can be directly compared to the BIC. We shall demonstrate that this modest modification to the BIC is consistent under weaker assumptions than those required for the BIC, as we shall elaborate upon in the sequel. For regular models, order selection consistency of the BIC was proved in the works of Nishii, 1988 , Vuong, 1989 , and Sin & White, 1996 . Unfortunately, the mixture model setting is not regular and thus alternative proofs of consistency are required. A prime and often cited example of such a result is that of Keribin, 2000 who establish the consistency of the BIC for mixture models under strong assumptions. Namely, among other things, the proof requires that the component density have up to five derivatives and that all of these higher derivatives have finite third moments. A remarkable result of Gassiat & Van Handel, 2012 (see also Gassiat, 2018 , Sec. 4.3 ) shows that the BIC is order consistent for choosing between models when \bar{k}=\infty , albeit at the price of much stronger assumptions. In particular, even when restricted to the class of location mixtures, the proof requires that \phi has up to three derivatives and that all derivatives up to second order have finite third moments, while third order derivatives have second moments. Beyond the BIC, Drton & Plummer, 2017 proposed the so-called singular BIC (sBIC) as an approach that operationalizes the expansion n\ell_{k,n}\left(\psi_{k}\right)+\lambda_{k}\left(f_{0}\right)\log n+\left\{m_{k}\left(f_{0}\right)-1\right\}\log\log n+O_{\text{P}}\left(1\right) of the so-called Bayes free energy of Watanabe, 2013 , related to the widely-applicable BIC (WBIC), where the coefficients \left(\lambda_{k}\left(f_{0}\right),m_{k}\left(f_{0}\right)\right)_{k\in\left[\bar{k}\right]} are dependent on the true and unknown model f_{0} . As noted in Chen, 2023 , the specification of the sBIC arises as a solution to a nonlinear system of stochastic equations making it challenging to faithfully implement in practice. In Baudry, 2015 , the log-likelihood is replaced by the so-called complete data log-likelihood and a consistency result is obtained with the usual BIC-form penalty. However, the proof makes the requirement that the limiting objective functionals of all order k\in\left[\bar{k}\right] have positive definite Hessian, which is not possible in the context of mixture models, as typically if k_{0}<k , then the minimizers of any risk over {\cal M}_{k}^{\phi} will have connected components.