Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures

Tuan Thai, TrungTin Nguyen, Dat Do, Nhat Ho, Christopher Drovandi · 2025

Read Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures by Tuan Thai, TrungTin Nguyen, Dat Do, Nhat Ho, Christopher Drovandi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13052.txt.

Introduction

Tuan Thai and co-authors tackle model choice for mixture of experts networks, in a 2025 paper. Such models split work among expert parts and are prized for flexibility and speed. They sit inside many top deep networks, yet picking the right count of experts lacks solid theory, since inputs enter both the gates and the experts.

The authors extend a tree diagram tool for mixing measures to the Gaussian gated, Gaussian expert case. Their method estimates the true expert count consistently and hits the best possible accuracy rate for the weights, even when the fitted model has too many parts. Tests on synthetic data recovered the right count well. The trick avoids training many rival models just to compare them.

Catalog record

CREATORTuan Thai, TrungTin Nguyen, Dat Do, Nhat Ho, Christopher Drovandi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE126 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures Tuan Thai, TrungTin Nguyen, Dat Do, Nhat Ho, Christopher Drovandi Abstract Mixture of Experts (MoE) models constitute a widely utilized class of ensemble learning approaches in statistics and machine learning, known for their flexibility and computational efficiency. They have become integral components in numerous state-of-the-art deep neural network architectures, particularly for analyzing heterogeneous data across diverse domains. Despite their practical success, the theoretical understanding of model selection, especially concerning the optimal number of mixture components or experts, remains limited and poses significant challenges. These challenges primarily stem from the inclusion of covariates in both the Gaussian gating functions and expert networks, which introduces intrinsic interactions governed by partial differential equations with respect to their parameters. In this paper, we revisit the concept of dendrograms of mixing measures and introduce a novel extension to Gaussian-gated Gaussian MoE models that enables consistent estimation of the true number of mixture components and achieves the pointwise optimal convergence rate for parameter estimation in overfitted scenarios. Notably, this approach circumvents the need to train and compare a range of models with varying numbers of components, thereby alleviating the computational burden, particularly in high-dimensional or deep neural network settings. Experimental results on synthetic data demonstrate the effectiveness of the proposed method in accurately recovering the number of experts. It outperforms common criteria such as the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood, while achieving optimal convergence rates for parameter estimation and accurately approximating the regression function. † † ⋆ Co-first author, † Corresponding author. 1 Introduction Mixture of experts (MoE) models. MoE models were initially introduced as neural network-based architectures in [ 31 , 33 ] . These conditional mixture density models have gained considerable attention, primarily due to their universal approximation capabilities [ 62 , 48 , 47 , 49 ] , and their favorable convergence properties, as demonstrated in mixture of regression settings [ 11 , 29 ] and various MoE frameworks [ 50 , 32 , 45 , 44 , 43 ] . Notably, these advancements enhance both the approximation and convergence behavior compared to unconditional mixture models, as shown in earlier works [ 21 , 64 , 7 , 61 , 67 , 26 , 27 , 58 , 54 ] . For comprehensive discussions on the theoretical foundations and practical implementations of MoE models across various fields, including biological data analysis, natural language processing, and computer vision, see [ 72 , 46 , 12 , 51 , 6 ] . Moreover, MoE models offer a more expressive alternative to traditional mixture and mixture of regression models [ 16 , 40 ] , primarily due to their ability to model both the expert components and the gating functions as functions of the input covariates. In regression contexts, a commonly adopted formulation consists of Gaussian experts in conjunction with either softmax-based or Gaussian-based gating mechanisms. A representative example is the Gaussian-gated Gaussian mixture of experts (GGMoE) model [ 8 ] , which generalizes several classical approaches, including the affine mixture of experts model [ 71 ] and inverse regression techniques such as those proposed in [ 30 , 37 ] . One limitation of softmax-gated MoE models lies in the complexity of their parameter estimation via the expectation-maximization (EM) algorithm [ 9 ] , where the M-step involves solving an internal optimization subroutine, typically through Newton-Raphson iterations. In contrast, the parameter updates for GGMoE models admit closed-form expressions, leading to more computationally efficient inference procedures. Hence, this work focuses on parameter estimation and model selection within the GGMoE framework, which is particularly effective for modeling complex, nonlinear input-output relationships arising from heterogeneous subpopulations. Related work on model selection in MoE models. Universal approximation theorems establish that, given a sufficiently large number of components, both mixture models and MoEs possess the capacity to approximate broad families of unconditional and conditional probability density functions, respectively, with arbitrary accuracy. Despite this theoretical guarantee, the practical determination of an adequate number of components remains a challenging task in real-world applications. This underscores the importance of principled model selection strategies for MMs and MoEs, a topic that has garnered sustained interest in the statistics and machine learning communities for several decades. For a recent synthesis of developments in this area, see, e.g., [ 4 , 22 ] . In practice, the selection of the number of components in MoEs typically relies on model selection criteria that balance fit and complexity. Commonly employed information criteria include the Akaike information criterion (AIC) [ 1 , 19 ] , the Bayesian information criterion (BIC) [ 65 ] and its adaptations MoE models [ 35 , 13 , 2 , 14 ] , the integrated classification likelihood (ICL) [ 3 , 18 ] , and the extended BIC (eBIC) used in graphical model contexts [ 15 , 42 ] . For dependent data settings, the Sin and White information criterion (SWIC) [ 68 , 70 ] is also utilized. A common limitation of these methods, however, is their asymptotic nature; they lack finite-sample guarantees and may be unreliable in small-sample regimes, rendering their practical use somewhat heuristic. To address these concerns, recent work by [ 57 , 56 , 41 , 52 , 53 , 59 ] has developed non-asymptotic risk bounds in the form of weak oracle inequalities. These results justify model selection via penalized likelihood, provided appropriate lower bounds on the penalties are enforced, and are applicable in high-dimensional regression settings across various MoE formulations, including GGMoE. Another widely used paradigm is Bayesian model selection via marginal likelihood evaluation, which provides a probabilistic framework for selecting the number of components [ 17 , 73 ] . Nonetheless, a key computational challenge persists: existing model selection methods generally require training and evaluating multiple candidate models across different component configurations. This exhaustive search becomes computationally infeasible in high-dimensional regimes or when MoE models are embedded within deep neural networks [ 36 , 66 , 63 , 12 ] . To address this issue, [ 55 ] introduced a computationally efficient Bayesian nonparametric formulation of the GGMoE model that circumvents the need to pre-specify the number of components. This method employs the merge-truncate-merge (MTM) post-processing strategy proposed by [ 23 ] to achieve posterior consistency. Nevertheless, its practical implementation is constrained by the presence of a sensitive tuning parameter, which undermines its robustness and limits its direct applicability to real-world datasets. Main contributions. To address the challenge of model selection in GGMoE models, we revisit the concept of dendrograms of mixing measures introduced by [ 10 ] and propose a novel extension: the dendrogram selection criterion (DSC) algorithm tailored for GGMoE models. This method enables consistent estimation of the true number of mixture components while significantly mitigating the dependence on sensitive tuning parameters, a common limitation in existing approaches, including the GGMoE framework combined with the MTM post-processing procedure [ 55 ] . In addition to its sensitivity to tuning parameters, the MTM approach does not alleviate the fundamental limitation of slow convergence rates in parameter estimation, even after merging. In contrast, the proposed DSC algorithm is theoretically guaranteed to improve the convergence rate of

Read the complete text