Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models TrungKhang Tran, TrungTin Nguyen, Md Abul Bashar, Nhat Ho, Richi Nayak, Christopher Drovandi, TrungKhang Tran, TrungTin Nguyen, Md Abul Bashar, Nhat Ho, Richi Nayak, Christopher Drovandi Abstract Mixture-of-Experts (MoE) architectures combine specialized predictors through a learned gate and are effective across regression and classification, but for classification with softmax multinomial-logistic gating, rigorous guarantees for stable maximum-likelihood training and principled model selection remain limited. We address both issues in the full-data (batch) regime. First, we derive a batch minorization-maximization (MM) algorithm for softmax-gated multinomial-logistic MoE using an explicit quadratic minorizer, yielding coordinate-wise closed-form updates that guarantee monotone ascent of the objective and global convergence to a stationary point (in the standard MM sense), avoiding approximate M-steps common in EM-type implementations. Second, we prove finite-sample rates for conditional density estimation and parameter recovery, and we adapt dendrograms of mixing measures to the classification setting to obtain a sweep-free selector of the number of experts that achieves near-parametric optimal rates after merging redundant fitted atoms. Experiments on biological protein–protein interaction prediction validate the full pipeline, delivering improved accuracy and better-calibrated probabilities than strong statistical and machine-learning baselines. † † affiliationnotice: * Co-first Author. . 1 Introduction 1.1 Mixture of Experts Models Mixture of experts (MoE) models ( 42 ; 45 ) represent heterogeneous predictor–response relationships by combining a gating model with multiple expert models, both depending on the input. The gate assigns input-dependent weights to experts, enabling conditional computation ( 2 ; 11 ) and inducing a data-adaptive partition of the input domain. In classification, softmax-gated multinomial-logistic MoE (SGMLMoE) ( 10 ; 100 ; 41 ; 87 ; 70 ) offer calibrated multiclass probabilities while retaining the conditional-computation benefits of gating. MoE theory is supported by approximation and estimation guarantees in several regimes, including classical mixtures ( 32 ; 89 ; 82 ; 37 ; 38 ; 78 ; 73 ; 12 ) , mixtures of regressions ( 20 ; 40 ) , and broader MoE frameworks ( 44 ; 68 ; 83 ; 64 ; 62 ; 67 ; 71 ; 70 ; 72 ) ; see also surveys ( 99 ; 56 ; 63 ; 81 ; 6 ) . Despite this progress, for SGMLMoE two core issues remain underdeveloped: stable maximum-likelihood training with provable convergence, and model selection (especially the number of experts) with rigorous guarantees and computational efficiency. 1.2 Stable Maximum-likelihood Training via MM in the Full-data Regime Likelihood-based estimation for SGMLMoE is challenging because the observed-data objective couples softmax gating and multinomial-logistic experts in a highly non-convex way. EM ( 17 ; 58 ; 57 ) is the classical template for latent-variable models, but in multinomial-logistic MoE the M-steps typically lack closed forms, so practical EM relies on inner-loop or approximate maximization, which can complicate monotonicity and global convergence guarantees. MM algorithms ( 84 ; 15 ; 16 ; 53 ; 92 ; 66 ; 52 ; 55 ; 54 ) instead optimize explicit surrogates that majorize/minorize the objective and can yield simple updates with stability guarantees. We focus on the batch regime, avoiding the stochastic-approximation noise inherent in incremental and mini-batch variants of EM/MM ( 5 ; 51 ; 18 ; 27 ; 97 ; 7 ; 8 ; 23 ; 25 ; 65 ; 50 ; 47 ; 14 ; 46 ; 85 ; 86 ; 9 ; 26 ; 24 ) and related variational/free-energy views ( 60 ) . 1.3 Model Selection in SGMLMoE Models Selecting the number of experts remains central. Classical criteria require fitting multiple model sizes and then applying AIC ( 1 ; 30 ) , BIC ( 90 ; 49 ; 21 ; 3 ; 22 ; 69 ; 36 ) , ICL ( 4 ; 29 ) , eBIC ( 28 ; 61 ) , or SWIC ( 91 ; 98 ) . For MoE, over-specification can induce non-identifiability and singularities, undermining the usual asymptotic justifications. Alternatives include non-asymptotic penalization ( 74 ; 75 ; 80 ; 77 ; 59 ; 79 ) and Bayesian/post-processing strategies such as merge-truncate-merge ( 31 ; 101 ; 33 ; 76 ) , but many remain tied to multi- K sweeps. A recent sweep-free direction uses dendrograms of mixing measures ( 19 ; 95 ; 34 ) , developed mainly for Gaussian-expert settings; we adapt this viewpoint to multinomial-logistic classification and integrate it with stable batch-MM training. 1.4 Contributions and Paper Organization We develop a unified framework for SGMLMoE that couples stable batch maximum-likelihood training with sweep-free model selection . On the optimization side, we construct an explicit quadratic MM surrogate, leading to coordinate-wise closed-form updates that exactly minimize the surrogate each iteration and therefore guarantee monotone ascent and global convergence to a stationary point without inner-loop M-steps. On the statistical side, we establish finite-sample rates for conditional density estimation and parameter recovery, and we introduce a dendrogram-based aggregation path on mixing measures that restores near-parametric rates after merging redundant fitted atoms (see Table 1 ) and yields a consistent order selector without multi- K training (see Figure 1 ). Organization. Section 2 introduces SGMLMoE and the likelihood objective. Section 3 presents the batch MM algorithm and its monotonicity. Section 4 develops the Voronoi loss, aggregation path, convergence rates, and dendrogram selection criterion. Experiments are reported in Section 5 . All additional technical proofs and implementation details are deferred to the appendix. Figure 1 : Illustration of the DSC merging path on an over-specified fit ( K=4,M=2 ): successive merges remove near-duplicate experts and recover the true expert structure ( K_{0}=2,M=2 ). 2 Preliminaries SGMLMoE Models. We refer to the output {\textnormal{y}}\in{\mathbb{Y}} as the response variable and the input \mathbf{{\mathbf{x}}}\in{\mathbb{X}}\subset{\mathbb{R}}^{P} , P\in{\mathbb{N}} , as the predictor variable. We consider a batch dataset of size N , denoted by \{({\mathbf{x}}_{n},{\textnormal{y}}_{n}):n\in[N]\} , consisting of i.i.d. copies of ({\mathbf{x}},{\textnormal{y}}) , where the conditional law of y given {\mathbf{x}}={\bm{x}} is characterized by an unknown conditional probability mass function (or density) \pi(\cdot\mid{\mathbf{x}}={\bm{x}}) . The corresponding observed values are written as ({\bm{x}},{y}) . Motivated by universal approximation results for MoE models, we approximate \pi(\cdot\mid{\mathbf{x}}={\bm{x}}) by a SGMLMoE of the form \displaystyle s_{{\bm{\theta}}}({y}\mid{\bm{x}})=\sum_{k=1}^{K}\mathsf{g}_{k}\!\big({\bm{w}}({\bm{x}})\big)\,\mathsf{e}_{k}\!\big({y};{\bm{v}}({\bm{x}})\big), (1) where {\bm{\theta}} collects all unknown parameters and K is the number of experts. The gating probabilities are defined by the softmax map \mathsf{g}_{k}\!({\bm{w}}({\bm{x}}))=\exp\!\big(w_{k}({\bm{x}})\big)/(\sum_{l=1}^{K}\exp\!\big(w_{l}({\bm{x}})\big)), \forall k\in[K], with gating score functions {\bm{w}}({\bm{x}})=(w_{1}({\bm{x}}),\ldots,w_{K}({\bm{x}})) . In the discrete-output setting, we take {\mathbb{Y}}=[M] , M\in{\mathbb{N}} , and specify each expert as a multinomial-logistic model \mathsf{e}_{k}\!\big({y}=m;{\bm{v}}({\bm{x}};{\bm{\upsilon}})\big)=\exp\!\big({v}_{m,k}({\bm{x}})\big)/\sum_{l=1}^{M}\exp\!\big({v}_{l,k}({\bm{x}})\big), \forall k\in[K],\ \forall m\in[M], where the expert scores {\bm{v}}({\bm{x}})=({v}_{m,k}({\bm{x}}))_{m\in[M],k\in[K]} are modeled as polynomial functions of {\bm{x}} . Concretely, for degrees D_{W},D_{V}\in{\mathbb{N}} , w_{k}({\bm{x}})=\sum_{d=0}^{D_{W}}{\bm{\omega}}_{k,d}^{\top}{\bm{x}}^{d}=\sum_{d=0}^{D_{W}}\Big(\sum_{p=1}^{P}\omega_{k,d,p}{x}_{p}^{d}\Big) , {v}_{m,k}({\bm{x}})=\sum_{d=0}^{D_{V}}{\bm{\upsilon}}_{m,k,d}^{\top}{\bm{x}}^{d}=\sum_{d=0}^{D_{V}}\Big(\sum_{p=1}^{P}\upsilon_{m,k,d,p}{x}_{p}^{d}\Big) with coefficient collections {\bm{\omega}}=(\omega_{k,d,p})_{k\in[K],\,d\in\{0,\ldots,D_{W}\},\,p\in[P]} and {\bm{\upsilon}}=(\upsilon_{m,k,d,p})_{m\in[M],\,k\in[K],\,d\in\{0,\ldots,D_{V}\},\,p\in[P]} . The full parameter vector is denoted by {\bm{\theta}}=\big({\bm{\omega}},{\bm{\upsilon}}\big) . For simplify, we take D_{W}=D_{V}=D and leave the general case D_{W}\neq D_{V} for future work. Maximum Log-likelihood Estimator. We denote by {\bm{\theta}}^{0} a maximizer of the population objective {\bm{\theta}}\mapsto\mathbb{E}_{{\textnormal{y}}\mid{\mathbf{x}}\sim\mathbb{P}}\left[\log s_{{\bm{\theta}}}({\textnormal{y}}\mid{\mathbf{x}})\right] . Given the batch dataset \{({\bm{x}}_{n},{y}_{n})\}_{n=1}^{N} , we estimate {\bm{\theta}} by maximum likelihood, \widehat{{\bm{\theta}}}_{N}\in\arg\max_{{\bm{\theta}}\in{\mathbb{T}}}\;\frac{1}{N}\mathcal{L}({\bm{\theta}}), where the observed-data log-likelihood is \displaystyle\mathcal{L}({\bm{\theta}})=\sum_{n=1}^{N}\log\!\left[\sum_{k=1}^{K}\mathsf{g}_{k}\!\big({\bm{w}}({\bm{x}}_{n})\big)\,\mathsf{e}_{k}\!\big({y}_{n};{\bm{v}}({\bm{x}}_{n})\big)\right]. (2) In this paper we work in the full-data (batch) regime, focusing on stable maximum-likelihood fitting for Equation 1 – Equation 2 without invoking stochastic-approximation devices used in incremental or mini-batch variants. Identifiability of SGMLMoE Models. Because the softmax gate is invariant to additive shifts, the gating parameters are not fully identifiable. In particular, if we translate each coefficient vector as {\bm{\omega}}_{k,d}\mapsto{\bm{\omega}}_{k,d}+{\bm{t}} for some {\bm{t}}\in{\mathbb{R}}^{P} , then the resulting gating probabilities \mathsf{g}_{k}({\bm{w}}({\bm{x}})) remain unchanged, so \{{\bm{\omega}}_{k,d}\} are identifiable only up to a common translation. This phenomenon, and standard remedies for it, are discussed in the context of establishing convergence rates for maximum-likelihood estimation of softmax-gated Gaussian MoE models in ( 71 ) . Following ( 35 ; 43 ) , we enforce identifiability by fixing a reference expert and imposing, without loss of generality, the constraint \{{\bm{\omega}}_{K,d,p}\}_{d\in\{0,\ldots,D\},\,p\in[P]}=\mathbf{0} , which yields the equivalent parameterization \mathsf{g}_{K}({\bm{x}};{\bm{\omega}})=1-\sum_{k=1}^{K-1}\mathsf{g}_{k}({\bm{x}};{\bm{\omega}}) and \mathsf{g}_{k}({\bm{x}};{\bm{\omega}})=\exp(w_{k}({\bm{x}};{\bm{\omega}}_{k}))\big/\!\left(1+\sum_{l=1}^{K-1}\exp(w_{l}({\bm{x}};{\bm{\omega}}_{l}))\right) for all k\in[K-1] . In addition to the gating identifiability constraint, we also impose an identifiability convention for the multinomial-logistic expert networks, as considered in the convergence-rate analysis for softmax-gated multinomial logistic MoE models in ( 70 ) . Specifically, we fix a reference class and set, without loss of generality, \{\upsilon_{M,k,d,p}\}_{d\in\{0,\ldots,D\},\,p\in[P]}=\mathbf{0} for all k\in[K] , so that the last class probability is determined by the remaining M-1 logits. Equivalently, for each expert k , we have \mathsf{e}_{k}({\bm{v}}_{M,k}({\bm{x}}))=1-\sum_{m=1}^{M-1}\mathsf{e}_{k}({\bm{v}}_{m,k}({\bm{x}})) and \mathsf{e}_{k}({\bm{v}}_{m,k}({\bm{x}}))=\exp({v}_{m,k}({\bm{x}}))\big/\!\left(1+\sum_{l=1}^{M-1}\exp({v}_{l,k}({\bm{x}}))\right) for all m\in[M-1] . 3 MM Algorithm for SGMLMoE Models We now specialize the MM framework to the full-data (batch) maximum-likelihood problem for the SGMLMoE model in Equation 1 with discrete output {\textnormal{y}}\in{\mathbb{Y}}=[M] . Let \{({\bm{x}}_{n},{\textnormal{y}}_{n})\}_{n\in[N]} be i.i.d. copies of ({\mathbf{x}},{\textnormal{y}}) and recall the observed-data log-likelihood \mathcal{L}({\bm{\theta}}) in Equation 2 . For each datum n\in[N] and expert k\in[K] , define the usual posterior responsibility at the current iterate {\bm{\theta}}^{(t)} by \displaystyle\tau_{n,k}^{(t)}=\frac{\mathsf{g}_{k}\!\left({\bm{w}}^{(t)}({\bm{x}}_{n})\right)\,\mathsf{e}_{k}\!\left({\bm{v}}^{(t)}_{{\textnormal{y}}_{n}}({\bm{x}}_{n})\right)}{\sum_{l=1}^{K}\mathsf{g}_{l}\!\left({\bm{w}}^{(t)}({\bm{x}}_{n})\right)\,\mathsf{e}_{l}\!\left({\bm{v}}^{(t)}_{{\textnormal{y}}_{n}}({\bm{x}}_{n})\right)}. (3) To simplify notation for polynomial features, let \displaystyle\hat{{\bm{x}}}_{n}=\left[{x}_{n,1}^{0},\dots,{x}_{n,1}^{D},\dots,{x}_{n,P}^{0},\dots,{x}_{n,P}^{D}\right]^{\top}\in{\mathbb{R}}^{P(D+1)}. Next, we use an indicator polynomial \mathbbm{1}(z,l) to encode class labels: \mathbbm{1}(z,l)=\frac{\prod_{q=1}^{M}(z-q)}{(z-l)(z-1)!(M-z)!(-1)^{M-z}},\forall z,l\in[M], so that \mathbbm{1}(z,l)\in\{0,1\} . This representation is convenient for coding and yields the identity {v}_{m,k}({\bm{x}})=\sum_{l=1}^{M}\mathbbm{1}(m,l)\,{v}_{l,k}({\bm{x}}), which allows recovering any class logit from the vector of logits. Batch Sufficient-statistics Notation. For each n\in[N] , define the gating-design vector in {\mathbb{R}}^{(K-1)P(D+1)} : \displaystyle{\bm{s}}_{n}^{(t)}=\big[\tau_{n,1}^{(t)}{x}_{n,1}^{0},\dots,\tau_{n,1}^{(t)}{x}_{n,1}^{D},\dots,\tau_{n,K-1}^{(t)}{x}_{n,P}^{D}\big]^{\top}, (4) and define the expert-design vector {\bm{r}}_{n}^{(t)} by stacking r_{n,d,l,p,k}^{(t)}=\mathbbm{1}({\textnormal{y}}_{n},l)\,\tau_{n,k}^{(t)}\,{x}_{n,p}^{d} , \penalty\ d\in\{0,\ldots,D\},\penalty\ l\in[M],\penalty\ p\in[P],\penalty\ k\in[K], into {\bm{r}}_{n}^{(t)}=\big[r_{n,0,1,1,1}^{(t)},\dots,r_{n,D,1,1,1}^{(t)},\dots,r_{n,D,M,1,1}^{(t)},\dots,r_{n,D,M,P,K}^{(t)}\big]^{\top}. We also define the class-parameter blocks for each expert k by {\bm{c}}_{k}=\big[\upsilon_{1,k,0,1},\dots,\upsilon_{1,k,D,1},\dots,{\bm{\upsilon}}_{M,k,D,1},\dots,\upsilon_{M,k,D,P}\big]^{\top} , {\bm{\upsilon}}=\vect([{\bm{c}}_{1},\dots,{\bm{c}}_{K}]). Finally, we aggregate the batch statistics as {\bm{s}}^{(t)}=\sum_{n=1}^{N}{\bm{s}}_{n}^{(t)} , {\bm{r}}^{(t)}=\sum_{n=1}^{N}{\bm{r}}_{n}^{(t)}. Quadratic Majorizers for the Gate and Experts. For k\in[K-1] , let w_{k}({\bm{x}})=\sum_{d=0}^{D}{\bm{\omega}}_{k,d}^{\top}{\bm{x}}^{d} and define the gate log-sum-exp term g_{n}({\bm{w}})=\log\!\left(1+\sum_{k=1}^{K-1}\exp\!\left(w_{k}({\bm{x}}_{n})\right)\right). For experts, define for each k\in[K] the multinomial log-sum-exp term e_{n}({\bm{c}}_{k})=\log\!\left(1+\sum_{m=1}^{M-1}\exp\!\left({v}_{m,k}({\bm{x}}_{n})\right)\right). For any K\in{\mathbb{N}} , we use the quadratic majorizers based on the matrices \displaystyle{\bm{B}}_{n,K} \displaystyle=\left(\frac{3}{4}{\bm{I}}_{K-1}-\frac{\bm{1}_{K-1}\bm{1}_{K-1}^{\top}}{2(K-1)}\right)\otimes\hat{{\bm{x}}}_{n}\hat{{\bm{x}}}_{n}^{\top}. (5) For compactness, introduce the centered increments \bar{{\bm{w}}}_{t}={\bm{w}}-{\bm{w}}^{(t)} and \bar{{\bm{c}}}_{k,t}={\bm{c}}_{k}-{\bm{c}}_{k}^{(t)} . Theorem 1 (Surrogate for the batch negative log-likelihood.) . The following bound defines a batch MM surrogate for -\mathcal{L}({\bm{\theta}}) , with proof deferred to Section D.2 : \displaystyle-\mathcal{L}({\bm{\theta}})\leq C^{(t)}+\mathcal{S}_{1}({\bm{\theta}},{\bm{\theta}}^{(t)})+\mathcal{S}_{2}({\bm{\theta}},{\bm{\theta}}^{(t)}) (6) Here C^{(t)} is an independent constant w.r.t {\bm{\theta}} , \mathcal{S}_{1}({\bm{\theta}},{\bm{\theta}}^{(t)})=\sum_{n=1}^{N}\{g_{n}({\bm{w}}^{(t)})+\bar{{\bm{w}}}_{t}^{\top}\nabla g_{n}({\bm{w}}^{(t)})+\frac{1}{2}\bar{{\bm{w}}}_{t}^{\top}{\bm{B}}_{n,K}\bar{{\bm{w}}}_{t}\} , and \mathcal{S}_{1}({\bm{\theta}},{\bm{\theta}}^{(t)})=\sum_{n=1}^{N}\sum_{k=1}^{K}\tau_{n,k}^{(t)}\{e_{n}({\bm{c}}_{k}^{(t)})+\bar{{\bm{c}}}_{k,t}^{\top}\nabla e_{n}({\bm{c}}_{k}^{(t)})+\frac{1}{2}\bar{{\bm{c}}}_{k,t}^{\top}{\bm{B}}_{n,M}\bar{{\bm{c}}}_{k,t}\} . Algorithm 1 Batch MM algorithm for SGMLMoE 0: Data \{({\bm{x}}_{n},{\textnormal{y}}_{n})\}_{n=1}^{N} ; (K,M) ; degree D ; init. {\bm{\theta}}^{(0)}=({\bm{w}}^{(0)},{\bm{v}}^{(0)}) ; tol. \varepsilon ; max iters T . 0: \{{\bm{\theta}}^{(t)}\}_{t\geq 0} . \For t=0,1,\dots,T-1 \State E-step. For each n\in[N] , compute responsibilities \{\tau_{n,k}^{(t)}\}_{k\in[K]} by Equation 3 . \State Form sufficient-statistics {\bm{s}}^{(t)}=\sum_{n}{\bm{s}}_{n}^{(t)} and {\bm{r}}^{(t)}=\sum_{n}{\bm{r}}_{n}^{(t)} using Equation 4 and the definition of {\bm{r}}_{n}^{(t)} (with \mathbbm{1}({\textnormal{y}}_{n},l) ). \State Build curvature matrices {\bm{B}}_{n,K-1}=\sum_{n}{\bm{B}}_{n,K} and {\bm{B}}_{n,M-1}=\sum_{n}{\bm{B}}_{n,M} with blocks {\bm{B}}_{n,K},{\bm{B}}_{n,M} from Equation 5 . \State Compute \nabla g({\bm{w}}^{(t)})=\sum_{n}\nabla g_{n}({\bm{w}}^{(t)}) and the expert log-sum-exp gradients \{\nabla e_{n}({\bm{v}}^{(t)})\}_{n\in[N]} (definitions in Equation 6 ). \State M-step. Update ({\bm{w}},{\bm{v}}) by the closed-form minimizers of the surrogate \mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)}) in Equation 6 : {\bm{w}}^{(t+1)}={\bm{w}}^{(t)}+{\bm{B}}_{n,K-1}^{-1}\!\Big({\bm{s}}^{(t)}-\nabla g({\bm{w}}^{(t)})\Big), {\bm{v}}^{(t+1)}=\arg\min_{{\bm{v}}}\ \mathcal{S}\big(({\bm{w}}^{(t+1)},{\bm{v}}),{\bm{\theta}}^{(t)}\big), where the explicit block-linear solve for {\bm{v}}^{(t+1)} is given in Appendix B . \State Set {\bm{\theta}}^{(t+1)}=({\bm{w}}^{(t+1)},{\bm{v}}^{(t+1)}) . \State Stop. If |\mathcal{L}({\bm{\theta}}^{(t+1)})-\mathcal{L}({\bm{\theta}}^{(t)})|\leq\varepsilon , break. \EndFor These updates in Algorithm 2 minimize the surrogate Equation 6 exactly, and therefore inherit the standard MM monotone-ascent property for the observed-data likelihood via Theorem 2 , whose proof follows from the MM majorization–tangency properties of \mathcal{S} and exact minimization; see Section D.3 in the appendix. Theorem 2 (MM monotonicity for batch SGMLMoE) . Let \mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)}) be the batch MM surrogate of -\mathcal{L}({\bm{\theta}}) in Equation 6 and {\bm{\theta}}^{(t+1)} be the output of Algorithm 1 . Then, {\bm{\theta}}^{(t+1)}\in\arg\min_{{\bm{\theta}}}\mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)}) and \mathcal{L}({\bm{\theta}}^{(t+1)})\geq\mathcal{L}({\bm{\theta}}^{(t)}) for all t . 4 Fast-rate-aware Aggregation and Sweep-free Model Selection for SGMLMoE 4.1 The “Rate Gap” under Overfitting In the SGMLMoE classification model Equation 1 – Equation 2 , likelihood-based estimation is challenging for the same structural reasons that arise in softmax-gated Gaussian experts ( 34 ) , but with multinomial-logistic experts: (i) softmax translation invariance makes gate parameters identifiable only up to a common shift; (ii) gate–expert coupling yields nontrivial interactions in local expansions of s_{{\bm{\theta}}}(\cdot\mid{\bm{x}}) ; (iii) softmax numerator–denominator coupling creates cancellation phenomena that can make densities close even when individual components are not. These effects become most visible when the fitted model is over-specified ( K>K_{0} ): several fitted atoms can cluster around a single true expert. In this regime, the conditional density s_{\widehat{G}_{N}}(\cdot\mid{\bm{x}}) can converge at near-parametric rates in total variation, while componentwise parameters converge more slowly along near-nonidentifiable directions. The goal of this section is to (a) introduce a Voronoi-type loss aligned with the gate-induced partition geometry that resolves these slow directions, and (b) leverage it to build a hierarchical aggregation path (a dendrogram of mixing measures) that yields a sweep-free and consistent selector of the number of experts. Practical Implication (Single-fit Workflow). Fit one over-specified SGMLMoE (moderate K\geq K_{0} ) using the batch MM procedure in Algorithm 2 , construct its dendrogram by iterative merging, and select \widehat{K} via a height–likelihood dendrogram selection criterion (DSC). This avoids training multiple models across candidate sizes. 4.2 Mixing-measure Representation for SGMLMoE Lifted Features and Intercept/Slope Decomposition. With the polynomial degree D fixed, define the non-constant lifted feature map \phi_{D}({\bm{x}}):=\vect\!\big([{\bm{x}}^{1},{\bm{x}}^{2},\ldots,{\bm{x}}^{D}]\big)\in{\mathbb{R}}^{PD} , \widehat{{\mathbb{X}}}:=\phi_{D}({\mathbb{X}})\subset{\mathbb{R}}^{PD}. (Since {\mathbb{X}} is compact and \phi_{D} is continuous, \widehat{{\mathbb{X}}} is compact via using Heine-Borel theorem). For each gate k\in[K-1] , decompose w_{k}({\bm{x}})=\bar{\omega}_{k}+\hat{\bm{\omega}}_{k}^{\top}\hat{\bm{x}} , \hat{\bm{x}}:=\phi_{D}({\bm{x}}), where \bar{\omega}_{k}:=\sum_{p=1}^{P}\omega_{k,0,p}\in{\mathbb{R}} and \hat{\bm{\omega}}_{k}:=\vect([{\bm{\omega}}_{k,1},\ldots,{\bm{\omega}}_{k,D}])\in{\mathbb{R}}^{PD} . For each expert k\in[K] and class m\in[M-1] , similarly write {v}_{m,k}({\bm{x}})=\bar{\upsilon}_{m,k}+\hat{\bm{\upsilon}}_{m,k}^{\top}\hat{\bm{x}}, with \bar{\upsilon}_{m,k}:=\sum_{p=1}^{P}\upsilon_{m,k,0,p}\in{\mathbb{R}} and \hat{\bm{\upsilon}}_{m,k}:=\vect([{\bm{\upsilon}}_{m,k,1},\ldots,{\bm{\upsilon}}_{m,k,D}])\in{\mathbb{R}}^{PD} . Identifiability Conventions (as in Section 2 ). We fix the reference expert and the reference class: (\bar{\omega}_{K},\hat{\bm{\omega}}_{K})=(0,\mathbf{0}), (\bar{\upsilon}_{M,k},\hat{\bm{\upsilon}}_{M,k})=(0,\mathbf{0}),\ \ \forall k\in[K]. Mixing Measures. Let {\mathbb{T}} denote the identifiable parameter space for one SGMLMoE atom: \eta=\Big(\hat{\bm{\omega}},\bar{\omega},\ \{(\hat{\bm{\upsilon}}_{m},\bar{\upsilon}_{m})\}_{m=1}^{M-1}\Big)\in{\mathbb{T}}, and let \mathcal{O}_{K}({\mathbb{T}}) be the set of finite mixing measures with at most K atoms: \mathcal{O}_{K}({\mathbb{T}})=\Big\{G=\sum_{i=1}^{K}\pi_{i}\,\delta_{\eta_{i}}:\ 1\leq K\leq K,\ \pi_{i}>0,\ \eta_{i}\in{\mathbb{T}}\Big\}. We parameterize weights by \pi_{i}=\exp(\bar{\omega}_{i}) (so \bar{\omega}_{i}=\log\pi_{i} ). Conditional Probability Mass Function Induced by G . Given G=\sum_{i=1}^{K}\pi_{i}\delta_{\eta_{i}} , define for \hat{\bm{x}}=\phi_{D}({\bm{x}}) \mathsf{g}_{i}({\bm{x}};G)=\frac{\pi_{i}\exp(\hat{\bm{\omega}}_{i}^{\top}\hat{\bm{x}})}{\sum_{j=1}^{K}\pi_{j}\exp(\hat{\bm{\omega}}_{j}^{\top}\hat{\bm{x}})}, \mathsf{e}_{i}({y}=m\mid{\bm{x}};G)=\frac{\exp(\bar{\upsilon}_{m,i}+\hat{\bm{\upsilon}}_{m,i}^{\top}\hat{\bm{x}})}{1+\sum_{\ell=1}^{M-1}\exp(\bar{\upsilon}_{\ell,i}+\hat{\bm{\upsilon}}_{\ell,i}^{\top}\hat{\bm{x}})}, with \mathsf{e}_{i}({y}=M\mid{\bm{x}};G)=1-\sum_{m=1}^{M-1}\mathsf{e}_{i}({y}=m\mid{\bm{x}};G) , and s_{G}({y}\mid{\bm{x}})=\sum_{i=1}^{K}\mathsf{g}_{i}({\bm{x}};G)\,\mathsf{e}_{i}({y}\mid{\bm{x}};G). For any identifiable {\bm{\theta}} , there exists G({\bm{\theta}})\in\mathcal{O}_{K}({\mathbb{T}}) such that s_{G({\bm{\theta}})}\equiv s_{{\bm{\theta}}} . MLE over at most K experts. Let G_{0}\in\mathcal{O}_{K_{0}}({\mathbb{T}}) be the true mixing measure ( 2\leq K_{0}\leq K ). Define the (mixing-measure) MLE \widehat{G}_{N}\in\arg\max_{G\in\mathcal{O}_{K}({\mathbb{T}})}\ \frac{1}{N}\sum_{n=1}^{N}\log s_{G}({y}_{n}\mid{\bm{x}}_{n}). 4.3 Voronoi Cells and Losses for SGMLMoE From “Density Closeness” to “Parameter Closeness”. A key step in overfitted mixture/MoE analysis is an inverse inequality : small discrepancy between s_{G} and s_{G_{0}} should imply small discrepancy between G and G_{0} in a geometry-adapted loss. For softmax-gated MoE, the adapted geometry is Voronoi : fitted atoms are grouped into cells around each true atom. Voronoi Partition. Write G=\sum_{i=1}^{K}\pi_{i}\delta_{\eta_{i}} and G_{0}=\sum_{k=1}^{K_{0}}\pi_{k}^{0}\delta_{\eta_{k}^{0}} . When G is sufficiently close to G_{0} (e.g. in the sense below), define a matching i\mapsto k(i)\in[K_{0}] that assigns each fitted atom \eta_{i} to its nearest true atom \eta_{k(i)}^{0} under the local metric on {\mathbb{T}} . Define Voronoi cells {\mathbb{V}}_{k}:=\{\,i\in[K]:\ k(i)=k\,\} , k\in[K_{0}]. Integrated Total Variation Discrepancy. For {\bm{x}}\in{\mathbb{X}} , define \mathcal{D}_{\mathrm{TV}}\!(s_{G}(\cdot\mid{\bm{x}}),s_{G_{0}}(\cdot\mid{\bm{x}})) as \frac{1}{2}\sum_{m=1}^{M}|s_{G}(m\mid{\bm{x}})-s_{G_{0}}(m\mid{\bm{x}})|, and its covariate-averaged version \mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}}) as \mathbb{E}_{{\mathbf{x}}}[\mathcal{D}_{\mathrm{TV}}\!\big(s_{G}(\cdot\mid{\mathbf{x}}),s_{G_{0}}(\cdot\mid{\mathbf{x}})\big)]. SGMLMoE Voronoi Loss. For i\in{\mathbb{V}}_{k} , define parameter differences \Delta\hat{\bm{\omega}}_{ik}:=\hat{\bm{\omega}}_{i}-\hat{\bm{\omega}}_{k}^{0}, \Delta\bar{\upsilon}_{ikm}:=\bar{\upsilon}_{m,i}-\bar{\upsilon}_{m,k}^{0}, \Delta\hat{\bm{\upsilon}}_{ikm}:=\hat{\bm{\upsilon}}_{m,i}-\hat{\bm{\upsilon}}_{m,k}^{0},\ \ m\in[M-1] , and the loss \displaystyle\mathcal{D}_{\mathrm{V}}(G,G_{0})\>=\mathcal{D}_{\mathrm{E}}(G,G_{0}) (7) \displaystyle+\sum_{\begin{subarray}{c}k\in[K_{0}]:\,|{\mathbb{V}}_{k}|>1\\ i\in{\mathbb{V}}_{k},m\in[M-1]\end{subarray}}\pi_{i}[\|\Delta\hat{\bm{\omega}}_{ik}\|^{2}+(|\Delta\bar{\upsilon}_{ikm}|^{2}+\|\Delta\hat{\bm{\upsilon}}_{ikm}\|^{2})], \text{where }\mathcal{D}_{\mathrm{E}}(G,G_{0}):=\sum_{k=1}^{K_{0}}\Big|\sum_{i\in{\mathbb{V}}_{k}}\pi_{i}-\pi_{k}^{0}\Big|+\sum_{\begin{subarray}{c}k\in[K_{0}]:\,|{\mathbb{V}}_{k}|=1\\ i\in{\mathbb{V}}_{k},m\in[M-1]\end{subarray}}\pi_{i}[\|\Delta\hat{\bm{\omega}}_{ik}\|+(|\Delta\bar{\upsilon}_{ikm}|+\|\Delta\hat{\bm{\upsilon}}_{ikm}\|)]. Local Structure Inequality (Inverse Bound). The next result is the “density \Rightarrow Voronoi” inverse bounds developed for SGMLMoE. Theorem 3 . For G\in\mathcal{O}_{K}({\mathbb{T}}) , as \mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})\to 0 , \mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})\ \gtrsim\ \mathcal{D}_{\mathrm{V}}(G,G_{0}). (8) 4.4 Why Merge Experts? The Dendrogram Viewpoint The Overfitted Picture. When K>K_{0} , some Voronoi cell {\mathbb{V}}_{k} may contain multiple fitted atoms. These atoms are “near duplicates” from the perspective of the conditional density, and they create slow/flat directions in the likelihood geometry. The dendrogram mechanism collapses each such cluster by repeated merging, producing a hierarchy of mixing measures \widehat{G}_{N}^{(K)}=\widehat{G}_{N}\ \rightsquigarrow\ \widehat{G}_{N}^{(K-1)}\ \rightsquigarrow\ \cdots\ \rightsquigarrow\ \widehat{G}_{N}^{(2)}. The key facts we exploit are: (i) each merge reduces redundancy, and (ii) the Voronoi loss becomes easier (monotonically) along the merge chain. Atom Dissimilarity (Merge Criterion). For two atoms \pi_{i}\delta_{\eta_{i}} and \pi_{j}\delta_{\eta_{j}} , define \displaystyle\mathrm{d}\!\big(\pi_{i}\delta_{\eta_{i}},\pi_{j}\delta_{\eta_{j}}\big):=\frac{\pi_{i}\pi_{j}}{\pi_{i}+\pi_{j}}\big(\|\hat{\bm{\omega}}_{i}-\hat{\bm{\omega}}_{j}\|^{2} \displaystyle\qquad+\sum_{m=1}^{M-1}\big(\|\hat{\bm{\upsilon}}_{m,i}-\hat{\bm{\upsilon}}_{m,j}\|^{2}+|\bar{\upsilon}_{m,i}-\bar{\upsilon}_{m,j}|^{2}\big)\big). (9) Merge Operator (Weight-averaged Atom). Merging atoms i and j produces a new atom (\pi_{*},{\bm{\eta}}_{*}) via parameter barycenters for {\bm{\eta}}_{*} as follows: given any m\in[M-1] , \displaystyle\pi_{*}=\pi_{i}+\pi_{j},\penalty\ \bar{\omega}_{*}=\log\pi_{*},\hat{\bm{\upsilon}}_{m,*}=\frac{\pi_{i}}{\pi_{*}}\hat{\bm{\upsilon}}_{m,i}+\frac{\pi_{j}}{\pi_{*}}\hat{\bm{\upsilon}}_{m,j}, \displaystyle\bar{\upsilon}_{m,*}=\frac{\pi_{i}}{\pi_{*}}\bar{\upsilon}_{m,i}+\frac{\pi_{j}}{\pi_{*}}\bar{\upsilon}_{m,j},\penalty\ \hat{\bm{\omega}}_{*}=\frac{\pi_{i}}{\pi_{*}}\hat{\bm{\omega}}_{i}+\frac{\pi_{j}}{\pi_{*}}\hat{\bm{\omega}}_{j}. (10) Theorem 4 (Voronoi monotonicity along the merge chain) . Let G^{(K)},G^{(K-1)},\ldots,G^{(K_{0})} be obtained by repeatedly merging the closest pair of atoms according to Section 4.4 , starting from G^{(K)}\in\mathcal{O}_{K}({\mathbb{T}}) . For G^{(K)} sufficiently close to G_{0} , \mathcal{D}_{\mathrm{V}}\!\big(G^{(K)},G_{0}\big)\ \gtrsim\ \mathcal{D}_{\mathrm{V}}\!\big(G^{(K-1)},G_{0}\big)\ \gtrsim\ \cdots\ \gtrsim\ \mathcal{D}_{\mathrm{V}}\!\big(G^{(K_{0})},G_{0}\big). 4.5 Finite-sample Rates along the Dendrogram Heights (structural signal). At level \kappa , define the dendrogram height h_{N}^{(\kappa)}:=\min_{i\neq j\ \text{atoms of }\widehat{G}_{N}^{(\kappa)}}\ \mathrm{d}\!\big(\pi_{i}\delta_{\eta_{i}},\pi_{j}\delta_{\eta_{j}}\big), \kappa\in[K], and let h_{0}^{(\kappa^{\prime})} be the analogous height on the true chain G_{0}^{(\kappa^{\prime})} , \kappa^{\prime}\in[K_{0}] . Density Rate for the MLE. We use the known near-parametric bound (up to logs \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) ) for the MLE in total variation as established for the multinomial classification regime in ( 70 ) : Fact 1 (Density convergence rate) . There exist universal constants C_{1},C_{2}>0 such that \mathbb{P}\!(\mathcal{D}_{\mathrm{TV}}(s_{\widehat{G}_{N}},s_{G_{0}})>C_{1}\sqrt{\log N/N})\lesssim N^{-C_{2}}. From Density Rates to Voronoi and Height Rates. Combining Theorem 3 with 1 yields Voronoi control at level K , and the monotonicity in Theorem 4 propagates it down the path: Theorem 5 (Voronoi and height rates along the path) . With probability at least 1-c_{1}N^{-c_{2}} , for each \kappa\in\{K_{0}+1,\ldots,K\} and \kappa^{\prime}\in[K_{0}] , \max\{\mathcal{D}_{\mathrm{V}}(\widehat{G}_{N}^{(\kappa)},G_{0}),h_{N}^{(\kappa)},|h_{N}^{(\kappa^{\prime})}-h_{0}^{(\kappa^{\prime})}|\}\ \lesssim\ \sqrt{\log N/N}. Table 1 : Summary of (log-adjusted) density and parameter rates for SGMLMoE with standard softmax gating. Setting Density Exact-specified Parameters ( |{\mathbb{V}}_{k}|=1 ) Over-specified Parameters ( |{\mathbb{V}}_{k}|>1 ) Exact-fit \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) — Over-fit \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/4}\big) Merged \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) \widetilde{\mathcal{O}}\!\big((\log N/N)^{1/2}\big) 4.6 Likelihood along the Path and Sweep-free Selection Empirical and Population Log-likelihoods. For G\in\mathcal{O}_{K}({\mathbb{T}}) , define \mathcal{L}(G):=\mathbb{E}_{({\mathbf{x}},{\textnormal{y}})\sim G_{0}}\big[\log s_{G}({\textnormal{y}}\mid{\mathbf{x}})\big], and \bar{\ell}_{N}(G):=\frac{1}{N}\sum_{n=1}^{N}\log s_{G}({y}_{n}\mid{\bm{x}}_{n}). Write \bar{\ell}_{N}^{(\kappa)}:=\bar{\ell}_{N}(\widehat{G}_{N}^{(\kappa)}) . Condition K (local lower bound). There exist constants c_{\alpha},c_{\beta}>0 such that for all sufficiently small \epsilon>0 and all {\bm{\theta}},{\bm{\theta}}^{0}\in{\mathbb{T}} with \|{\bm{\theta}}-{\bm{\theta}}^{0}\|\leq\epsilon , for all ({\mathbf{x}},{\textnormal{y}})\in{\mathbb{X}}\times{\mathbb{Y}} , \log s_{{\bm{\theta}}}({\textnormal{y}}\mid{\mathbf{x}})\geq(1+c_{\beta}\epsilon)\,\log s_{{\bm{\theta}}^{0}}({\textnormal{y}}\mid{\mathbf{x}})-c_{\alpha}\epsilon. Theorem 6 (Likelihood control along the dendrogram) . Assume Theorem 5 and Condition K hold. Then, for any \kappa\in\{K_{0}+1,\ldots,K\} , \bar{\ell}_{N}\big(\widehat{G}_{N}^{(\kappa)}\big)-\mathcal{L}(G_{0})\ \lesssim\ (\log N/N)^{1/4}. Moreover, for any \kappa^{\prime}\in[K_{0}] , \bar{\ell}_{N}(\widehat{G}_{N}^{(\kappa^{\prime})})\to\mathcal{L}(G_{0}^{(\kappa^{\prime})}) in \mathbb{P}_{G_{0}} -probability as N\to\infty . DSC (height + likelihood). Given 1\ll\omega_{N}\ll(N/\log N)^{1/4} , define the dendrogram selection criterion \mathrm{DSC}_{N}^{(\kappa)}:=-\Big(h_{N}^{(\kappa)}+\omega_{N}\,\bar{\ell}_{N}^{(\kappa)}\Big) . A practical choice is \omega_{N}=\log N . Select \widehat{K}_{N}:=\arg\min_{\kappa\in\{2,\ldots,K\}}\mathrm{DSC}_{N}^{(\kappa)}. Theorem 7 (Consistency of DSC for SGMLMoE) . Assume K_{0}\geq 2 and the conditions of Theorem 6 hold. Then \widehat{K}_{N}\to K_{0} in \mathbb{P}_{G_{0}} -probability as N\to\infty . Interpretation of Why DSC Beats Pure Likelihood Penalties under Overfit. Heights detect structural redundancy: small h_{N}^{(\kappa)} indicates that the \kappa -atom model contains a pair of nearly indistinguishable atoms (often created by over-specification), which standard criteria (AIC/BIC/ICL) do not directly penalize. DSC combines this structural signal with fit, yielding a sweep-free selector aligned with the Voronoi geometry. 5 Numerical Experiment In this section, we conduct experiments on both synthetic and real-world datasets to systematically evaluate the effectiveness, stability, and practical utility of the proposed pipeline: batch MM training, Voronoi-loss diagnostics, and dendrogram-based selection/merging via DSC. On synthetic datasets , we first demonstrate that the proposed Batch MM algorithm converges reliably to the global optimum, while classical optimization methods, including EM and popular gradient-based optimizers, often become trapped at suboptimal stationary points, in both well-specified and over-parameterized regimes. We then empirically verify that the decay of the Voronoi loss aligns with the finite-sample convergence rates established in Theorem 5 , under both exact-fit and over-fit settings. Finally, we show that the proposed DSC-based merging strategy consistently outperforms standard model selection criteria (AIC, BIC, and ICL) in recovering the true number of experts, especially in small-sample regimes, while retaining consistency as sample size grows. On real-world datasets , we show that SGMLMoE trained via Batch MM yields superior predictive performance compared to strong baselines such as SVM and Random Forest. Moreover, DSC selects an appropriate number of experts that achieves an attractive trade-off between predictive accuracy and memory efficiency, while the resulting dendrogram provides an interpretable hierarchical summary of the fitted mixing measure via Figure 5 . Figure 2 : Convergence behavior of the Batch MM algorithm (parameter error versus sample size). (a) Exact-fit: empirical convergence of the Voronoi loss. (b) Over-fit: empirical convergence of the Voronoi loss. Figure 3 : Empirical convergence of the Voronoi loss under exact specification and over-specification. 5.1 Simulation Study Dataset. We consider a controlled synthetic setting with polynomial degree D=1 , number of experts K_{0}=2 , number of categories M=2 , and input dimension P=1 . The ground-truth gate parameters are {\bm{\omega}}_{0,0,0}={\bm{\omega}}_{1,0,0}={\bm{\omega}}_{1,1,0}=0 and {\bm{\omega}}_{0,0,1}=8 , and the expert parameters are {\bm{\upsilon}}_{0,0,0,0}={\bm{\upsilon}}_{1,0,1,0}={\bm{\upsilon}}_{0,1,0,0}={\bm{\upsilon}}_{1,1,1,0}=0 , {\bm{\upsilon}}_{0,0,1,0}={\bm{\upsilon}}_{0,1,1,0}=20 , and {\bm{\upsilon}}_{1,0,0,0}=-{\bm{\upsilon}}_{1,1,0,0}=10 . To generate the dataset ({\mathbb{X}},{\mathbb{Y}}) , we first fix a sample size N (which varies across experiments). We then sample inputs {\mathbb{X}}=\{x_{n}\}_{n=1}^{N} i.i.d. from the standard normal distribution \mathcal{N}(0,1) . Conditional on {\mathbb{X}} , we generate discrete outputs {\mathbb{Y}}=\{y_{n}\}_{n=1}^{N} according to the SGMLMoE model specified by ({\bm{\omega}}_{0},{\bm{\upsilon}}_{0}) . Initialization. In the well-specified setting ( K=K_{0} ), parameters are initialized by perturbing the ground-truth values {\bm{\omega}}_{0} and {\bm{\upsilon}}_{0} with additive Gaussian noise: each coordinate is set to the corresponding ground-truth value plus an independent \mathcal{N}(0,1) perturbation, scaled by a prescribed noise level. In the over-parameterized setting ( K>K_{0} ), we introduce additional expert components beyond the truth. For each extra cluster k\in\{K_{0}+1,\ldots,K\} , we set {\bm{\omega}}_{k,0,0}=0 , {\bm{\omega}}_{k,0,1}=8 , {\bm{\upsilon}}_{0,k,0,0}={\bm{\upsilon}}_{1,k,1,0}=0 , {\bm{\upsilon}}_{0,k,1,0}=20 , and {\bm{\upsilon}}_{1,k,0,0}=10 . The original K_{0} experts are initialized as in the well-specified case. This design ensures that over-parameterization genuinely creates near-duplicate atoms, which is precisely the regime targeted by our Voronoi analysis and merging strategy. Convergence of BatchMM. We study the convergence behavior of SGMLMoE trained using the proposed Batch MM algorithm, and compare it against classical optimization methods, including stochastic gradient descent (SGD), Adam, AdamW, RMSprop, and EM. In the well-specified setting, we vary the sample size from N=10^{2} to N=10^{5} . In the over-parameterized setting, we fix K=3 and vary N from 10^{2.75} to 10^{4} . Convergence is evaluated via the Frobenius norm between the estimated parameters ({\bm{\omega}},{\bm{\upsilon}}) and the ground-truth ({\bm{\omega}}_{0},{\bm{\upsilon}}_{0}) . As shown in Figure 2 , Batch MM consistently converges to the global minimum, while the competing methods are often markedly slower and, in the over-parameterized regime, frequently converge to suboptimal solutions as N grows. Convergence of Voronoi Loss. We empirically verify the convergence rates predicted by Theorem 5 . In the exact-fit setting ( Figure 3(a) ), varying the sample size from 10^{3} to 10^{5.5} yields a log–log slope consistent with the parametric rate N^{-1/2} . In the over-specified setting with K\in\{3,4\} and N\in[10^{2.5},10^{4}] , the fitted regression lines in Figure 3(b) match the predicted rates. For K=4 , the Frobenius error of individual components shows that one spurious component converges at a slower sub-parametric rate, empirically close to N^{-0.25} , in agreement with the rate predicted by Theorem 5 for over-covered Voronoi cells. In contrast, another overfitted component exhibits increasing error as N grows, reflecting a mild repulsion effect in over-parameterized models whereby redundant components are displaced away from the true parameters to resolve non-identifiability (here the relevant cells are {\mathbb{V}}_{k} ). Merging using DSC. Finally, we evaluate the effectiveness of our DSC-based merging strategy for recovering the correct number of experts. We fix K=4 and vary the sample size from N=10^{2} to 10^{4} . For each N , we repeat the experiment 20 times with different random seeds. We compare DSC against AIC, BIC, and ICL by recording (i) the frequency with which each criterion selects the correct number of experts and (ii) the average number of selected experts; results are shown in Figure 4 . Across sample sizes, DSC exhibits a higher correct-selection frequency than the baseline criteria. In particular, in the small-sample regime, DSC is notably more robust, while for sufficiently large N it consistently recovers the true number of experts, matching the consistency guarantees established in our theory. Figure 4 : DSC model selection performance. 5.2 Real Dataset Initialization. Since the true latent expert assignments are unknown in real data, we adopt a practical two-stage initialization strategy grounded in unsupervised learning. We first apply a classical clustering method (e.g., K-means or spectral clustering) to obtain preliminary groupings. We then construct initial SGMLMoE parameters from these assignments, providing a structured, data-informed initialization for Batch MM. Expert Estimation. We use a subset of 5,000 samples from a protein–protein interaction prediction dataset ( 94 ; 88 ) to train the proposed DSC method as well as baseline criteria including AIC, BIC, and ICL for selecting the number of experts, with the maximum capped at K=8 . A separate set of 5,000 samples is then used for model training and cross-validation across candidate values of K . Table 2 reports cross-validation performance as a function of K , along with the expert count selected by each criterion. DSC selects K=3 , achieving strong predictive performance while maintaining a compact model. In contrast, ICL favors a smaller model ( K=2 ), leading to reduced precision and F1 score, whereas AIC and BIC select larger models ( K=4 ), increasing complexity with only marginal performance gains. Overall, these results show that DSC achieves a better trade-off between accuracy and model complexity than classical criteria. Accuracy Comparision. We fix the number of experts to K=3 and train SGMLMoE using the Batch MM algorithm on 5,000 samples. We compare its performance with standard baseline methods, Naive Bayes, Random Forest, Support Vector Machine, and Logistic Regression, using recall, precision, and F1 score. As shown in Table 3 , SGMLMoE achieves the best overall performance among all methods, attaining the highest precision ( 0.74 ) and F1 score ( 0.80 ), while maintaining competitive recall ( 0.88 ). Naive Bayes exhibits lower precision and F1 score, indicating limited discriminative capacity, while Random Forest, SVM, Logistic Regression, and 3-Hidden Layer MLP improve upon NB but remain consistently below SGMLMoE in precision and overall F1. These results underscore the benefit of SGMLMoE in capturing heterogeneity, and they empirically support the practical stability of Batch MM relative to standard training baselines. Finally, Figure 5 displays the dendrogram of the fitted mixing measure obtained by Algorithm 2 , which provides an interpretable hierarchical view of the inferred expert structure. Figure 5 : Dendrogram of mixing measure. K Recall Precision F1 # P CoMs 1 0.54 \pm 0.04 0.43 \pm 0.03 0.48 \pm 0.003 6 2 0.72 \pm 0.02 0.61 \pm 0.02 0.66 \pm 0.003 12 ICL 3 0.88 \pm 0.01 0.73 \pm 0.02 0.80 \pm 0.003 18 DSC 4 0.88 \pm 0.01 0.74 \pm 0.01 0.80 \pm 0.002 24 AIC, BIC 5 0.89 \pm 0.01 0.75 \pm 0.01 0.81 \pm 0.002 30 6 0.91 \pm 0.01 0.76 \pm 0.01 0.83 \pm 0.001 36 7 0.92 \pm 0.01 0.76 \pm 0.01 0.83 \pm 0.001 42 8 0.92 \pm 0.01 0.76 \pm 0.01 0.83 \pm 0.001 48 Table 2 : Performance across number of experts K . Table 3 : Comparison with baseline methods. Method Recall Precision F1 SGMLMoE 0.88 \pm 0.012 0.74 \pm 0.010 0.80 \pm 0.002 NB 0.83 \pm 0.011 0.64 \pm 0.009 0.72 \pm 0.001 RF 0.85 \pm 0.012 0.68 \pm 0.008 0.76 \pm 0.001 SVM 0.85 \pm 0.010 0.69 \pm 0.009 0.76 \pm 0.002 LR 0.84 \pm 0.014 0.65 \pm 0.009 0.73 \pm 0.003 3HL-MLP 0.87 \pm 0.008 0.71 \pm 0.009 0.77 \pm 0.001 Conclusion and Perspectives. In summary, our experiments confirm the paper’s main message: Batch MM yields stable maximum-likelihood training for SGMLMoE in the full-data regime, while the Voronoi-loss/dendrogram view provides a principled remedy for over-parameterization. Empirically, Batch MM converges reliably, the Voronoi loss decays in line with the predicted finite-sample rates, and DSC consistently selects a compact expert count that preserves accuracy and improves calibration on protein–protein interaction prediction. Future work includes extending the MM surrogates to incremental/mini-batch settings, broadening the dendrogram theory to richer expert parameterizations and structured regularization, and developing robustness guarantees under contamination and covariate shift. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here Appendix Organization. This appendix collects material that supports the main paper at three complementary levels. Appendix A extends the discussion with empirical takeaways and concrete future directions. Appendix B provides implementation-ready details of the Batch MM routine for SGMLMoE, including the linear-algebra identities, the full algorithmic steps, and practical notes on conditioning and stopping. Appendix C documents experimental protocols (data preprocessing, evaluation, and reproducibility details). Finally, Appendix D contains complete proofs. Appendix A Extended Conclusion and Perspectives Overall, the numerical experiments support the main message of the paper: in SGMLMoE, the proposed Batch MM updates provide a stable and reliable maximum-likelihood training routine in the full-data regime, and the Voronoi-loss/dendrogram viewpoint yields a practically effective handle on over-parameterization. Empirically, we observe (i) monotone and repeatable likelihood improvement with Batch MM across well-specified and over-specified settings, (ii) decay of the Voronoi loss consistent with the finite-sample theory in both exact-fit and over-fit regimes, and (iii) robust recovery of the effective number of experts via DSC, which translates into compact models with competitive or improved predictive performance and probability calibration on protein–protein interaction data. Looking forward, several extensions are natural. First, it would be valuable to couple the present batch-MM stability guarantees with incremental or mini-batch variants to handle larger-scale datasets while preserving the monotone-surrogate structure. Second, extending the dendrogram/DSC analysis to richer expert parameterizations (e.g., deeper feature maps, structured sparsity, or context-dependent expert sharing) could broaden applicability while maintaining interpretability through the merging hierarchy. Third, the robustness observed under mild misspecification suggests studying contamination and covariate-shift regimes theoretically, with DSC-style structural penalties as a principled alternative to purely likelihood-based criteria. Finally, developing open-source, production-grade implementations and broader biological benchmarks (beyond PPI) would further clarify when and how hierarchical aggregation best trades off accuracy, calibration, and memory/compute in heterogeneous classification problems. Appendix B Detailed Batch MM Algorithm Inverse identities. For implementation, we use the block structure of Equation 5 . In particular, \displaystyle{\bm{B}}_{n,K-1}^{-1} \displaystyle=\left(\frac{4}{3}{\bm{I}}_{K-1}+\frac{8}{3(K-1)}\bm{1}_{K-1}\bm{1}_{K-1}^{\top}\right)\otimes\left(\sum_{n=1}^{N}\hat{{\bm{x}}}_{n}\hat{{\bm{x}}}_{n}^{\top}\right)^{-1}, (11) \displaystyle\left[{\bm{B}}^{K}_{n,M-1}\right]^{-1} \displaystyle={\bm{I}}_{K}\otimes{\bm{B}}_{n,M-1}^{-1}. (12) In addition, for any diagonal matrix \diag({\bm{a}}) with positive entries, we have \diag({\bm{a}})^{-1}=\diag(1/{\bm{a}}) , which we apply to {\bm{Z}}_{n}^{(t)} when needed. Batch MM Algorithm for SGMLMoE. Algorithm 2 implements a batch MM procedure for fitting SGMLMoE. At each iteration, it first computes responsibilities {\tau_{n,k}^{(t)}} for all samples and experts using the current gate and expert probabilities, and then constructs the batch sufficient-statistic–like terms ({\bm{s}}_{n}^{(t)},{\bm{r}}_{n}^{(t)}) together with the corresponding curvature blocks ({\bm{B}}_{n,K},{\bm{B}}_{n,M}) that define the quadratic MM surrogate in Equation 6 . These per-sample quantities are then aggregated over the batch to form global updates, including the diagonal weighting matrices {\bm{Z}}_{n}^{(t)} and the gradients of the log-sum-exp terms g_{n} (gate) and e_{n} (experts). Finally, the method performs closed-form MM updates for the gate and expert parameters by solving two preconditioned linear systems, yielding ({\bm{w}}^{(t+1)},{\bm{v}}^{(t+1)}) . By construction, the resulting iterates monotonically improve the log-likelihood, and the routine terminates when the likelihood increment falls below the tolerance \varepsilon (or when the maximum number of iterations is reached). Algorithm 2 Batch MM Algorithm for SGMLMoE 0: Batch data \{({\bm{x}}_{n},{\textnormal{y}}_{n})\}_{n=1}^{N} , numbers of experts/classes (K,M) , polynomial degree D , initialization {\bm{\theta}}^{(0)}=({\bm{w}}^{(0)},{\bm{v}}^{(0)}) , tolerance \varepsilon>0 , maximum iterations T . 0: MM iterates \{{\bm{\theta}}^{(t)}\}_{t\geq 0} . \For t=0,1,\dots,T-1 \State E-step (Responsibilities). \For n=1,\dots,N \State Form the feature vector \hat{{\bm{x}}}_{n}=\big[{x}_{n,1}^{0},\dots,{x}_{n,1}^{D},\dots,{x}_{n,P}^{0},\dots,{x}_{n,P}^{D}\big]^{\top} . \State Compute gate probabilities \{\mathsf{g}_{k}({\bm{w}}^{(t)}({\bm{x}}_{n}))\}_{k=1}^{K} . \State Compute expert probabilities \{\mathsf{e}_{k}({\textnormal{y}}_{n}=m;{\bm{v}}^{(t)}({\bm{x}}_{n}))\}_{m=1}^{M} for each k\in[K] . \State Compute posteriors \tau_{n,k}^{(t)}=\frac{\mathsf{g}_{k}({\bm{w}}^{(t)}({\bm{x}}_{n}))\,\mathsf{e}_{k}({\textnormal{y}}_{n};{\bm{v}}^{(t)}({\bm{x}}_{n}))}{\sum_{\ell=1}^{K}\mathsf{g}_{\ell}({\bm{w}}^{(t)}({\bm{x}}_{n}))\,\mathsf{e}_{\ell}({\textnormal{y}}_{n};{\bm{v}}^{(t)}({\bm{x}}_{n}))},\penalty\ k\in[K]. \State Build {\bm{s}}_{n}^{(t)} and {\bm{r}}_{n}^{(t)} (using \mathbbm{1}({\textnormal{y}}_{n},l) and the definitions in Equation 6 and the surrounding text). \State Build the curvature blocks {\bm{B}}_{n,K} and {\bm{B}}_{n,M} (as in Equation 5 ). \EndFor \State Aggregate Batch Quantities. \State {\bm{s}}^{(t)}=\sum_{n=1}^{N}{\bm{s}}_{n}^{(t)} , {\bm{r}}^{(t)}=\sum_{n=1}^{N}{\bm{r}}_{n}^{(t)} . \State {\bm{B}}_{n,K-1}=\sum_{n=1}^{N}{\bm{B}}_{n,K} , {\bm{B}}_{n,M-1}=\sum_{n=1}^{N}{\bm{B}}_{n,M} , {\bm{B}}^{K}_{n,M-1}={\bm{I}}_{K}\otimes{\bm{B}}_{n,M-1} . \For n=1,\dots,N \State Set {\bm{\tau}}_{n,:}^{(t)}=[\tau_{n,1}^{(t)},\dots,\tau_{n,K}^{(t)}]^{\top} and {\bm{Z}}_{n}^{(t)}=\diag\!\big({\bm{\tau}}_{n,:}^{(t)}\otimes\bm{1}_{P(D+1)}\big) . \EndFor \State Compute Gradients of Log-sum-exp terms. \State \nabla g({\bm{w}}^{(t)})=\sum_{n=1}^{N}\nabla g_{n}({\bm{w}}^{(t)}) , where g_{n}({\bm{w}})=\log\!\left(1+\sum_{k=1}^{K-1}\exp\!\left(w_{k}({\bm{x}}_{n})\right)\right) . \State For each n\in[N] , form \nabla e_{n}({\bm{v}}^{(t)})=\vect\big([\nabla e_{n}({\bm{c}}_{1}^{(t)}),\dots,\nabla e_{n}({\bm{c}}_{K}^{(t)})]\big) , where e_{n}({\bm{c}}_{k})=\log\!\left(1+\sum_{m=1}^{M-1}\exp\!\left({v}_{m,k}({\bm{x}}_{n})\right)\right) . \State M-step (Closed-form MM Updates). \State Update the gate parameters: {\bm{w}}^{(t+1)}={\bm{w}}^{(t)}+{\bm{B}}_{n,K-1}^{-1}\Big({\bm{s}}^{(t)}-\nabla g({\bm{w}}^{(t)})\Big). \State Update the expert parameters: \displaystyle{\bm{v}}^{(t+1)}={\bm{v}}^{(t)}+\Big(\sum_{n=1}^{N}{\bm{Z}}_{n}^{(t)}\,{\bm{B}}^{K}_{n,M-1}\Big)^{-1}\Big({\bm{r}}^{(t)}-\sum_{n=1}^{N}{\bm{Z}}_{n}^{(t)}\,\nabla e_{n}({\bm{v}}^{(t)})\Big). \State Set {\bm{\theta}}^{(t+1)}=({\bm{w}}^{(t+1)},{\bm{v}}^{(t+1)}) . \State Stopping Criterion. If |\mathcal{L}({\bm{\theta}}^{(t+1)})-\mathcal{L}({\bm{\theta}}^{(t)})|\leq\varepsilon , break . \EndFor Appendix C Experimental Details Real Dataset Pre-processing. We first use the dataset provided in ( 94 ; 88 ) to obtain positive interaction labels. Following 93 , we adopt protein sequence data as inputs, collected from 13 , yielding 9,760 protein sequences. Each sequence is encoded as a 20-dimensional amino-acid composition vector using the ProtLearn library. We then restrict the dataset to proteins appearing in the DIP database, resulting in 2,321 proteins with retained identifiers and corresponding sequence encodings. To construct labeled interaction samples, we select n protein pairs with verified interactions in DIP and compute the element-wise absolute difference between their sequence encodings, producing 20-dimensional feature vectors labeled as 1. Similarly, we sample n non-interacting protein pairs absent from DIP and construct their feature vectors in the same manner, assigning label 0. The positive and negative samples are concatenated to form a 2n\times 20 feature matrix with an associated binary label vector, and the resulting dataset is randomly shuffled prior to training. Monotonicity of Maximum Loglikelihood Estimator. Classical optimization methods such as SGD and Adam often lead to unstable MLE behavior in mixture or Mixture-of-Experts models. This instability arises from the highly non-convex likelihood landscape, where inappropriate step sizes can cause oscillations or divergence, making performance sensitive to careful learning-rate tuning. Such tuning is problem-dependent and can be computationally costly, limiting robustness and reproducibility. In contrast, our proposed method avoids these issues by adopting an MM-based optimization scheme that guarantees monotonic improvement of the MLE objective, as established in Theorem 2 . This monotonicity ensures stable and predictable optimization dynamics. We empirically illustrate this behavior by plotting the MLE trajectory across iterations in Figure 6 . As shown, our method exhibits a smooth, non-decreasing likelihood progression, in contrast to the fluctuating behavior observed under gradient-based optimizers. The experiment is conducted on the synthetic dataset described in Section 5.2 with N=1000 samples. Figure 6 : Evolution of the maximum log-likelihood for SGMLMoE over the first 100 optimization iterations with N=1000 . Surrogate function construction plays a pivotal role in optimization techniques such as MM algorithms. By leveraging mathematical inequalities, it is possible to approximate or bound complex objective functions with simpler, more tractable alternatives. Below are several fundamental inequalities commonly employed in the development of surrogate functions: Appendix D Proofs of Main Results D.1 Technical Contributions Following the MM paradigm, we first propose a novel surrogate objective for SGMLMoE in Theorem 1 and rigorously show that it satisfies the monotonicity property required by the MM algorithm in Theorem 2 . Building on this foundation, we turn to model merging and substantially modify the proof of Theorem 3.1 in 70 to accommodate our proposed Voronoi metric and the SGMLMoE framework, yielding the key inequality in Theorem 3 . Leveraging this result, we then follow the argument of Lemma 2 in 19 (see also 95 ; 34 ) to derive an analogous monotone chain property for our model in Theorem 4 . Finally, we adapt the proofs of Theorems 1–3 in 19 for mixture models to our Voronoi loss and SGMLMoE setting, leading to theoretical guarantees on the convergence rate, model selection consistency, and merge convergence in Theorem 5 , Theorem 6 , and Theorem 7 , respectively. D.2 Proof of Theorem 1 We derive the surrogate bound in Equation 6 by combining (i) a variational upper bound based on responsibilities (E-step) and (ii) a uniform quadratic majorization of the baseline log-sum-exp terms (MM step). We first recall the notation blocks used throughout this proof. Batch Sufficient-statistics Notation. For each n\in[N] , define the gating-design vector in {\mathbb{R}}^{(K-1)P(D+1)} : \displaystyle{\bm{s}}_{n}^{(t)}=\big[\tau_{n,1}^{(t)}{x}_{n,1}^{0},\dots,\tau_{n,1}^{(t)}{x}_{n,1}^{D},\dots,\tau_{n,K-1}^{(t)}{x}_{n,P}^{D}\big]^{\top}. (13) Define the expert-design vector {\bm{r}}_{n}^{(t)} by stacking r_{n,d,l,p,k}^{(t)}=\mathbbm{1}({\textnormal{y}}_{n},l)\,\tau_{n,k}^{(t)}\,{x}_{n,p}^{d} , \penalty\ d\in\{0,\ldots,D\},\penalty\ l\in[M],\penalty\ p\in[P],\penalty\ k\in[K], into {\bm{r}}_{n}^{(t)}=\big[r_{n,0,1,1,1}^{(t)},\dots,r_{n,D,1,1,1}^{(t)},\dots,r_{n,D,M,1,1}^{(t)},\dots,r_{n,D,M,P,K}^{(t)}\big]^{\top}. We also define the class-parameter blocks for each expert k by {\bm{c}}_{k}=\big[{\bm{\upsilon}}_{1,k,0,1},\dots,{\bm{\upsilon}}_{1,k,D,1},\dots,{\bm{\upsilon}}_{M,k,D,1},\dots,{\bm{\upsilon}}_{M,k,D,P}\big]^{\top},\qquad{\bm{\upsilon}}=\vect([{\bm{c}}_{1},\dots,{\bm{c}}_{K}]). Finally, we aggregate the batch statistics as {\bm{s}}^{(t)}=\sum_{n=1}^{N}{\bm{s}}_{n}^{(t)},\qquad{\bm{r}}^{(t)}=\sum_{n=1}^{N}{\bm{r}}_{n}^{(t)}. Quadratic Majorizers for the Gate and Experts. For k\in[K-1] , let w_{k}({\bm{x}})=\sum_{d=0}^{D}{\bm{\omega}}_{k,d}^{\top}{\bm{x}}^{d} and define the gate log-sum-exp term g_{n}({\bm{w}})=\log\!\left(1+\sum_{k=1}^{K-1}\exp\!\left(w_{k}({\bm{x}}_{n})\right)\right). For experts, define for each k\in[K] the multinomial log-sum-exp term e_{n}({\bm{c}}_{k})=\log\!\left(1+\sum_{m=1}^{M-1}\exp\!\left({v}_{m,k}({\bm{x}}_{n})\right)\right). For any K\in{\mathbb{N}} , we use the quadratic majorizers based on the matrices \displaystyle{\bm{B}}_{n,K}=\left(\frac{3}{4}{\bm{I}}_{K-1}-\frac{\bm{1}_{K-1}\bm{1}_{K-1}^{\top}}{2(K-1)}\right)\otimes\hat{{\bm{x}}}_{n}\hat{{\bm{x}}}_{n}^{\top}, (14) and similarly {\bm{B}}_{n,M} is defined by replacing K with M in Equation 14 . For compactness, introduce the centered increments \bar{{\bm{w}}}_{t}={\bm{w}}-{\bm{w}}^{(t)} and \bar{{\bm{c}}}_{k,t}={\bm{c}}_{k}-{\bm{c}}_{k}^{(t)} . (Here \hat{{\bm{x}}}_{n}=[{x}_{n,1}^{0},\dots,{x}_{n,1}^{D},\dots,{x}_{n,P}^{0},\dots,{x}_{n,P}^{D}]^{\top} .) Step 1: Variational Upper Bound via Responsibilities. For each n\in[N] , define a_{n,k}({\bm{\theta}}):=\mathsf{g}_{k}({\bm{w}}({\bm{x}}_{n}))\,\mathsf{e}_{k}({\textnormal{y}}_{n};{\bm{v}}({\bm{x}}_{n})),\qquad s_{\bm{\theta}}({\textnormal{y}}_{n}\mid{\bm{x}}_{n})=\sum_{k=1}^{K}a_{n,k}({\bm{\theta}}). By the log-sum inequality (equivalently, Jensen’s inequality / KL nonnegativity), for any {\bm{\tau}}_{n,:}\in\Delta_{K} , -\log s_{\bm{\theta}}({\textnormal{y}}_{n}\mid{\bm{x}}_{n})\leq-\sum_{k=1}^{K}\tau_{n,k}\log a_{n,k}({\bm{\theta}})+\sum_{k=1}^{K}\tau_{n,k}\log\tau_{n,k}. (15) Choosing \tau_{n,k}=\tau_{n,k}^{(t)}:=a_{n,k}({\bm{\theta}}^{(t)})/\sum_{\ell}a_{n,\ell}({\bm{\theta}}^{(t)}) makes Equation 15 tight at {\bm{\theta}}={\bm{\theta}}^{(t)} . Expanding \log a_{n,k}({\bm{\theta}})=\log\mathsf{g}_{k}({\bm{w}}({\bm{x}}_{n}))+\log\mathsf{e}_{k}({\textnormal{y}}_{n};{\bm{v}}({\bm{x}}_{n})) yields two convex log-sum-exp terms and linear contributions in the scores: \displaystyle-\sum_{k=1}^{K}\tau_{n,k}^{(t)}\log\mathsf{g}_{k}({\bm{w}}({\bm{x}}_{n})) \displaystyle=g_{n}({\bm{w}})-\sum_{k=1}^{K-1}\tau_{n,k}^{(t)}\,w_{k}({\bm{x}}_{n}), (16) \displaystyle-\sum_{k=1}^{K}\tau_{n,k}^{(t)}\log\mathsf{e}_{k}({\textnormal{y}}_{n};{\bm{v}}({\bm{x}}_{n})) \displaystyle=\sum_{k=1}^{K}\tau_{n,k}^{(t)}\,e_{n}({\bm{c}}_{k})-\sum_{k=1}^{K}\tau_{n,k}^{(t)}\sum_{l=1}^{M}\mathbbm{1}({\textnormal{y}}_{n},l)\,{v}_{l,k}({\bm{x}}_{n}). (17) The linear terms in Equations 16 and 17 match the sufficient-statistics inner products {\bm{w}}^{\top}{\bm{s}}_{n}^{(t)} and {\bm{\upsilon}}^{\top}{\bm{r}}_{n}^{(t)} under the stacking conventions above (so that w_{k}({\bm{x}}_{n})={\bm{\omega}}_{k}^{\top}\hat{{\bm{x}}}_{n} and {v}_{l,k}({\bm{x}}_{n})={\bm{\upsilon}}_{l,k}^{\top}\hat{{\bm{x}}}_{n} ). Collecting Equations 15 , 16 and 17 and summing over n gives \displaystyle-\log s_{\bm{\theta}}({\textnormal{y}}\mid{\mathbf{x}})\leq C^{(t)}-\sum_{n=1}^{N}{\bm{w}}^{\top}{\bm{s}}_{n}^{(t)}-\sum_{n=1}^{N}{\bm{\upsilon}}^{\top}{\bm{r}}_{n}^{(t)}+\sum_{n=1}^{N}g_{n}({\bm{w}})+\sum_{n=1}^{N}\sum_{k=1}^{K}\tau_{n,k}^{(t)}\,e_{n}({\bm{c}}_{k}), (18) where C^{(t)}:=\sum_{n=1}^{N}\sum_{k=1}^{K}\tau_{n,k}^{(t)}\log\tau_{n,k}^{(t)} does not depend on {\bm{\theta}} . Step 2: Quadratic Majorization of g_{n} and e_{n} . Both g_{n} and e_{n} are baseline log-sum-exp functions composed with affine score maps. A standard calculation gives \nabla^{2}g_{n}({\bm{w}})=\big(\diag(\hat{{\bm{p}}}_{n})-\hat{{\bm{p}}}_{n}\hat{{\bm{p}}}_{n}^{\top}\big)\otimes\hat{{\bm{x}}}_{n}\hat{{\bm{x}}}_{n}^{\top}, where \hat{{\bm{p}}}_{n} are the baseline softmax probabilities over k\in[K-1] . Applying Proposition 1 with q=K-1 (gate) and q=M-1 (expert), and then lifting the resulting Hessian bound through the linear score maps via Lemma 1 , yields the quadratic majorizers in Equation 5 . Indeed, by Lemma 1 (with q=K-1 ) and Proposition 1 , \nabla^{2}g_{n}({\bm{w}})\ \preceq\ {\bm{B}}_{n,K}, with {\bm{B}}_{n,K} defined in Equation 14 . Therefore, the second-order Taylor formula along the segment between {\bm{w}}^{(t)} and {\bm{w}} yields the global quadratic upper bound g_{n}({\bm{w}})\leq g_{n}({\bm{w}}^{(t)})+\bar{{\bm{w}}}_{t}^{\top}\nabla g_{n}({\bm{w}}^{(t)})+\frac{1}{2}\,\bar{{\bm{w}}}_{t}^{\top}{\bm{B}}_{n,K}\bar{{\bm{w}}}_{t},\qquad\forall{\bm{w}}. (19) The same argument applies to e_{n}({\bm{c}}_{k}) (with q=M-1 ), giving for each k\in[K] , e_{n}({\bm{c}}_{k})\leq e_{n}({\bm{c}}_{k}^{(t)})+\bar{{\bm{c}}}_{k,t}^{\top}\nabla e_{n}({\bm{c}}_{k}^{(t)})+\frac{1}{2}\,\bar{{\bm{c}}}_{k,t}^{\top}{\bm{B}}_{n,M}\bar{{\bm{c}}}_{k,t},\qquad\forall{\bm{c}}_{k}. (20) Step 3: Surrogate Bound. Plug Equations 19 and 20 into Equation 18 and absorb all terms that do not depend on {\bm{\theta}} into C^{(t)} . This yields exactly the surrogate inequality stated in Equation 6 , namely -\log s_{{\bm{\theta}}}({\textnormal{y}}\mid{\mathbf{x}})\leq-\log\!\big[g({\bm{\theta}},{\bm{x}}_{[N]},{\textnormal{y}}_{[N]};{\bm{\theta}}^{(t)})\big], with the explicit quadratic form in Equation 6 . This completes the proof. Lemma 1 (Kronecker product preserves Loewner order) . If {\bm{A}}\preceq{\bm{B}} and {\bm{C}}\succeq\mathbf{0} are symmetric, then {\bm{A}}\otimes{\bm{C}}\preceq{\bm{B}}\otimes{\bm{C}}. Proof of Lemma 1 . Since {\bm{A}}\preceq{\bm{B}} , we have {\bm{B}}-{\bm{A}}\succeq\mathbf{0} . For any vector {\bm{x}} , {\bm{x}}^{\top}\big(({\bm{B}}-{\bm{A}})\otimes{\bm{C}}\big){\bm{x}}=\big\|\big(({\bm{B}}-{\bm{A}})^{1/2}\otimes{\bm{C}}^{1/2}\big){\bm{x}}\big\|^{2}\geq 0, so ({\bm{B}}-{\bm{A}})\otimes{\bm{C}}\succeq\mathbf{0} , which is equivalent to {\bm{A}}\otimes{\bm{C}}\preceq{\bm{B}}\otimes{\bm{C}} . ∎ Proposition 1 (Uniform bound for baseline-softmax Hessian) . Let q\in{\mathbb{N}} and let \tilde{{\bm{p}}}\in(0,1)^{q+1} satisfy \sum_{j=1}^{q+1}\tilde{p}_{j}=1 . Write \hat{{\bm{p}}}=[\tilde{p}_{1},\ldots,\tilde{p}_{q}]^{\top} and {\bm{\Lambda}}=\diag(\hat{{\bm{p}}}) . Then {\bm{\Lambda}}-\hat{{\bm{p}}}\hat{{\bm{p}}}^{\top}\preceq{\bm{A}}_{q}:=\frac{3}{4}{\bm{I}}_{q}-\frac{\bm{1}_{q}\bm{1}_{q}^{\top}}{2q}. Proof of Proposition 1 . In this proof, we use the following definitions: \bm{1}_{q+1}:=(1,\dots,1)^{\top}\in{\mathbb{R}}^{q+1} , and \bm{1}_{q+1}^{\perp}:=\left\{{\bm{u}}\in{\mathbb{R}}^{q+1}:\ \bm{1}_{q+1}^{\top}{\bm{u}}=0\right\}. Step 1: Full Simplex Covariance Bound. Define the (q\!+\!1)\times(q\!+\!1) covariance-type matrix {\bm{W}}(\tilde{{\bm{p}}})=\diag(\tilde{{\bm{p}}})-\tilde{{\bm{p}}}\tilde{{\bm{p}}}^{\top}. Let {\mathbf{z}}\in\{{\mathb{e}}_{1},\ldots,{\mathb{e}}_{q+1}\} be a one-hot random vector with \mathbb{E}[{\mathbf{z}}]=\tilde{{\bm{p}}} . Then {\bm{W}}(\tilde{{\bm{p}}})=\mathrm{Cov}({\mathbf{z}}) and for any {\bm{u}} , {\bm{u}}^{\top}{\bm{W}}(\tilde{{\bm{p}}}){\bm{u}}=\mathrm{Var}({\bm{u}}^{\top}{\mathbf{z}}). Since {\bm{u}}^{\top}{\mathbf{z}} takes values in \{{\bm{u}}_{1},\ldots,{\bm{u}}_{q+1}\} , its range is at most \max_{i,j}|{\bm{u}}_{i}-{\bm{u}}_{j}|=\max_{i,j}|{\bm{u}}^{\top}({\mathb{e}}_{i}-{\mathb{e}}_{j})|\leq\|{\bm{u}}\|\max_{i,j}\|{\mathb{e}}_{i}-{\mathb{e}}_{j}\|=\sqrt{2}\,\|{\bm{u}}\|. By Popoviciu’s inequality, \mathrm{Var}({\bm{u}}^{\top}{\mathbf{z}})\leq\frac{1}{4}(\text{range})^{2}\leq\frac{1}{2}\|{\bm{u}}\|^{2} . Hence {\bm{W}}(\tilde{{\bm{p}}})\preceq\frac{1}{2}{\bm{I}}_{q+1} . Moreover, {\bm{W}}(\tilde{{\bm{p}}})\bm{1}_{q+1}=\mathbf{0} , so {\bm{W}}(\tilde{{\bm{p}}}) is supported on \bm{1}_{q+1}^{\perp} , and therefore {\bm{W}}(\tilde{{\bm{p}}})\preceq\frac{1}{2}\Big({\bm{I}}_{q+1}-\frac{\bm{1}_{q+1}\bm{1}_{q+1}^{\top}}{q+1}\Big). Step 2: Take the Principal Submatrix. The top-left q\times q principal submatrix of {\bm{W}}(\tilde{{\bm{p}}}) is exactly {\bm{\Lambda}}-\hat{{\bm{p}}}\hat{{\bm{p}}}^{\top} . Taking the corresponding principal submatrix of the right-hand side yields {\bm{\Lambda}}-\hat{{\bm{p}}}\hat{{\bm{p}}}^{\top}\preceq\frac{1}{2}\Big({\bm{I}}_{q}-\frac{\bm{1}_{q}\bm{1}_{q}^{\top}}{q+1}\Big). Step 3: Loosen to {\bm{A}}_{q} Used in Equation 5 . It remains to show \frac{1}{2}\big({\bm{I}}_{q}-\frac{\bm{1}\bm{1}^{\top}}{q+1}\big)\preceq\frac{3}{4}{\bm{I}}_{q}-\frac{\bm{1}\bm{1}^{\top}}{2q} . Their difference equals \Big(\frac{3}{4}-\frac{1}{2}\Big){\bm{I}}_{q}-\Big(\frac{1}{2q}-\frac{1}{2(q+1)}\Big)\bm{1}_{q}\bm{1}_{q}^{\top}=\frac{1}{4}{\bm{I}}_{q}-\frac{1}{2q(q+1)}\bm{1}_{q}\bm{1}_{q}^{\top}, whose eigenvalues are \frac{1}{4} on \bm{1}_{q}^{\perp} and \frac{1}{4}-\frac{q}{2q(q+1)}=\frac{q-1}{4(q+1)}\geq 0 along \bm{1}_{q} . Thus the difference is positive semidefinite, proving the claim. ∎ D.3 Proof of Theorem 2 : Verification of MM Properties for Algorithm 1 Throughout this proof we use the notation and building blocks introduced in Section D.2 , in particular the batch sufficient-statistics {\bm{s}}_{n}^{(t)} , {\bm{r}}_{n}^{(t)} in Equation 4 and the quadratic curvature matrices {\bm{B}}_{n,K} , {\bm{B}}_{n,M} in Equation 5 . Let \mathcal{L}({\bm{\theta}})=\sum_{n=1}^{N}\log s_{{\bm{\theta}}}({\textnormal{y}}_{n}\mid{\bm{x}}_{n}) be the observed-data log-likelihood. Step 1: MM Majorization and Tangency (from Theorem 1 ). Define the batch surrogate \mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)}) for -\mathcal{L}({\bm{\theta}}) by the right-hand side of Equation 6 (absorbing all terms independent of {\bm{\theta}} into a constant). By Theorem 1 , for every iterate {\bm{\theta}}^{(t)} the surrogate satisfies the MM conditions -\mathcal{L}({\bm{\theta}})\leq\mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)})\ \ \forall{\bm{\theta}},\qquad-\mathcal{L}({\bm{\theta}}^{(t)})=\mathcal{S}({\bm{\theta}}^{(t)},{\bm{\theta}}^{(t)}). (21) Step 2: Exact Minimization of the Surrogate by Algorithm 1 . Fix {\bm{\theta}}^{(t)} and use the centered increments \bar{{\bm{w}}}_{t}={\bm{w}}-{\bm{w}}^{(t)} and \bar{{\bm{c}}}_{k,t}={\bm{c}}_{k}-{\bm{c}}_{k}^{(t)} as in Section D.2 . Since the surrogate is the sum of (i) a gate quadratic term in {\bm{w}} and (ii) K independent expert quadratics in \{{\bm{c}}_{k}\}_{k=1}^{K} , it decomposes as \mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)})=C^{(t)}+\mathcal{S}_{g}({\bm{w}};{\bm{\theta}}^{(t)})+\sum_{k=1}^{K}\mathcal{S}_{e,k}({\bm{c}}_{k};{\bm{\theta}}^{(t)}), where C^{(t)} does not depend on {\bm{\theta}} , and (up to additive constants) \displaystyle\mathcal{S}_{g}({\bm{w}};{\bm{\theta}}^{(t)}) \displaystyle=-{\bm{w}}^{\top}{\bm{s}}^{(t)}+\sum_{n=1}^{N}\Big\{\nabla g_{n}({\bm{w}}^{(t)})^{\top}\bar{{\bm{w}}}_{t}+\frac{1}{2}\,\bar{{\bm{w}}}_{t}^{\top}{\bm{B}}_{n,K}\bar{{\bm{w}}}_{t}\Big\}, (22) \displaystyle\mathcal{S}_{e,k}({\bm{c}}_{k};{\bm{\theta}}^{(t)}) \displaystyle=-{\bm{c}}_{k}^{\top}{\bm{r}}_{k}^{(t)}+\sum_{n=1}^{N}\tau_{n,k}^{(t)}\Big\{\nabla e_{n}({\bm{c}}_{k}^{(t)})^{\top}\bar{{\bm{c}}}_{k,t}+\frac{1}{2}\,\bar{{\bm{c}}}_{k,t}^{\top}{\bm{B}}_{n,M}\bar{{\bm{c}}}_{k,t}\Big\}. (23) Here {\bm{s}}^{(t)}=\sum_{n=1}^{N}{\bm{s}}_{n}^{(t)} , {\bm{r}}^{(t)}=\sum_{n=1}^{N}{\bm{r}}_{n}^{(t)} , and {\bm{r}}_{k}^{(t)} is the block of {\bm{r}}^{(t)} corresponding to expert k (under the stacking convention defining {\bm{\upsilon}}=\vect([{\bm{c}}_{1},\dots,{\bm{c}}_{K}]) ). Introduce the aggregated curvatures and gradients {\bm{B}}_{K-1}:=\sum_{n=1}^{N}{\bm{B}}_{n,K},\qquad{\bm{B}}_{M-1,k}^{(t)}:=\sum_{n=1}^{N}\tau_{n,k}^{(t)}\,{\bm{B}}_{n,M}, \nabla g({\bm{w}}^{(t)}):=\sum_{n=1}^{N}\nabla g_{n}({\bm{w}}^{(t)}),\qquad\nabla e_{k}^{(t)}:=\sum_{n=1}^{N}\tau_{n,k}^{(t)}\,\nabla e_{n}({\bm{c}}_{k}^{(t)}). Then Equation 22 is a (strictly) convex quadratic in {\bm{w}} whenever {\bm{B}}_{K-1}\succ\mathbf{0} , and differentiating yields \nabla_{{\bm{w}}}\mathcal{S}_{g}({\bm{w}};{\bm{\theta}}^{(t)})=-{\bm{s}}^{(t)}+\nabla g({\bm{w}}^{(t)})+{\bm{B}}_{K-1}({\bm{w}}-{\bm{w}}^{(t)}). Setting this to \mathbf{0} gives the unique minimizer {\bm{w}}^{(t+1)}={\bm{w}}^{(t)}+{\bm{B}}_{K-1}^{-1}\big({\bm{s}}^{(t)}-\nabla g({\bm{w}}^{(t)})\big), (24) which matches the gate update in Algorithm 1 (up to the same aggregation notation). Similarly, for each k\in[K] , Equation 23 is a convex quadratic in {\bm{c}}_{k} and \nabla_{{\bm{c}}_{k}}\mathcal{S}_{e,k}({\bm{c}}_{k};{\bm{\theta}}^{(t)})=-{\bm{r}}_{k}^{(t)}+\nabla e_{k}^{(t)}+{\bm{B}}_{M-1,k}^{(t)}({\bm{c}}_{k}-{\bm{c}}_{k}^{(t)}). Setting this to \mathbf{0} yields {\bm{c}}_{k}^{(t+1)}={\bm{c}}_{k}^{(t)}+\big({\bm{B}}_{M-1,k}^{(t)}\big)^{-1}\big({\bm{r}}_{k}^{(t)}-\nabla e_{k}^{(t)}\big),\qquad k\in[K], (25) equivalent to the expert update in Algorithm 1 (the algorithm writes this in a vectorized form). Since \mathcal{S} decomposes across the gate block and expert blocks, the concatenation {\bm{\theta}}^{(t+1)}=({\bm{w}}^{(t+1)},{\bm{\upsilon}}^{(t+1)}) satisfies {\bm{\theta}}^{(t+1)}\in\arg\min_{{\bm{\theta}}}\ \mathcal{S}({\bm{\theta}},{\bm{\theta}}^{(t)}). (26) If some curvature matrix is singular, the same first-order conditions hold using the Moore–Penrose pseudoinverse, or by adding a small ridge \lambda{\bm{I}} ; in either case the MM validity in Equation 21 is unchanged. Step 3: Monotonicity (MM Ascent). By exact minimization Equation 26 , \mathcal{S}({\bm{\theta}}^{(t+1)},{\bm{\theta}}^{(t)})\leq\mathcal{S}({\bm{\theta}}^{(t)},{\bm{\theta}}^{(t)}). By tangency in Equation 21 , \mathcal{S}({\bm{\theta}}^{(t)},{\bm{\theta}}^{(t)})=-\mathcal{L}({\bm{\theta}}^{(t)}) . By majorization in Equation 21 evaluated at {\bm{\theta}}={\bm{\theta}}^{(t+1)} , -\mathcal{L}({\bm{\theta}}^{(t+1)})\leq\mathcal{S}({\bm{\theta}}^{(t+1)},{\bm{\theta}}^{(t)}) . Combining the three displays yields -\mathcal{L}({\bm{\theta}}^{(t+1)})\leq-\mathcal{L}({\bm{\theta}}^{(t)})\qquad\Longleftrightarrow\qquad\mathcal{L}({\bm{\theta}}^{(t+1)})\geq\mathcal{L}({\bm{\theta}}^{(t)}), which proves Theorem 2 . D.4 Proof of Theorem 3 This result extends the inverse-inequality technique developed in 70 to our SGMLMoE parameterization with polynomial score functions. Throughout, we use the lifted feature \hat{\bm{x}}=\phi_{D}({\bm{x}})\in\widehat{{\mathbb{X}}}\subset{\mathbb{R}}^{PD} and the identifiable mixing-measure representation introduced in Section 4.2 – Section 4.3 . Recall that for G=\sum_{l=1}^{K}\pi_{l}\delta_{\eta_{l}}\in\mathcal{O}_{K}({\mathbb{T}}) and G_{0}=\sum_{k=1}^{K_{0}}\pi_{k}^{0}\delta_{\eta_{k}^{0}}\in\mathcal{O}_{K_{0}}({\mathbb{T}}) , we write s_{G}({y}\mid{\bm{x}})=\sum_{l=1}^{K}\mathsf{g}_{l}({\bm{x}};G)\,\mathsf{e}_{l}({y}\mid{\bm{x}};G) , and we use the covariate-averaged TV discrepancy \mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})=\mathbb{E}_{{\mathbf{x}}}\!\big[\mathcal{D}_{\mathrm{TV}}(s_{G}(\cdot\mid{\mathbf{x}}),s_{G_{0}}(\cdot\mid{\mathbf{x}}))\big] . We prove the local and global parts separately. Local Structure. We show the following local inverse inequality: \liminf_{\varepsilon\to 0}\ \inf_{\begin{subarray}{c}G\in\mathcal{O}_{K}({\mathbb{T}}):\\ \mathcal{D}_{\mathrm{V}}(G,G_{0})\leq\varepsilon\end{subarray}}\frac{\mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})}{\mathcal{D}_{\mathrm{V}}(G,G_{0})}\ >\ 0. (27) Assume by contradiction that Equation 27 fails. Then there exists a sequence G^{(r)}=\sum_{l=1}^{K}\pi_{l}^{(r)}\delta_{\eta_{l}^{(r)}}\in\mathcal{O}_{K}({\mathbb{T}}) such that \mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})\to 0\qquad\text{and}\qquad\frac{\mathcal{D}_{\mathrm{TV}}(s_{G^{(r)}},s_{G_{0}})}{\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})}\to 0\qquad(r\to\infty). (28) For r large, the Voronoi partition \{{\mathbb{V}}_{k}\}_{k=1}^{K_{0}} is well-defined as in Section 4.3 . For l\in{\mathbb{V}}_{k} , write the local differences \Delta\hat{\bm{\omega}}_{lk}^{(r)}=\hat{\bm{\omega}}_{l}^{(r)}-\hat{\bm{\omega}}_{k}^{0} , \Delta\bar{\upsilon}_{lkm}^{(r)}=\bar{\upsilon}_{m,l}^{(r)}-\bar{\upsilon}_{m,k}^{0} , \Delta\hat{\bm{\upsilon}}_{lkm}^{(r)}=\hat{\bm{\upsilon}}_{m,l}^{(r)}-\hat{\bm{\upsilon}}_{m,k}^{0} for m\in[M-1] , and \Delta\pi_{k}^{(r)}=\sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}-\pi_{k}^{0} . By Equation 7 and \mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})\to 0 , all these quantities vanish. Step 1: Finite Taylor Representation. Define the true gate normalizer Z_{0}(\hat{\bm{x}}):=\sum_{k=1}^{K_{0}}\pi_{k}^{0}\exp\!\big((\hat{\bm{\omega}}_{k}^{0})^{\top}\hat{\bm{x}}\big). For each class m\in[M] , consider the rescaled difference T_{r}(m,\hat{\bm{x}}):=Z_{0}(\hat{\bm{x}})\,\Big(s_{G^{(r)}}(m\mid{\bm{x}})-s_{G_{0}}(m\mid{\bm{x}})\Big). (29) Using the softmax form for \mathsf{g}_{l}(\cdot;G) and the multinomial-logistic form for \mathsf{e}_{l}(\cdot\mid\cdot;G) , a Taylor expansion around each true atom \eta_{k}^{0} yields the decomposition T_{r}(m,\hat{\bm{x}})=\sum_{k=1}^{K_{0}}\ \sum_{\xi\in\Xi_{k}}A_{k,\xi}^{(r)}\ \Psi_{k,\xi}(m,\hat{\bm{x}})\ +\ R_{r}(m,\hat{\bm{x}}), (30) where: (i) \{\Psi_{k,\xi}(m,\hat{\bm{x}})\} is a finite collection of analytic basis functions of the form \hat{\bm{x}}^{\alpha}\exp\!\big((\hat{\bm{\omega}}_{k}^{0})^{\top}\hat{\bm{x}}\big)\,\partial^{\beta}\mathsf{e}(\,m\mid\hat{\bm{x}};\bar{\upsilon}_{k}^{0},\hat{\bm{\upsilon}}_{k}^{0}), with multi-indices (\alpha,\beta) of bounded total order (first order when |{\mathbb{V}}_{k}|=1 , and up to second order when |{\mathbb{V}}_{k}|>1 ); (ii) the coefficients A_{k,\xi}^{(r)} are explicit finite sums over l\in{\mathbb{V}}_{k} of monomials in \Delta\hat{\bm{\omega}}_{lk}^{(r)} , \Delta\bar{\upsilon}_{lkm}^{(r)} , \Delta\hat{\bm{\upsilon}}_{lkm}^{(r)} , and \Delta\pi_{k}^{(r)} , weighted by \pi_{l}^{(r)} ; (iii) the remainder satisfies \sup_{m\in[M]}\ \mathbb{E}_{{\mathbf{x}}}\!\big[\,|R_{r}(m,\phi_{D}({\mathbf{x}}))|\,\big]\ =\ o\!\big(\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})\big),\qquad(r\to\infty). (31) Step 2: Some Normalized Coefficient Must be Non-vanishing. Let m_{r}:=\max_{k,\xi}\ \frac{|A_{k,\xi}^{(r)}|}{\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})}. We claim \liminf_{r\to\infty}m_{r}>0 . If instead m_{r}\to 0 , then all coefficients in Equation 30 would satisfy |A_{k,\xi}^{(r)}|=o(\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})) . But the collection of coefficients \{A_{k,\xi}^{(r)}\} contains, by construction, the same first- and second-order weighted increments that appear in Equation 7 : for cells |{\mathbb{V}}_{k}|=1 it contains the weighted linear terms \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}\|\Delta\hat{\bm{\omega}}_{lk}^{(r)}\| , \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}|\Delta\bar{\upsilon}_{lkm}^{(r)}| , \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}\|\Delta\hat{\bm{\upsilon}}_{lkm}^{(r)}\| , and for cells |{\mathbb{V}}_{k}|>1 it contains the weighted quadratic terms \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}\|\Delta\hat{\bm{\omega}}_{lk}^{(r)}\|^{2} , \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}|\Delta\bar{\upsilon}_{lkm}^{(r)}|^{2} , \sum_{l\in{\mathbb{V}}_{k}}\pi_{l}^{(r)}\|\Delta\hat{\bm{\upsilon}}_{lkm}^{(r)}\|^{2} , as well as the mass discrepancy terms |\Delta\pi_{k}^{(r)}| . Therefore m_{r}\to 0 would force \mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})=o(\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})) , a contradiction. Hence \liminf_{r\to\infty}m_{r}>0 . Step 3: Fatou and Linear Independence Gives a Contradiction. Divide Equation 30 by m_{r}\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0}) and take covariate expectation. Using Equation 28 , the definition Equation 29 , and Equation 31 , we obtain \mathbb{E}_{{\mathbf{x}}}\!\Bigg[\sum_{m=1}^{M}\frac{|T_{r}(m,\phi_{D}({\mathbf{x}}))|}{m_{r}\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})}\Bigg]\ \longrightarrow\ 0. By Fatou’s lemma, for each m\in[M] we can extract a subsequence (not relabeled) such that \frac{T_{r}(m,\hat{\bm{x}})}{m_{r}\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})}\ \longrightarrow\ 0\qquad\text{for a.e. }\hat{\bm{x}}\in\widehat{{\mathbb{X}}}. Along this subsequence, the normalized coefficients converge (by boundedness) to limits a_{k,\xi}=\lim_{r\to\infty}A_{k,\xi}^{(r)}/(m_{r}\mathcal{D}_{\mathrm{V}}(G^{(r)},G_{0})) , with at least one a_{k,\xi}\neq 0 because \liminf m_{r}>0 . Passing to the limit in Equation 30 yields, for each m\in[M] and a.e. \hat{\bm{x}} , \sum_{k=1}^{K_{0}}\ \sum_{\xi\in\Xi_{k}}a_{k,\xi}\ \Psi_{k,\xi}(m,\hat{\bm{x}})\ =\ 0. (32) Finally, by identifiability of the SGMLMoE and the fact that \hat{\bm{\omega}}_{1}^{0},\ldots,\hat{\bm{\omega}}_{K_{0}}^{0} are distinct, the family of analytic functions \{\Psi_{k,\xi}(m,\cdot)\}_{k,\xi} is linearly independent on \widehat{{\mathbb{X}}} . Thus Equation 32 forces all a_{k,\xi}=0 , contradicting that at least one is nonzero. This contradiction establishes the local bound Equation 27 . Global Structure. By the local result, there exists \varepsilon^{\prime}>0 and c^{\prime}>0 such that \mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})\ \geq\ c^{\prime}\,\mathcal{D}_{\mathrm{V}}(G,G_{0})\qquad\text{whenever }\mathcal{D}_{\mathrm{V}}(G,G_{0})\leq\varepsilon^{\prime}. It remains to prove the complementary bound on the set \{G\in\mathcal{O}_{K}({\mathbb{T}}):\mathcal{D}_{\mathrm{V}}(G,G_{0})\geq\varepsilon^{\prime}\} : \inf_{\begin{subarray}{c}G\in\mathcal{O}_{K}({\mathbb{T}}):\\ \mathcal{D}_{\mathrm{V}}(G,G_{0})\geq\varepsilon^{\prime}\end{subarray}}\frac{\mathcal{D}_{\mathrm{TV}}(s_{G},s_{G_{0}})}{\mathcal{D}_{\mathrm{V}}(G,G_{0})}\ >\ 0. (33) Assume Equation 33 fails. Then there exists G^{\prime(r)}\in\mathcal{O}_{K}({\mathbb{T}}) such that \mathcal{D}_{\mathrm{V}}(G^{\prime(r)},G_{0})\geq\varepsilon^{\prime} and \mathcal{D}_{\mathrm{TV}}(s_{G^{\prime(r)}},s_{G_{0}})\to 0 . Under the standing compactness assumption on {\mathbb{T}} (hence on \mathcal{O}_{K}({\mathbb{T}}) in the weak topology), extract a subsequence with G^{\prime(r)}\Rightarrow G^{\prime} for some G^{\prime}\in\mathcal{O}_{K}({\mathbb{T}}) . By continuity of G\mapsto s_{G}(\cdot\mid{\bm{x}}) and dominated convergence, \mathcal{D}_{\mathrm{TV}}(s_{G^{\prime}},s_{G_{0}})=0 , hence s_{G^{\prime}}(\cdot\mid{\bm{x}})=s_{G_{0}}(\cdot\mid{\bm{x}}) for a.e. {\bm{x}} . By identifiability of SGMLMoE, this implies G^{\prime}\equiv G_{0} , contradicting \mathcal{D}_{\mathrm{V}}(G^{\prime(r)},G_{0})\geq\varepsilon^{\prime} . Therefore Equation 33 holds, and combining local and global parts yields Equation 8 . This completes the proof. D.5 Proof of Theorem 4 We prove the first link of the chain, \mathcal{D}_{\mathrm{V}}\!\big(G^{(K)},G_{0}\big)\gtrsim\mathcal{D}_{\mathrm{V}}\!\big(G^{(K-1)},G_{0}\big), since the remaining inequalities follow by repeating the same argument at each merge step. Throughout, we write a generic K -atom measure as G=\sum_{l=1}^{K}\pi_{l}\,\delta_{\eta_{l}},\qquad\pi_{l}=\exp(\bar{\omega}_{l}),\qquad\eta_{l}=\Big(\hat{\bm{\omega}}_{l},\bar{\omega}_{l},\{(\hat{\bm{\upsilon}}_{m,l},\bar{\upsilon}_{m,l})\}_{m=1}^{M-1}\Big)\in{\mathbb{T}}, and we use the Voronoi partition \{{\mathbb{V}}_{k}\}_{k=1}^{K_{0}} induced by G_{0} as in Section 4.3 . For l\in{\mathbb{V}}_{k} , recall the parameter gaps \Delta\hat{\bm{\omega}}_{lk}=\hat{\bm{\omega}}_{l}-\hat{\bm{\omega}}_{k}^{0},\ \Delta\bar{\upsilon}_{lkm}=\bar{\upsilon}_{m,l}-\bar{\upsilon}_{m,k}^{0},\ \Delta\hat{\bm{\upsilon}}_{lkm}=\hat{\bm{\upsilon}}_{m,l}-\hat{\bm{\upsilon}}_{m,k}^{0} for m\in[M-1] . Step 1: the Closest Pair Lies in a Common Voronoi Cell. Let G^{(K)} be sufficiently close to G_{0} so that the Voronoi cells {\mathbb{V}}_{k} are well-defined and stable. Since the merge rule in Section 4.4 uses squared Euclidean dissimilarities in \hat{\bm{\omega}} and (\bar{\upsilon}_{m},\hat{\bm{\upsilon}}_{m})_{m\in[M-1]}, any pair of atoms belonging to different Voronoi cells stays separated by a fixed positive amount (depending only on G_{0} and the local metric on {\mathbb{T}} ), whereas within-cell pairs can be arbitrarily close when over-fitting occurs. Consequently, for G^{(K)} close enough to G_{0} , the minimizing pair in h^{(K)}=\min_{l\neq l^{\prime}}\mathrm{d}(\pi_{l}\delta_{\eta_{l}},\pi_{l^{\prime}}\delta_{\eta_{l^{\prime}}}) must belong to the same cell {\mathbb{V}}_{k} . Fix such a cell and denote it by {\mathbb{V}}_{k^{\star}} , and assume the merged indices are (l_{1},l_{2})\subset{\mathbb{V}}_{k^{\star}} . Step 2: Notation for the Merge. Let G^{(K-1)} be obtained from G^{(K)} by merging atoms l_{1} and l_{2} according to Section 4.4 . Write the merged atom as (\pi_{\ast},\eta_{\ast}) , where \pi_{\ast}=\pi_{l_{1}}+\pi_{l_{2}},\qquad\hat{\bm{\omega}}_{\ast}=\frac{\pi_{l_{1}}}{\pi_{\ast}}\hat{\bm{\omega}}_{l_{1}}+\frac{\pi_{l_{2}}}{\pi_{\ast}}\hat{\bm{\omega}}_{l_{2}}, and, for each m\in[M-1] , \bar{\upsilon}_{m,\ast}=\frac{\pi_{l_{1}}}{\pi_{\ast}}\bar{\upsilon}_{m,l_{1}}+\frac{\pi_{l_{2}}}{\pi_{\ast}}\bar{\upsilon}_{m,l_{2}},\qquad\hat{\bm{\upsilon}}_{m,\ast}=\frac{\pi_{l_{1}}}{\pi_{\ast}}\hat{\bm{\upsilon}}_{m,l_{1}}+\frac{\pi_{l_{2}}}{\pi_{\ast}}\hat{\bm{\upsilon}}_{m,l_{2}}. All other atoms are unchanged. Since l_{1},l_{2}\in{\mathbb{V}}_{k^{\star}} and G^{(K)} is close to G_{0} , the merged atom also remains assigned to the same cell {\mathbb{V}}_{k^{\star}} . Step 3: the Voronoi Weight-mismatch Term is Preserved. For each k\in[K_{0}] , the weight-mismatch contribution \big|\sum_{l\in{\mathbb{V}}_{k}}\pi_{l}-\pi_{k}^{0}\big| is unchanged by merging within {\mathbb{V}}_{k} : indeed, for k\neq k^{\star} nothing changes, and for k=k^{\star} , \sum_{l\in{\mathbb{V}}_{k^{\star}}}\pi_{l}=\sum_{l\in{\mathbb{V}}_{k^{\star}}\setminus\{l_{1},l_{2}\}}\pi_{l}+\pi_{l_{1}}+\pi_{l_{2}}=\sum_{l\in{\mathbb{V}}_{k^{\star}}\setminus\{l_{1},l_{2}\}}\pi_{l}+\pi_{\ast}, so the absolute deviation from \pi_{k^{\star}}^{0} is identical before and after merging. Step 4: Within-cell Quadratic Terms Decrease under Barycentric Merging. Consider first the gate-slope quadratic term on the over-covered cell |{\mathbb{V}}_{k^{\star}}|>1 : \sum_{l\in{\mathbb{V}}_{k^{\star}}}\pi_{l}\|\Delta\hat{\bm{\omega}}_{lk^{\star}}\|^{2}=\sum_{l\in{\mathbb{V}}_{k^{\star}}\setminus\{l_{1},l_{2}\}}\pi_{l}\|\Delta\hat{\bm{\omega}}_{lk^{\star}}\|^{2}+\pi_{l_{1}}\|\Delta\hat{\bm{\omega}}_{l_{1}k^{\star}}\|^{2}+\pi_{l_{2}}\|\Delta\hat{\bm{\omega}}_{l_{2}k^{\star}}\|^{2}. By convexity of u\mapsto\|u\|^{2} and the definition \Delta\hat{\bm{\omega}}_{\ast k^{\star}}=\frac{\pi_{l_{1}}}{\pi_{\ast}}\Delta\hat{\bm{\omega}}_{l_{1}k^{\star}}+\frac{\pi_{l_{2}}}{\pi_{\ast}}\Delta\hat{\bm{\omega}}_{l_{2}k^{\star}}, we have the Jensen inequality \pi_{l_{1}}\|\Delta\hat{\bm{\omega}}_{l_{1}k^{\star}}\|^{2}+\pi_{l_{2}}\|\Delta\hat{\bm{\omega}}_{l_{2}k^{\star}}\|^{2}\geq\pi_{\ast}\|\Delta\hat{\bm{\omega}}_{\ast k^{\star}}\|^{2}. Hence, \sum_{l\in{\mathbb{V}}_{k^{\star}}}\pi_{l}\|\Delta\hat{\bm{\omega}}_{lk^{\star}}\|^{2}\geq\sum_{l\in{\mathbb{V}}_{k^{\star}}\setminus\{l_{1},l_{2}\}}\pi_{l}\|\Delta\hat{\bm{\omega}}_{lk^{\star}}\|^{2}+\pi_{\ast}\|\Delta\hat{\bm{\omega}}_{\ast k^{\star}}\|^{2}, which is exactly the corresponding contribution after merging l_{1},l_{2} into \ast . The same argument applies to each expert block, for every m\in[M-1] : since u\mapsto|u|^{2} and u\mapsto\|u\|^{2} are convex, \pi_{l_{1}}|\Delta\bar{\upsilon}_{l_{1}k^{\star}m}|^{2}+\pi_{l_{2}}|\Delta\bar{\upsilon}_{l_{2}k^{\star}m}|^{2}\geq\pi_{\ast}|\Delta\bar{\upsilon}_{\ast k^{\star}m}|^{2}, \pi_{l_{1}}\|\Delta\hat{\bm{\upsilon}}_{l_{1}k^{\star}m}\|^{2}+\pi_{l_{2}}\|\Delta\hat{\bm{\upsilon}}_{l_{2}k^{\star}m}\|^{2}\geq\pi_{\ast}\|\Delta\hat{\bm{\upsilon}}_{\ast k^{\star}m}\|^{2}, and therefore the entire within-cell quadratic contribution in Equation 7 for cell {\mathbb{V}}_{k^{\star}} does not increase after merging. Step 5: Conclusion. All other cells are unaffected, and the linear (single-covered) terms in \mathcal{D}_{\mathrm{E}}(G,G_{0}) are unchanged because they depend only on weights and on singletons. Hence, \mathcal{D}_{\mathrm{V}}\!\big(G^{(K)},G_{0}\big)\ \gtrsim\ \mathcal{D}_{\mathrm{V}}\!\big(G^{(K-1)},G_{0}\big), with an absolute constant (in particular, one may take the implicit constant to be 1 in the within-cell comparison). Iterating the same argument along the successive merges yields the full chain claimed in Theorem 4 . ∎ D.6 Proof of Theorem 5 Throughout, implicit constants in \lesssim,\gtrsim are universal and may change line to line. Wasserstein Distance for Finite Mixing Measures. For two mixing measures G=\sum_{l=1}^{K}\pi_{l}\,\delta_{\eta_{l}},\qquad G^{\prime}=\sum_{l^{\prime}=1}^{K^{\prime}}\pi^{\prime}_{l^{\prime}}\,\delta_{\eta^{\prime}_{l^{\prime}}}, and any r\geq 1 , define the Wasserstein- r distance W_{r}(G,G^{\prime})=\Bigg(\inf_{Q\in\Pi(\pi,\pi^{\prime})}\sum_{l=1}^{K}\sum_{l^{\prime}=1}^{K^{\prime}}Q_{ll^{\prime}}\,\|\eta_{l}-\eta^{\prime}_{l^{\prime}}\|^{r}\Bigg)^{1/r}, where \Pi(\pi,\pi^{\prime}) is the set of couplings between \pi=(\pi_{1},\ldots,\pi_{K}) and \pi^{\prime}=(\pi^{\prime}_{1},\ldots,\pi^{\prime}_{K^{\prime}}) , i.e., \Pi(\pi,\pi^{\prime})=\Big\{Q\in{\mathbb{R}}_{+}^{K\times K^{\prime}}:\ \sum_{l^{\prime}=1}^{K^{\prime}}Q_{ll^{\prime}}=\pi_{l},\ \sum_{l=1}^{K}Q_{ll^{\prime}}=\pi^{\prime}_{l^{\prime}},\ \forall l\in[K],\,l^{\prime}\in[K^{\prime}]\Big\}. A Useful Lower Bound (Wasserstein Controls Componentwise Discrepancy). Fix the true mixing measure G_{0}=\sum_{k=1}^{K_{0}}\pi_{k}^{0}\,\delta_{\eta_{k}^{0}}\in\mathcal{O}_{K_{0}}({\mathbb{T}}) and consider G=\sum_{l=1}^{K}\pi_{l}\,\delta_{\eta_{l}}\in\mathcal{O}_{K}({\mathbb{T}}) such that W_{r}(G,G_{0})\to 0 . Then (see, e.g., ( 39 ) ) there exists a Voronoi assignment l\mapsto k(l)\in[K_{0}] and cells {\mathbb{V}}_{k}=\{l\in[K]:k(l)=k\} such that W_{r}^{r}(G,G_{0})\ \gtrsim\ \sum_{k=1}^{K_{0}}\Bigg(\Big|\sum_{l\in{\mathbb{V}}_{k}}\pi_{l}-\pi_{k}^{0}\Big|+\sum_{l\in{\mathbb{V}}_{k}}\pi_{l}\,\|\eta_{l}-\eta_{k}^{0}\|^{r}\Bigg). (34) Part 1: Voronoi and Height Rates on Over-fitted Levels \kappa\in\{K_{0}+1,\ldots,K\} . Let A_{N}:=\Big\{\mathcal{D}_{\mathrm{TV}}(s_{\widehat{G}_{N}},s_{G_{0}})\leq C\sqrt{\log N/N}\Big\}, where C>0 is the constant from 1 . By 1 , \mathbb{P}(A_{N})\geq 1-c_{1}N^{-c_{2}}. On A_{N} , combining Theorem 3 with the density bound yields \mathcal{D}_{\mathrm{V}}(\widehat{G}_{N},G_{0})\ \lesssim\ \sqrt{\log N/N}. (35) Next, by Theorem 4 , the Voronoi loss is monotone (up to constants) along the merge chain, hence for each \kappa\in\{K_{0}+1,\ldots,K\} , \mathcal{D}_{\mathrm{V}}(\widehat{G}_{N}^{(\kappa)},G_{0})\ \lesssim\ \sqrt{\log N/N}. (36) We now bound the dendrogram height h_{N}^{(\kappa)} for \kappa\geq K_{0}+1 . Since \kappa>K_{0} , at level \kappa there exists at least one Voronoi cell containing two distinct atoms of \widehat{G}_{N}^{(\kappa)} . Fix such a cell {\mathbb{V}}_{k^{\star}} and pick two distinct indices l_{1}\neq l_{2}\in{\mathbb{V}}_{k^{\star}} . Write \widehat{G}_{N}^{(\kappa)}=\sum_{l=1}^{\kappa}\pi_{l}^{(\kappa)}\delta_{\eta_{l}^{(\kappa)}} and abbreviate \pi_{l}:=\pi_{l}^{(\kappa)},\ \eta_{l}:=\eta_{l}^{(\kappa)}. From the definition of \mathcal{D}_{\mathrm{V}} in Equation 7 and Equation 36 , we obtain \displaystyle\pi_{l_{1}}\Big(\|\hat{\bm{\omega}}_{l_{1}}-\hat{\bm{\omega}}_{k^{\star}}^{0}\|^{2}+\sum_{m=1}^{M-1}\big(|\bar{\upsilon}_{m,l_{1}}-\bar{\upsilon}_{m,k^{\star}}^{0}|^{2}+\|\hat{\bm{\upsilon}}_{m,l_{1}}-\hat{\bm{\upsilon}}_{m,k^{\star}}^{0}\|^{2}\big)\Big) \displaystyle\quad+\pi_{l_{2}}\Big(\|\hat{\bm{\omega}}_{l_{2}}-\hat{\bm{\omega}}_{k^{\star}}^{0}\|^{2}+\sum_{m=1}^{M-1}\big(|\bar{\upsilon}_{m,l_{2}}-\bar{\upsilon}_{m,k^{\star}}^{0}|^{2}+\|\hat{\bm{\upsilon}}_{m,l_{2}}-\hat{\bm{\upsilon}}_{m,k^{\star}}^{0}\|^{2}\big)\Big)\ \lesssim\ \sqrt{\log N/N}. (37) Using the elementary inequality \min\{\pi_{l_{1}},\pi_{l_{2}}\}\ \geq\ \frac{\pi_{l_{1}}\pi_{l_{2}}}{\pi_{l_{1}}+\pi_{l_{2}}}, and the triangle inequality \|\hat{\bm{\omega}}_{l_{1}}-\hat{\bm{\omega}}_{l_{2}}\|\leq\|\hat{\bm{\omega}}_{l_{1}}-\hat{\bm{\omega}}_{k^{\star}}^{0}\|+\|\hat{\bm{\omega}}_{l_{2}}-\hat{\bm{\omega}}_{k^{\star}}^{0}\| (and similarly for expert parameters), we can lower bound the left-hand side of Section D.6 by a constant multiple of the dissimilarity \mathrm{d}(\pi_{l_{1}}\delta_{\eta_{l_{1}}},\pi_{l_{2}}\delta_{\eta_{l_{2}}}) defined in Section 4.4 . Concretely, there exists a universal c>0 such that \mathrm{d}\!\big(\pi_{l_{1}}\delta_{\eta_{l_{1}}},\pi_{l_{2}}\delta_{\eta_{l_{2}}}\big)\ \leq\ c\cdot\text{LHS of \lx@cref{creftype\lx@tilde refnum}{eq_two_atom_quadratic_small}}\ \lesssim\ \sqrt{\log N/N}. Since h_{N}^{(\kappa)} is the minimum dissimilarity over all pairs of atoms at level \kappa , h_{N}^{(\kappa)}\ \lesssim\ \sqrt{\log N/N},\qquad\forall\kappa\in\{K_{0}+1,\ldots,K\}. (38) Part 2: Under-fitted Levels \kappa^{\prime}\in[K_{0}] . At the exact level K_{0} , each Voronoi cell is a singleton, and by inspection of Equation 7 , \mathcal{D}_{\mathrm{V}}(\widehat{G}_{N}^{(K_{0})},G_{0}) is equivalent (up to constants) to a W_{1} -type discrepancy between the two K_{0} -atom measures. In particular, W_{1}(\widehat{G}_{N}^{(K_{0})},G_{0})\ \lesssim\ \sqrt{\log N/N}. (39) Write \widehat{G}_{N}^{(K_{0})}=\sum_{k=1}^{K_{0}}\pi_{k,N}\,\delta_{\eta_{k,N}},\qquad G_{0}=\sum_{k=1}^{K_{0}}\pi_{k}^{0}\,\delta_{\eta_{k}^{0}}, under the natural matching implied by the Voronoi construction (valid for N large enough on A_{N} ). Then Equation 39 implies componentwise control, e.g., |\pi_{k,N}-\pi_{k}^{0}|\lesssim\sqrt{\log N/N},\qquad\|\eta_{k,N}-\eta_{k}^{0}\|\lesssim\sqrt{\log N/N},\qquad k\in[K_{0}], and hence, for any fixed pair (k_{1},k_{2}) , the pairwise dissimilarity at level K_{0} satisfies the perturbation bound \Big|\mathrm{d}\!\big(\pi_{k_{1},N}\delta_{\eta_{k_{1},N}},\pi_{k_{2},N}\delta_{\eta_{k_{2},N}}\big)-\mathrm{d}\!\big(\pi_{k_{1}}^{0}\delta_{\eta_{k_{1}}^{0}},\pi_{k_{2}}^{0}\delta_{\eta_{k_{2}}^{0}}\big)\Big|\ \lesssim\ \sqrt{\log N/N}, (40) where we used that all relevant parameters are uniformly bounded (and \pi_{k}^{0} are bounded away from 0 ). Now let (k_{1}^{\star},k_{2}^{\star}) be the (unique, for simplicity) minimizing pair for the true height h_{0}^{(K_{0})}=\min_{k_{1}\neq k_{2}}\mathrm{d}(\pi_{k_{1}}^{0}\delta_{\eta_{k_{1}}^{0}},\pi_{k_{2}}^{0}\delta_{\eta_{k_{2}}^{0}}). By Equation 40 , for N large enough the minimizing pair for the empirical dissimilarity at level K_{0} coincides with (k_{1}^{\star},k_{2}^{\star}) , and therefore |h_{N}^{(K_{0})}-h_{0}^{(K_{0})}|\ \lesssim\ \sqrt{\log N/N}. After merging this pair once, the merged atom parameters are barycenters as in Section 4.4 , so the same Lipschitz-type perturbation argument (using Equation 39 and boundedness of parameters/weights away from 0 ) yields W_{1}\!\big(\widehat{G}_{N}^{(K_{0}-1)},G_{0}^{(K_{0}-1)}\big)\ \lesssim\ \sqrt{\log N/N},\qquad|h_{N}^{(K_{0}-1)}-h_{0}^{(K_{0}-1)}|\ \lesssim\ \sqrt{\log N/N}. Iterating this argument inductively down the chain proves that for every \kappa^{\prime}\in[K_{0}] , |h_{N}^{(\kappa^{\prime})}-h_{0}^{(\kappa^{\prime})}|\ \lesssim\ \sqrt{\log N/N}. (41) Conclusion. Combining Equation 36 , Equation 38 , and Equation 41 , and recalling \mathbb{P}(A_{N})\geq 1-c_{1}N^{-c_{2}} , completes the proof of Theorem 5 . ∎ D.7 Proof of Theorem 6 Suppose ({\mathbf{x}}_{1},{\textnormal{y}}_{1}),\ldots,({\mathbf{x}}_{N},{\textnormal{y}}_{N}) are i.i.d. from G_{0} and write the empirical measure P_{N}:=\frac{1}{N}\sum_{n=1}^{N}\delta_{({\bm{x}}_{n},{y}_{n})} (with ({\bm{x}}_{n},{y}_{n}) the observed sample). For any mixing measure G\in\mathcal{O}_{K}({\mathbb{T}}) , define the empirical process \nu_{N}(G):=\sqrt{N}\,(P_{N}-P_{G_{0}})\log\frac{s_{G}}{s_{G_{0}}}. We will repeatedly use the following exponential inequality. Fact 2 (Theorem 5.11 from ( 96 ) ) . Let R,C,C_{1},a>0 satisfy a\leq C_{1}\sqrt{N}\,R^{2}\ \wedge\ 8\sqrt{N}\,R, and a\geq\sqrt{C^{2}(C_{1}+1)}\left(\int_{a/(2^{6}\sqrt{N})}^{R}H_{B}^{1/2}\!\left(\frac{u}{\sqrt{2}},\ \{\,s_{G}:\ h(s_{G},s_{G_{0}})\leq R\,\},\ \nu\right)\,du\ \vee\ R\right). Then \mathbb{P}_{G_{0}}\!\left(\sup_{h(s_{G},s_{G_{0}})\leq R}\big|\nu_{N}(G)\big|\geq a\right)\leq C\exp\!\left(-\frac{a^{2}}{C^{2}(C_{1}+1)R^{2}}\right). Case 1: \kappa\geq K_{0} . For \kappa\in\{K_{0},\ldots,K\} , let \widehat{G}_{N}^{(\kappa)} denote the \kappa -atom mixing measure along the dendrogram path and write \bar{\ell}_{N}^{(\kappa)}=\bar{\ell}_{N}(\widehat{G}_{N}^{(\kappa)})=\frac{1}{N}\sum_{n=1}^{N}\log s_{\widehat{G}_{N}^{(\kappa)}}({y}_{n}\mid{\bm{x}}_{n}) . Step 1 (a concavity bound). Using concavity of \log , for any two densities p,q , \frac{1}{2}\log\frac{p}{q}\leq\log\frac{p+q}{2q}. Applying this with p=s_{\widehat{G}_{N}^{(\kappa)}}({y}\mid{\bm{x}}) and q=s_{G_{0}}({y}\mid{\bm{x}}) gives \frac{1}{2}\,P_{N}\log\frac{s_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}\leq P_{N}\log\frac{\bar{s}_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}},\qquad\bar{s}_{\widehat{G}_{N}^{(\kappa)}}:=\frac{s_{\widehat{G}_{N}^{(\kappa)}}+s_{G_{0}}}{2}. Moreover, P_{N}\log\frac{\bar{s}_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}=(P_{N}-P_{G_{0}})\log\frac{\bar{s}_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}-D_{\mathrm{KL}}\!\left(s_{G_{0}}\,\|\,\bar{s}_{\widehat{G}_{N}^{(\kappa)}}\right)\leq(P_{N}-P_{G_{0}})\log\frac{\bar{s}_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}. Hence, P_{N}\log s_{\widehat{G}_{N}^{(\kappa)}}-\mathcal{L}(G_{0})=P_{N}\log\frac{s_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}+(P_{N}-P_{G_{0}})\log s_{G_{0}}\leq 2(P_{N}-P_{G_{0}})\log\frac{s_{\widehat{G}_{N}^{(\kappa)}}}{s_{G_{0}}}+(P_{N}-P_{G_{0}})\log s_{G_{0}}. (42) Step 2 (Hellinger radius at over-fitted levels). By Theorem 5 , with probability at least 1-c_{1}N^{-c_{2}} , \mathcal{D}_{\mathrm{V}}\!\big(\widehat{G}_{N}^{(\kappa)},G_{0}\big)\ \lesssim\ \sqrt{\log N/N},\qquad\kappa\in\{K_{0},\ldots,K\}. Since W_{2}^{2}(\cdot,\cdot)\lesssim\mathcal{D}_{\mathrm{V}}(\cdot,\cdot) (for measures on bounded parameter sets with the Voronoi construction), we obtain on the same event W_{2}\!\big(\widehat{G}_{N}^{(\kappa)},G_{0}\big)\ \lesssim\ (\log N/N)^{1/4}. Using the standard inequality h(s_{G},s_{G_{0}})\lesssim W_{2}(G,G_{0}) for these finite-mixture/MoE families on bounded domains (see, e.g., ( 82 ) ), we deduce that there exists R_{N}\asymp(\log N/N)^{1/4} and an event A_{N} with \mathbb{P}_{G_{0}}(A_{N})\geq 1-c_{1}N^{-c_{2}} such that h\!\left(s_{\widehat{G}_{N}^{(\kappa)}},s_{G_{0}}\right)\leq R_{N},\qquad\forall\kappa\in\{K_{0},\ldots,K\}. (43) Step 3 (empirical-process control). Set R:=R_{N} in 2 and choose a:=\sqrt{N}\,R_{N}\,\log(1/R_{N})\ \asymp\ N^{1/4}\log^{1/4}N\cdot\log N (which is of the same order as \sqrt{N}R_{N}\log(1/R_{N}) ). Using the entropy bound H_{B}^{1/2}(u,\{s_{G}:\ h(s_{G},s_{G_{0}})\leq R\},\nu)\lesssim\log(1/u) (e.g. Lemma 4 from ( 72 ) ), the conditions of 2 hold for N large enough, yielding \mathbb{P}_{G_{0}}\!\left(\sup_{h(s_{G},s_{G_{0}})\leq R_{N}}\big|\nu_{N}(G)\big|\ \gtrsim\ a\right)\lesssim N^{-c} for some c>0 . Equivalently, dividing by \sqrt{N} , \mathbb{P}_{G_{0}}\!\left(\sup_{h(s_{G},s_{G_{0}})\leq R_{N}}\big|(P_{N}-P_{G_{0}})\log(s_{G}/s_{G_{0}})\big|\ \gtrsim\ R_{N}\log(1/R_{N})\right)\lesssim N^{-c}. (44) Combining Equation 43 with Equation 44 yields that, for each fixed \kappa\in\{K_{0},\ldots,K\} , \big|(P_{N}-P_{G_{0}})\log(s_{\widehat{G}_{N}^{(\kappa)}}/s_{G_{0}})\big|\ \lesssim\ R_{N}\log(1/R_{N})\ \lesssim\ (\log N/N)^{1/4} (45) with probability at least 1-c_{1}^{\prime}N^{-c_{2}^{\prime}} . Step 4 (control of (P_{N}-P_{G_{0}})\log s_{G_{0}} ). By Chebyshev’s inequality, \mathbb{P}_{G_{0}}\!\left(\big|(P_{N}-P_{G_{0}})\log s_{G_{0}}\big|\geq t\right)\leq\frac{\mathrm{Var}(\log s_{G_{0}})}{Nt^{2}}. (46) Taking t\asymp(\log N/N)^{1/4} gives \big|(P_{N}-P_{G_{0}})\log s_{G_{0}}\big|\ \lesssim\ (\log N/N)^{1/4} (47) with probability at least 1-\mathrm{Var}(\log s_{G_{0}})/\sqrt{\log N} . Step 5 (conclusion for \kappa\geq K_{0} ). Plugging Equation 45 and Equation 47 into Equation 42 yields, with probability tending to 1 , \bar{\ell}_{N}^{(\kappa)}-\mathcal{L}(G_{0})=P_{N}\log s_{\widehat{G}_{N}^{(\kappa)}}-\mathcal{L}(G_{0})\ \lesssim\ (\log N/N)^{1/4},\qquad\kappa\in\{K_{0},\ldots,K\}. This proves the first claim of Theorem 6 . Case 2: \kappa=K_{0} (a Matching Lower Bound using Condition K). At the exact-fitted level, Theorem 5 implies W_{1}(\widehat{G}_{N}^{(K_{0})},G_{0})\ \lesssim\ \sqrt{\log N/N}. Let \epsilon_{N}:=\sqrt{\log N/N} . By the definition of \mathcal{D}_{\mathrm{V}} at singleton Voronoi cells (exact-fit), this implies a componentwise bound \|\eta_{k,N}-\eta_{k}^{0}\|\lesssim\epsilon_{N} and |\pi_{k,N}-\pi_{k}^{0}|\lesssim\epsilon_{N} (under the canonical matching), hence Condition K yields the pointwise inequality \log s_{\widehat{G}_{N}^{(K_{0})}}({\textnormal{y}}\mid{\mathbf{x}})\geq(1+c_{\beta}\epsilon_{N})\log s_{G_{0}}({\textnormal{y}}\mid{\mathbf{x}})-c_{\alpha}\epsilon_{N},\qquad({\mathbf{x}},{\textnormal{y}})\in{\mathbb{X}}\times{\mathbb{Y}}, for some constants c_{\alpha},c_{\beta}>0 . Averaging over P_{N} and subtracting \mathcal{L}(G_{0}) gives \bar{\ell}_{N}^{(K_{0})}-\mathcal{L}(G_{0})\geq-c_{\alpha}\epsilon_{N}+c_{\beta}\epsilon_{N}\,P_{N}\log s_{G_{0}}+(1+c_{\beta}\epsilon_{N})(P_{N}-P_{G_{0}})\log s_{G_{0}}. Using Equation 46 with t=\epsilon_{N} implies (P_{N}-P_{G_{0}})\log s_{G_{0}}=O_{\mathbb{P}}(\epsilon_{N}), and since P_{N}\log s_{G_{0}}=P_{G_{0}}\log s_{G_{0}}+O_{\mathbb{P}}(\epsilon_{N}) , we conclude that \bar{\ell}_{N}^{(K_{0})}-\mathcal{L}(G_{0})\ \geq\ -C\epsilon_{N} in \mathbb{P}_{G_{0}} -probability. Together with the upper bound from Case 1 at \kappa=K_{0} , this yields \big|\bar{\ell}_{N}^{(K_{0})}-\mathcal{L}(G_{0})\big|\ \lesssim\ (\log N/N)^{1/4} (up to logs), which is consistent with the statement in Theorem 6 . Case 3: \kappa0 small. By norm equivalence on finite-dimensional spaces, it suffices to assume that for all m\in[M-1] , \|\hat{\bm{\upsilon}}_{m}-\hat{\bm{\upsilon}}_{m}^{0}\|\leq\epsilon,\qquad|\bar{\upsilon}_{m}-\bar{\upsilon}_{m}^{0}|\leq\epsilon, (and the reference class m=M is fixed at (\hat{\bm{\upsilon}}_{M},\bar{\upsilon}_{M})=(\mathbf{0},0) ). We show that there exist constants c_{\alpha},c_{\beta}>0 such that for all ({\mathbf{x}},{\textnormal{y}})\in{\mathbb{X}}\times{\mathbb{Y}} , \log s_{{\bm{\theta}}}({\textnormal{y}}\mid{\mathbf{x}})\geq(1+c_{\beta}\epsilon)\log s_{{\bm{\theta}}^{0}}({\textnormal{y}}\mid{\mathbf{x}})-c_{\alpha}\epsilon. Since {\mathbb{X}} is compact and the parameter space is bounded, the score map (\hat{\bm{\upsilon}},\bar{\upsilon},{\bm{x}})\mapsto\log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};\hat{\bm{\upsilon}},\bar{\upsilon}) is locally Lipschitz uniformly over {\bm{x}}\in{\mathbb{X}} . Thus there exists L>0 such that, for all m\in[M-1] and all {\bm{x}}\in{\mathbb{X}} , \big|(\hat{\bm{\upsilon}}_{m}-\hat{\bm{\upsilon}}_{m}^{0})^{\top}{\bm{x}}\big|\leq L\epsilon,\qquad|\bar{\upsilon}_{m}-\bar{\upsilon}_{m}^{0}|\leq\epsilon. Moreover, the map (\{\hat{\bm{\upsilon}}_{\ell},\bar{\upsilon}_{\ell}\}_{\ell=1}^{M-1},{\bm{x}})\mapsto\log\!\big(1+\sum_{\ell=1}^{M-1}\exp(\bar{\upsilon}_{\ell}+\hat{\bm{\upsilon}}_{\ell}^{\top}{\bm{x}})\big) is smooth, hence locally Lipschitz, and therefore its perturbation is O(\epsilon) uniformly over {\bm{x}}\in{\mathbb{X}} . Combining these uniform Lipschitz bounds yields \log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};{\bm{\theta}})\geq\log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};{\bm{\theta}}^{0})-C\epsilon for some C>0 and all ({\bm{x}},{\textnormal{y}}) . Finally, since \log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};{\bm{\theta}}^{0})\leq 0 and is bounded below on {\mathbb{X}}\times{\mathbb{Y}} , we may choose c_{\beta}>0 small enough and c_{\alpha}>0 large enough so that \log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};{\bm{\theta}})\geq(1+c_{\beta}\epsilon)\log\mathsf{e}({\textnormal{y}}\mid{\bm{x}};{\bm{\theta}}^{0})-c_{\alpha}\epsilon. Applying the same reasoning to the gating part and summing over experts inside s_{\bm{\theta}} gives Condition K. ∎ D.9 Proof of Theorem 7 Recall \bar{\ell}_{N}^{(\kappa)}:=\bar{\ell}_{N}(\widehat{G}_{N}^{(\kappa)}) and \mathrm{DSC}_{N}^{(\kappa)}:=-\big(h_{N}^{(\kappa)}+\omega_{N}\bar{\ell}_{N}^{(\kappa)}\big) . From Theorem 5 , h_{N}^{(\kappa)}=\begin{cases}O_{\mathbb{P}}\!\big((\log N/N)^{1/2}\big),&\kappa>K_{0},\\[2.58334pt] h_{0}^{(\kappa)}+O_{\mathbb{P}}\!\big((\log N/N)^{1/2}\big),&\kappa\leq K_{0}.\end{cases} From Theorem 6 , \bar{\ell}_{N}^{(\kappa)}=\begin{cases}\mathcal{L}(G_{0})+O_{\mathbb{P}}\!\big((\log N/N)^{1/4}\big),&\kappa\geq K_{0},\\[2.58334pt] \mathcal{L}(G_{0}^{(\kappa)})+o_{\mathbb{P}}(1),&\kappa0 ). Therefore, \mathrm{DSC}_{N}^{(\kappa)}=\begin{cases}-\omega_{N}\mathcal{L}(G_{0})+O_{\mathbb{P}}\!\big(\omega_{N}(\log N/N)^{1/4}\big),&\kappa>K_{0},\\[3.44444pt] -\omega_{N}\mathcal{L}(G_{0})-h_{0}^{(K_{0})}+O_{\mathbb{P}}\!\big(\omega_{N}(\log N/N)^{1/4}\big),&\kappa=K_{0},\\[3.44444pt] -\omega_{N}\mathcal{L}(G_{0}^{(\kappa)})-h_{0}^{(\kappa)}+o_{\mathbb{P}}(\omega_{N}),&\kappa0 for \kappa