On Linear Mode Connectivity of Mixture-of-Experts Architectures Viet-Hoang Tran Van Hoan Trinh Khanh Vinh Bui Tan M. Nguyen Abstract Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes of neural networks, wherein independently trained models have been observed to be connected–up to permutation symmetries–by linear paths in parameter space along which the loss remains consistently low. This observation challenges classical views of non-convex optimization and has implications for model ensembling, generalization, and our understanding of neural loss geometry. Inspired by recent studies on LMC in standard neural networks, we systematically investigate this phenomenon within Mixture-of-Experts (MoE) architectures–a class of models known for their scalability and computational efficiency, which combine traditional neural networks–referred to as experts–through a learnable gating mechanism. We begin by conducting a comprehensive analysis of both dense and sparse gating regimes, demonstrating that the symmetries inherent to MoE architectures are fully characterized by permutations acting on both the expert components and the gating function. Building on these foundational findings, we propose a matching algorithm that enables alignment between independently trained MoEs, thereby facilitating the discovery of LMC. Finally, we empirically validate the presence of LMC using our proposed algorithm across diverse MoE configurations–including dense, sparse, and shared-expert variants–under a wide range of model settings and datasets of varying scales and modalities. Our results confirm the existence of LMC in MoE architectures and offer fundamental insights into the functional landscape and optimization dynamics of deep learning models. ${}^{*}$ ${}^{*}$ footnotetext: Co-first authors. Correspondence to: hoang.tranviet@u.nus.edu and tanmn@nus.edu.sg 1 Introduction Despite the high-dimensional and non-convex nature of deep neural network (DNN) training, stochastic optimization methods–such as stochastic gradient descent (SGD) and its variants–consistently find solutions that generalize well. This empirical success contrasts with the theoretical complexity of the loss landscape, which contains many local minima. A growing body of work has uncovered a surprising phenomenon known as mode connectivity , wherein minima found by independent SGD runs can often be connected by continuous low-loss paths. Mode Connectivity. Related work on mode connectivity includes ( goodfellow2014qualitatively ; keskar2016large ; sagun2017empirical ; venturi2019spurious ; neyshabur2020being ; tatro2020optimizing ; yunis2022convexity ; zhou2023going ) . Empirical studies demonstrate low-loss connections between independently trained models on MNIST and CIFAR-10 ( freeman2016topology ; garipov2018loss ; pmlr-v80-draxler18a ) , and have shown that nearly any two solutions can be linked by a low-error curve ( garipov_mode_connectivity_2018 ) . The implications of mode connectivity extend beyond theoretical curiosity. It provides insight into why weight-space ensembling techniques, such as Stochastic Weight Averaging, yield improved generalization ( izmailov2018averaging ; rame2022diverse ; wortsman2022model ) . Moreover, it has proven useful in studying adversarial robustness ( zhao2020bridging ) , generalization theory ( pittorino2022deep ; juneja2022linear ; lubana2023mechanistic ) , and the geometry of loss landscapes ( gotmare2018using ; vlaar2022can ; lucas2021analyzing ) . A more restrictive and analytically convenient variant of this phenomenon is linear mode connectivity (LMC), where two models are connected by a linear interpolation in parameter space along which the loss remains low ( frankle2020linear ; entezari2021role ) . Permutation Invariance. A central challenge in observing LMC stems from the permutation invariance of neural networks: permuting neurons within a hidden layer leaves the network function unchanged ( allen2019convergence ; du2019gradient ; frankle2018lottery ; belkin2019reconciling ; neyshabur2018towards ) . Consequently, two functionally equivalent models may appear disconnected in parameter space unless one is suitably permuted. To address this, recent work considers LMC up to permutation , wherein a low-loss linear path exists after aligning hidden units ( singh2020model ; ainsworth2022git ; sinkhorn_re_basin_2023 ) . Optimal Transport (OT) methods have been proposed to compute soft matchings between neurons ( singh2020model ; singh_jaggi_otfusion_2021 ) , enabling applications such as model fusion in federated learning. Empirically, OT-based alignment achieves near-zero error barrier LMC between independently trained ResNets on CIFAR-10 ( ainsworth2022git ) , with connectivity improving with width and degrading with depth. Theoretically, dropout-stable networks have been shown to exhibit mode connectivity ( kuditipudi2019explaining ; shevchenko2020landscape ) , and ( entezari2021role ) demonstrate that LMC up to permutation can already arise at initialization, especially in the NTK regime ( jacot2018neural ) . ( ferbach2024proving ) further provide formal guarantees for LMC under OT alignment. These insights support the convexity conjecture ( entezari_permutation_invariances_2022 ) , which posits that the SGD solution set is approximately convex once symmetries are accounted for. This view is strengthened by ( sharma2024simultaneous ) , who propose simultaneous linear connectivity , where a single model aligns linearly with multiple others. Additional studies explore the geometry of the solution space ( ainsworth_git_re_basin_2022 ; xiao2023compact ) and identify star-shaped regions conducive to LMC ( sonthalia2024deep ) . Functional Equivalence. Prior work on LMC has primarily focused on feedforward and convolutional architectures, owing to the well-characterized permutation symmetries in these models ( brea2019weight ; novak2018sensitivity ; bui2020functional ; tran2024monomial ; vo2024equivariant ) . In contrast, the symmetry structures of more modern architectures–such as Transformers ( vaswani2017attention ; devlin2018bert ; brown2020language ) and Mixture-of-Experts ( jacobs1991adaptive ; shazeer2017outrageously ; lepikhin2020gshard ; fedus2022switch ) –remain relatively underexplored. The study of these symmetries falls under the broader concept of functional equivalence , which aims to characterize when different parameter configurations yield identical input–output behavior ( hecht1990algebraic ; chen1993geometry ; fefferman1993recovering ; kuurkova1994functionally ; albertini1993neural ; albertini1993identifiability ) . Recent work has extended this analysis to attention-based models, with ( tran2024equivariant ; knyazev2024accelerating ) examining symmetries in attention mechanisms, particularly permutations of heads and group actions induced by the general linear group. A rigorous understanding of functional equivalence is essential for uncovering and formalizing LMC in such complex, structured architectures. Permutation Alignment Methods. These methods align parameter permutations to establish LMC. ( entezari_permutation_invariances_2022 ) proposed a simulated annealing-based algorithm. ( singh_jaggi_otfusion_2021 ) employed Optimal Transport, while ( akash2022wasserstein ) utilized the Wasserstein Barycenter. ( ainsworth2022git ) introduced three methods: activation matching (using intermediate activations), weight matching (in parameter space), and the Straight-Through Estimator (which minimizes interpolation midpoint loss via gradients); all are based on solving the Linear Assignment Problem ( kuhn1955hungarian ; jonker1988shortest ; crouse2016implementing ) . ( sinkhorn_re_basin_2023 ) developed Sinkhorn re-basin, a differentiable method that improves alignment but struggles with residual connections due to layer-independent optimization. Contribution. Motivated by this line of work, we extend the investigation of LMC to Mixture-of-Experts (MoE) architectures, which are related to traditional neural networks in that they aggregate outputs of multiple subnetworks via a gating mechanism. The paper is organized as follows: 1. In Section 3 , we introduce the concept of the weight space of MoE architectures and define a group action on this space that preserves the functional behavior of MoE models. 2. In Section 4 , we present two core results concerning functional equivalence in MoE models. We demonstrate that the proposed group action characterizes all inherent symmetries of the MoE gating mechanism, with rigorous theoretical justification. 3. In Section 5 , we first observe that permutation invariance alone is sufficient to induce LMC. Building on this insight, we develop a Weight Matching algorithm that enables alignment between independently trained MoEs, thereby facilitating the discovery of LMC. 4. In Section 6 , we provide empirical evidence of LMC across a wide range of MoE configurations–including dense, sparse, and shared-expert variants–evaluated under diverse model settings and across datasets of varying scales and modalities. Additionally, we assess the effectiveness of our proposed expert-matching algorithms, and conduct ablation studies to analyze LMC at different layers within deep MoE models. Section 2 offers background on LMC and MoE architectures. Table of notation, along with theoretical foundations, experimental details, and additional content, is provided in the Appendix. 2 Preliminaries This section provides an overview of Linear Mode Connectivity and Mixture-of-Experts architectures. 2.1 Linear Mode Connectivity Let f(\cdot;\theta) be a function parameterized by \theta\in\Theta , where \Theta denotes the weight space. Given a non-negative loss function \mathcal{L}(\theta) , we seek to minimize \mathcal{L}(\theta) over \Theta . The notion of the loss barrier was originally introduced by ( frankle2020linear ) and later formalized by ( entezari2021role ) as follows: for \theta_{A},\theta_{B}\in\Theta , define B(\theta_{A},\theta_{B})=\sup_{t\in[0,1]}\left[\mathcal{L}(t\theta_{A}+(1-t)\theta_{B})-\left(t\mathcal{L}(\theta_{A})+(1-t)\mathcal{L}(\theta_{B})\right)\right]. (1) \theta_{A} and \theta_{B} are said to be linearly mode connected if their loss barrier is negligible, i.e., B(\theta_{A},\theta_{B})\approx 0 . In many architectures, the function f(\cdot;\theta) is invariant under certain permutations of the weight space–namely, for a permutation g and all \theta\in\Theta , we have f(\cdot;\theta)=f(\cdot;g\theta) . Therefore, \theta_{A} and \theta_{B} are said to be linearly mode connected up to permutation if there exists a permutation g such that B(\theta_{A},g\theta_{B})\approx 0 . Any such permutation g is referred to as a winning permutation ( entezari2021role ) . 2.2 Mixture-of-Experts Architectures A Mixture-of-Experts (MoE) is a neural network architecture that combines the outputs of multiple models into a single prediction through a gating mechanism that assigns input-dependent weights. The two most commonly used gating strategies are dense and sparse gating. While we provide a brief overview of these concepts here, a comprehensive and formal treatment is deferred to Appendix A . Expert. Let d denote the input dimension. We define an Expert as a function \mathcal{E}(\cdot;\theta)\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} , parameterized by \theta\in\mathbb{R}^{e} , where e\in\mathbb{N} represents the total number of trainable parameters of \mathcal{E} . In this work, we consider each expert \mathcal{E}(\cdot;\theta) to be a feedforward neural network with ReLU activation functions. Unless stated otherwise, all experts are assumed to share the same architecture. Mixture-of-Experts with dense gating. Given n denoting the number of experts, we define Mixture-of-Experts with dense gating as a function \mathcal{D}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} such that \displaystyle\mathcal{D}\big(x;\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n}\big)=\sum_{i=1}^{n}\text{softmax}_{i}\big(s_{1}(x),\ldots,s_{n}(x)\big)\mathcal{E}(x;\theta_{i}), (2) where s_{i}(x)=W_{i}x+b_{i} determines the contribution of each expert to the final output. Here, \theta_{i} are the parameters of the i^{\text{th}} expert. The function s=(s_{1},\ldots,s_{n}) is called the gating score , and is parameterized as s(\cdot;\{W_{i},b_{i}\}_{i=1}^{n}) with (W_{i},b_{i})\in\mathbb{R}^{d}\times\mathbb{R} are the corresponding gating parameters. Mixture-of-Experts with sparse gating. Given a positive integer k\leq n denoting the number of activated experts, define the \mathrm{Top}\text{-}k map by \text{Top-}k(z)=\{i_{1},\ldots,i_{k}\} for z=(z_{1},\ldots,z_{n})\in\mathbb{R}^{n} , where i_{1},\ldots,i_{k} are the indices corresponding to the k largest components of x . In the event of ties, we select smaller indices first. We define a Mixture-of-Experts with sparse gating (SMoE) as a function \mathcal{S}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} as follows. For x\in\mathbb{R}^{d} , let T(x)=\text{Top-}k(s(x)) , then define: \displaystyle\mathcal{S}\big(x;\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n}\big)=\sum_{i\in T(x)}\operatorname{softmax}_{i}\Big(\big(s_{i}(x)\big)_{i\in T(x)}\Big)\cdot\mathcal{E}(x;\theta_{i}) (3) In other words, the Top- k selects the k highest-scoring experts used to compute the output. 3 Group Action on Weight Space of Mixture-of-Experts In this section, we introduce the formal notion of the weight space associated with MoE models. Furthermore, we define a group action on this space that preserves the functionality of MoE. A complete and rigorous exposition of these definitions and their implications is provided in Appendix A . 3.1 Weight Space of Mixture-of-Experts The map \mathcal{D} is parameterized by: \displaystyle\phi=(W_{i},b_{i},\theta_{i})_{i=1,\ldots,n}\in\Phi(n)\coloneqq(\mathbb{R}^{d}\times\mathbb{R}\times\Theta)^{n}=(\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R}^{e})^{n}. (4) Here, \Phi(n) is called the weight space of a Mixture-of- n -Experts. Varying the number of experts leads to a general Mixture-of-Experts weight space that spans across expert sets of different sizes, denoted by \Phi=\sqcup_{n>0}\Phi(n) . Note that the weight space of \mathcal{E} coincides with that of \mathcal{S} , since the map \text{Top-}k does not introduce any new trainable parameters. 3.2 Group Action on Weight Space of Mixture-of-Experts We define the group G(n) as the direct product G(n)=\mathbb{R}^{d}\times\mathbb{R}\times\text{S}_{n} of the groups \mathbb{R}^{d} , \mathbb{R} with addition, and the permutation group \text{S}_{n} . Each element g\in G(n) is of the form g=(c_{W},c_{b},\tau) , where c_{W}\in\mathbb{R}^{d},c_{b}\in\mathbb{R} and \tau\in\text{S}_{n} . The group G(n) acts on the weight space \Phi(n) as follows. For g\in G(n) and \phi\in\Phi(n) presented as in Equation ( 4 ), define: \displaystyle g\phi\coloneqq\big(W_{\tau(i)}+c_{W},b_{\tau(i)}+c_{b},\theta_{\tau(i)}\big)_{i=1,\ldots,n}\in\Phi(n). (5) The result below establishes that this group action preserves the MoE map. Proposition 3.1 (Weight space invariance of Mixture-of-Experts) . The MoE function \mathcal{D} is G(n) -invariant under the action of G(n) on its weight space \Phi(n) , i.e., \mathcal{D}(\cdot;\phi)=\mathcal{D}(\cdot;g\phi) . A proof of Proposition 3.1 is presented in Proposition A.3 . An analogous invariance result holds in the case of the SMoE function \mathcal{S} . However, since the Top- k selection map is generally discontinuous–primarily due to tie cases in the gating scores–additional conditions are required to ensure the validity of the invariance result. To address this, we focus on a subset of \mathbb{R}^{d} where the Top- k scores are unambiguously defined. Specifically, for \{W_{i},b_{i}\}_{i=1}^{n}\in(\mathbb{R}^{d}\times\mathbb{R})^{n} , we define: \displaystyle\Omega\big(\{W_{i},b_{i}\}_{i=1}^{n}\big)\coloneqq\big\{x\in\mathbb{R}^{d}~\colon~(W_{i}x+b_{i})_{i=1}^{n}\text{ are pairwise distinct}\big\}. (6) The SMoE function \mathcal{S} exhibits well-behaved properties on this domain. First, consider the case where the pairs \{W_{i},b_{i}\}_{i=1}^{n} are not pairwise distinct. In this case, the set \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) is clearly empty. However, when the pairs \{W_{i},b_{i}\}_{i=1}^{n} are pairwise distinct, the set \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) becomes an open and dense subset of \mathbb{R}^{d} . Moreover, the SMoE function \mathcal{S} is continuous on \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) . These properties are formally established in Propositions A.1 and A.2 . We now show that the invariance property of the SMoE map holds under restriction to this domain. Proposition 3.2 (Weight space invariance of Sparse Mixture-of-Experts) . Given the SMoE function \mathcal{S} , as defined in Equation ( 3 ). Assume that \{W_{i},b_{i}\} are pairwise distinct for i=1,\ldots,n . Then, the set \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) is invariant under the group action of G(n) , i.e., for g=(c_{W},c_{b},\tau)\in G(n) , we have \Omega(\{W_{i},b_{i}\}_{i=1}^{n})=\Omega(\{W_{\tau(i)}+c_{W},b_{\tau(i)}+c_{b}\}_{i=1}^{n}) . Moreover, the function \mathcal{S} , restricted to \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) , is invariance under the action of G(n) on its weight space \Phi(n) , i.e. \mathcal{S}(\cdot;\phi)=\mathcal{S}(\cdot;g\phi) on \Omega(\{W_{i},b_{i}\}_{i=1}^{n}) . A proof of Proposition 3.2 is presented in Proposition A.4 . Remark 3.3 . The permutation invariance of the summation operator and the translation invariance of the softmax function are the two primary sources of invariance for the functions \mathcal{D} and \mathcal{S} , as stated in Propositions 3.1 and 3.2 . In the case of SMoE, these invariance properties are additionally upheld by the permutation and translation invariance of the Top- k operator. 4 Symmetries and Functional Equivalence in Mixture-of-Experts In the remainder of this section, we let \phi\in\Phi(n) and \phi^{\prime}\in\Phi(n^{\prime}) denote the parameters of two Mixture-of-Experts models with n and n^{\prime} experts, respectively: \displaystyle\phi=(W_{i},b_{i},\theta_{i})_{i=1,\ldots,n}\in\Phi(n),\quad\text{and}\quad\phi^{\prime}=(W^{\prime}_{i},b^{\prime}_{i},\theta^{\prime}_{i})_{i=1,\ldots,n^{\prime}}\in\Phi(n^{\prime}). (7) 4.1 Functional Equivalence in Mixture-of-Experts We aim to characterize the conditions under which distinct parameter sets yield functionally equivalent MoE models. The central result of this section is that the symmetries of the gating mechanism are fully characterized by the group action of G(n) , as defined in Equation ( 5 ). Given the fundamentally different structural and analytical characteristics of the two gating mechanisms, we treat each case independently in the analysis. In addition, we introduce a set of assumptions, the motivations for which are discussed in detail in the following subsection. We begin with the case of dense gating. Theorem 4.1 (Functional equivalence in Mixture-of-Experts with Dense Gating) . Suppose \phi,\phi^{\prime} define the same MoE function, i.e., \mathcal{D}(\cdot;\phi)=\mathcal{D}(\cdot;\phi^{\prime}) . Assume that the following conditions hold: 1. Both \{\mathcal{E}(\cdot;\theta_{i})\}_{i=1}^{n} and \{\mathcal{E}(\cdot;\theta_{i}^{\prime})\}_{i=1}^{n^{\prime}} consist of pairwise distinct functions; 2. Both \{W_{i}-W_{j}\}_{1\leq i1 , and established under a set of assumptions that are stronger than those previously required. Theorem 4.2 (Functional equivalence in Mixture-of-Experts with Sparse Gating) . Suppose \phi,\phi^{\prime} define the same SMoE function, i.e., \mathcal{S}(\cdot;\phi)=\mathcal{S}(\cdot;\phi^{\prime}) . Assume that the following conditions hold: 1. Both \{\mathcal{E}(\cdot;\theta_{i})\}_{i=1}^{n} and \{\mathcal{E}(\cdot;\theta_{i}^{\prime})\}_{i=1}^{n^{\prime}} consist of pairwise strongly distinct functions; 2. Both \{W_{i-1}-W_{i}\}_{i=2}^{n} and \{W_{i-1}^{\prime}-W^{\prime}_{i}\}_{i=2}^{n^{\prime}} are linearly independent subsets of \mathbb{R}^{d} . Then n=n^{\prime} , and there exists g=(c_{W},c_{b},\tau)\in G(n) such that for all i=1,\ldots,n , we have W_{i}^{\prime}=W_{\tau(i)}+c_{W} , b_{i}^{\prime}=b_{\tau(i)}+c_{b} , and \mathcal{E}(x;\theta_{i}^{\prime})=\mathcal{E}(x;\theta_{\tau(i)}) for all x\in\Omega(\{W_{i},b_{i}\}_{i=1}^{n}) such that \tau(i)\in\textup{Top-}k((W_{j}x+b_{j})_{j=1}^{n}) . The proofs of Theorems 4.1 and 4.2 are provided in Appendix B and Appendix C , respectively. Both arguments build upon two essential ingredients: a result establishing the linear independence of exponential functions, as formulated in Lemma B.3 , and a key observation on the piecewise affine structure of ReLU networks, discussed in Appendix B.1 . Remark 4.3 . While Theorem 4.2 shares a conceptual parallel with Theorem 4.1 , it is important to underscore that the sparse gating case presents significantly deeper mathematical difficulties . The core source of complexity lies in the discontinuous behavior of the Top- k operator, which induces nontrivial discontinuities in the gating function, thereby disrupting smoothness and complicating the functional analysis required to establish equivalence. As a result, standard analytical techniques used in the dense setting become insufficient, necessitating more delicate arguments to account for the combinatorial and piecewise structure inherent to sparse gating. Theorems 4.1 and 4.2 provide a rigorous characterization of functional equivalence in both dense and sparse MoE models, with particular attention given to the role and structure of the gating mechanism. Nonetheless, it is important to note that these results do not account for all potential symmetries inherent in the MoE and SMoE mappings defined in Equations ( 2 ) and ( 3 ). Notably, further symmetries may exist within the expert networks, especially when they are implemented as ReLU neural networks. Given that this work is primarily concerned with the MoE architecture itself, our analysis is restricted to the behavior of the gating mechanism. Consequently, we treat the experts as black-box functions and abstract away from their internal parameterizations. 4.2 Necessity and Implications of Technical Assumptions At a conceptual level, symmetry analysis aims to identify universal invariances –those that persist regardless of specific parameter values–while explicitly excluding singular symmetries , which emerge only under special, degenerate configurations of the model parameters. The assumptions introduced in our results are designed precisely to rule out such singularities, which do not represent inherent structural invariances of the architecture but instead arise from pathological or measure-zero subsets of parameter space. We provide a brief overview of the motivation behind these assumptions here; a more detailed justification, along with illustrative examples, is given in Remarks B.6 and C.8 . The case of dense gating. Assumption 1 is introduced to eliminate degenerate scenarios in which two experts compute identical functions and receive identical gating scores–situations where permuting the corresponding experts has no effect on the model’s output. By excluding linear dependencies among the gating weight vectors, Assumption 2 ensures that expert activations remain distinguishable–thereby preventing the emergence of non-structural, symmetry-like artifacts. The case of sparse gating. Theorem 4.2 relies on a stronger set of assumptions than those required in Theorem 4.1 . This added strength is necessary due to a key feature of sparse gating: an expert’s behavior on inputs where it is not selected is unconstrained, meaning that the expert can act arbitrarily outside its region of activation. Consequently, different sets of expert functions may result in the same overall model output, provided their outputs agree where they are active. The case of k=1 . In the particular case where k=1 , the SMoE architecture employs a Top- 1 gating mechanism that activates only the expert associated with the highest gating score. Consequently, the softmax output reduces to a one-hot vector, with a single component equal to 1 . Under this regime, the SMoE map exhibits additional nontrivial symmetries, specifically invariance under the multiplicative group \mathbb{R}_{>0} . That is, for any positive scalar c>0 , the following identity holds: \displaystyle\mathcal{S}\big(x;\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n}\big)=\mathcal{S}\big(x;\{cW_{i},cb_{i},\theta_{i}\}_{i=1}^{n}\big). (8) This invariance holds because the argmax used for expert selection is unaffected by positive scaling, i.e., \text{argmax}_{i=1,\ldots,n}\left(W_{i}x+b_{i}\right)=\text{argmax}_{i=1,\ldots,n}(cW_{i}x+cb_{i}) , for all x\in\Omega(\{W_{i},b_{i}\}_{i=1}^{n}) . Additionally, since only a single expert contributes to the output, the model lacks any form of explicit aggregation across experts. Due to the analytical challenges posed by these additional invariances, we do not consider k=1 in our primary results and instead defer its investigation to future work. 5 Linear Mode Connectivity of Mixture-of-Experts Architectures This section outlines the theoretical foundations of LMC in MoE architectures. We first show that permutation invariance suffices to explain the existence of LMC in MoEs, then propose an algorithm to identify such permutations, which we later employ to empirically assess LMC in MoE models. 5.1 Permutation Invariance Sufficiency in Linear Mode Connectivity of Mixture-of-Experts Theorems 4.1 and 4.2 show that the group action as in Equation ( 5 ) suffices to uncover LMC between two MoE models. Each group element g=(c_{W},c_{b},\tau)\in G(n) consists of two components: a translation term (c_{W},c_{b})\in\mathbb{R}^{d}\times\mathbb{R} applied to the gating function, and a permutation \tau that reorders both the experts and their associated gating scores. Observe that the translation component does not affect the barrier loss defined in Equation ( 1 ). Indeed, for any h=(c_{W},c_{b},\textup{id}_{n})\in G(n) –with \textup{id}_{n} denoting the identity permutation in \textup{S}_{n} –the loss barrier remains unchanged: B(\phi_{A},\phi_{B})=B(\phi_{A},h\phi_{B}). (9) This invariance, established in Proposition D.1 , indicates that only the permutation component \tau influences LMC behavior. Combined with prior work showing that permutation symmetries are sufficient to induce LMC in standard neural networks ( entezari2021role ; ferbach2024proving ) , this suggests that analyzing permutations alone suffices to capture LMC in MoE models. 5.2 Permutation Alignment Algorithm for Mixture-of-Experts We propose the Permutation Alignment Algorithm to align MoEs. Inspired by the data-independent Weight Matching algorithm ( ainsworth_git_re_basin_2022 ) , our method operates in the parameter space of two MoEs, enabling efficient permutation alignment tailored to MoE architectures in large-scale settings. Proposed Algorithm. In MoEs, equivalent functionality can arise from different expert orderings and internal neuron configurations. To address this, we propose a two-stage alignment method that first reorders experts to match their roles, then aligns the internal weights of corresponding experts. Both steps are formulated as Linear Assignment Problems (LAPs) bertsekas1998network . An LAP assigns N tasks to N agents using a cost matrix C\in\mathbb{R}^{N\times N} , where each entry C_{i,j} represents the cost of assigning task i to agent j . The objective is to find a permutation \pi that minimizes the total cost: \min_{\pi\in\text{S}_{N}}\sum_{i=1}^{N}C_{i,\pi(i)} . This is efficiently solved using the Hungarian algorithm in O(N^{3}) time ( kuhn1955hungarian ; jonker1988shortest ; crouse2016implementing ) . Suppose we have \phi=(W_{i},b_{i},\theta_{i})_{i=1,\ldots,n} and \phi^{\prime}=(W^{\prime}_{i},b^{\prime}_{i},\theta^{\prime}_{i})_{i=1,\ldots,n} representing the parameters of two distinct MoEs with n experts. In practice, each expert is implemented as an MLP with one hidden layer ( fedus2022switch ; lepikhin2020gshard ; du2022glam ) . We denote the parameters of expert i in \phi and expert j in \phi^{\prime} as \theta_{i}=\{A_{i},u_{i},B_{i},v_{i}\} and \theta^{\prime}_{j}=\{A^{\prime}_{j},u^{\prime}_{j},B^{\prime}_{j},v^{\prime}_{j}\} , respectively, where A_{i}\in\mathbb{R}^{h\times d} , u_{i}\in\mathbb{R}^{h} , B_{i}\in\mathbb{R}^{d\times h} , v_{i}\in\mathbb{R}^{d} , with h as the hidden dimension. Denote \widetilde{A}_{i}=[A_{i},u_{i}]\in\mathbb{R}^{h\times(d+1)},\quad\widetilde{B}_{i}=[B_{i},v_{i}]\in\mathbb{R}^{d\times(h+1)}, and similarly for \widetilde{A}^{\prime}_{j} and \widetilde{B}^{\prime}_{j} , as concatenated matrices. Table 1: Configurations for experiments analyzing LMC in Transformers with dense MoE replacing the FFN in the first layer (refer to Table 5 for SMoE and DeepSeekMoE). For each dataset, number of layers, and number of experts, we visualize LMC curves for the three pairs among three models in the referenced figures. Datasets denoted as A\rightarrow B indicate pretraining on dataset A , fine-tuning on dataset B , and evaluating on B . Dataset Layers Experts Figure Dataset Layers Experts Figure MNIST 1 [2, 4] [ 5 , 6 ] ImageNet-21k \rightarrow CIFAR-10 12 [2, 4, 6] [ 18 , 19 , 20 ] 2 [2, 4] [ 7 , 8 ] ImageNet-21k \rightarrow CIFAR-100 12 [2, 4, 6] [ 21 , 22 , 23 ] CIFAR-10 2 [2, 4, 6] [ 9 , 10 , 11 ] ImageNet-1k 12 [2, 4, 6, 8] [ 24 , 25 , 26 , 27 ] 6 [2, 4, 6] [ 12 , 13 , 14 ] WikiText103 12 [2, 4, 6, 8] [ 28 , 29 , 30 , 31 ] CIFAR-100 6 [2, 4, 6] [ 15 , 16 , 17 ] One Billion Word 12 [2, 4, 6, 8] [ 32 , 33 , 34 , 35 ] Figure 1: LMC curves for ViT (subplots 1-3) and GPT-2 (subplot 4) with a 4-expert MoE replacement at the first Transformer layer, on CIFAR-100, ImageNet21k \rightarrow CIFAR-100, ImageNet-1k, and One Billion Word datasets, respectively. Plots show consistent low-loss linear interpolation paths between fine-tuned models, indicating strong linear mode connectivity. Matching Order of Experts. In MoEs, experts and their gating can be reordered without affecting functionality. We propose two methods to align the expert order between \phi and \phi^{\prime} , using LAP. Method 1 - Gating Weights. To address the translation-invariance of the softmax function in the gating, we center the weights and biases of the gating. For \phi , these are \widehat{W}_{i}=W_{i}-\frac{1}{n}\sum_{m=1}^{n}W_{m} and \hat{b}_{i}=b_{i}-\frac{1}{n}\sum_{m=1}^{n}b_{m} per expert i , and similarly for \phi^{\prime} . The cost matrix with respect to gating weights is defined as C=\{C_{i,j}\}_{1\leq i\leq n,1\leq j\leq n}\in\mathbb{R}^{n\times n} , where C_{i,j}=\left(\|\widehat{W}_{i}-\widehat{W}^{\prime}_{j}\|_{2}^{2}+(\hat{b}_{i}-\hat{b}^{\prime}_{j})^{2}\right)^{\frac{1}{2}}, (10) measuring dissimilarity between centered gating parameters of expert i in \phi and expert j in \phi^{\prime} . Method 2 - Internal Weights. Experts exhibit permutation invariance in hidden neuron orderings. We use Gram matrices for a permutation-invariant representation (see Appendix D.2 ). The cost matrix with respect to expert internal weights is defined as C=\{C_{i,j}\}_{1\leq i\leq n,1\leq j\leq n}\in\mathbb{R}^{n\times n} , where C_{i,j}=\left(\left\|(\widetilde{A}_{i})^{\top}\widetilde{A}_{i}-(\widetilde{A}^{\prime}_{j})^{\top}\widetilde{A}^{\prime}_{j}\right\|_{F}^{2}+\left\|\widetilde{B}_{i}(\widetilde{B}_{i})^{\top}-\widetilde{B}^{\prime}_{j}(\widetilde{B}^{\prime}_{j})^{\top}\right\|_{F}^{2}\right)^{\frac{1}{2}}, (11) quantifying dissimilarity between weight representations of expert i in \phi and expert j in \phi^{\prime} . For both methods, the optimal expert ordering \tau is determined by solving the LAP: \tau=\arg\min_{\pi\in\text{S}_{n}}\sum_{i=1}^{n}C_{i,\pi(i)} in O(n^{3}) time. Aligning Expert Internal Weights. After finding the permutation \tau to align experts between \phi and \phi^{\prime} , we align the internal weights of each matched expert pair (i,\tau(i)) using established MLP alignment methods ( ainsworth2022git ; sinkhorn_re_basin_2023 ; entezari_permutation_invariances_2022 ; singh_jaggi_otfusion_2021 ) , specifically the Weight Matching algorithm ( ainsworth_git_re_basin_2022 ) . This LAP-based algorithm aligns two MLPs with one hidden layer of size h in O(h^{3}) time, ensuring efficient scaling. Applying this to each pair (i,\tau(i)) guarantees consistent neuron ordering across aligned experts. 5.3 Weight Matching Algorithm We propose a Weight Matching Algorithm designed to align MoEs by addressing permutation invariance in expert ordering and internal parameters, as detailed in Algorithm 1 . The algorithm employs gate-based and expert-based methods to optimally permute experts, with both performing comparably well in loss barrier evaluations during model interpolation (see Section 6.3 ). Operating solely in parameter space, the algorithm avoids data-dependent computational overhead, with a complexity of O(n^{3}+nh^{3}) , ensuring scalability for large-scale MoE alignment tasks. 6 Experiments Table 2: Comparison of the two expert matching methods for 12-layer ViT-MoE models with 4 experts on CIFAR-10 and CIFAR-100, showing Rank (out of 24 permutations) and \hat{L} across layer placements, averaged over 10 checkpoint pairs. See Tables 7 and 8 for SMoE and DeepSeekMoE. Dataset Layer replaced Expert Weight Matching Gate Weight Matching Dataset Layer replaced Expert Weight Matching Gate Weight Matching Rank \downarrow \hat{L} \downarrow Rank \downarrow \hat{L} \downarrow Rank \downarrow \hat{L} \downarrow Rank \downarrow \hat{L} \downarrow CIFAR-10 1 2.50 \pm 1.50 2.12 \pm 0.42 3.00 \pm 1.00 3.42 \pm 0.55 CIFAR-100 1 2.80 \pm 0.40 3.17 \pm 0.25 2.90 \pm 0.70 2.73 \pm 1.03 4 2.10 \pm 0.50 1.04 \pm 0.32 2.60 \pm 0.92 1.54 \pm 0.44 4 3.60 \pm 1.20 1.15 \pm 0.55 2.70 \pm 1.00 2.03 \pm 0.93 8 2.70 \pm 0.46 0.60 \pm 0.27 2.90 \pm 0.30 0.74 \pm 0.17 8 3.30 \pm 0.78 0.67 \pm 0.14 3.20 \pm 0.87 1.13 \pm 0.55 12 4.60 \pm 2.00 0.13 \pm 0.05 3.80 \pm 1.66 0.09 \pm 0.03 12 3.40 \pm 0.92 0.07 \pm 0.03 4.20 \pm 0.89 0.11 \pm 0.04 Figure 2: Loss curves for 12-layer ViT-MoE models with a 4-expert MoE replacement in either the first layer (subplots 1, 3) or the last layer (subplots 2, 4), on CIFAR-10 and CIFAR-100. The curves compare two Expert Order Matching methods across 24 permutations, with Weight Matching applied post-reordering to all permutations. Corresponding accuracy metrics are presented in Figure 96 . This section provides empirical validation of LMC across three MoE variants, including dense MoE, sparse MoE (SMoE), and DeepSeekMoE, detailed in Section 6.1 . By analyzing interpolation paths between independently fine-tuned models–each subject to permutation invariance–low-loss connections are consistently observed, indicating shared solution basins in the optimization landscape. 6.1 Experimental Setup Experimental Design. We investigate LMC by replacing the Feedforward Network (FFN) in the first Transformer vaswani2017attention layer with a randomly initialized MoE, based on empirical evidence from Section 6.4 , Appendices E and G.2 , indicating lower perturbation sensitivity when replacing deeper FFN layers with MoEs. Only the MoE parameters are fine-tuned using multiple random seeds for each experiment. LMC is evaluated by linearly interpolating between all checkpoint pairs, measuring model performance on the test set at 25 evenly spaced points along the interpolation path. We examine three MoE variants: dense MoE ( jacobs1991adaptive ; jordan1994hierarchical ) , SMoE with top-2 routing ( k=2 ) ( fedus2022switch ) , and DeepSeekMoE with top-2 and one shared expert ( k=2,s=1 ) (see Appendix D.4 for formal formulations) ( liu2024deepseek ) . Datasets and Models. We use ViT ( dosovitskiy2020image ) for image classification (MNIST ( lecun1998gradient ) , CIFAR-10/100 ( krizhevsky2009learning ) , ImageNet ( deng2009imagenet ) ) and GPT-2 ( radford2019language ) for language modeling (WikiText103 ( merity2016pointer ) and One Billion Word ( chelba2013one ) ). Hyperparameters such as batch size, optimizer, number of experts, and hidden size are fixed, while the learning rate is tuned per setting. Full details are provided in Appendix F . 6.2 Linear Mode Connectivity Verification Experimental Objective. We analyze LMC in pretrained Transformer models where the FFN of a Transformer layer is replaced with an MoE. Our study focuses on two configurations: replacing the FFN in the first and last Transformer layers. For each configuration, we fine-tune three independently initialized models. We vary the number of experts (e.g., 2, 4, 6, 8, 16) and the number of Transformer layers to assess LMC under different architectural scales. LMC is evaluated by linearly interpolating between all model pairs using our proposed algorithm (Algorithm 1 ). Results. Table 1 shows results for first-layer MoE replacement, with full details for three MoE variants in Table 5 . Last-layer results are presented in Appendix G.2 . LMC persists in all cases, but is less pronounced in the last layer due to greater stability and reduced influence on convergence. Early layers strongly shape the solution basin, making first-layer changes more revealing of connectivity. This depth-dependent effect is also demonstrated in Section 6.4 and Appendix E . Figure 1 presents representative LMC curves across our testing cases. 6.3 Expert Order Matching Method Experimental Objective. We evaluate two Expert Order Matching methods (Step 1, Algorithm 1 )–Expert Weight Matching and Gate Weight Matching–using 12-layer ViT models with 4-expert MoE replacements. The methods are assessed by ranking the selected permutation among all 24 possible expert permutations after Weight Matching (Step 2), with MoE integrations at layers 1, 4, 8, or 12 on CIFAR-10 and CIFAR-100, repeated five times. We report the rank and a scaled metric \hat{L}=\frac{L_{\text{method}}-L_{\text{top1}}}{L_{\text{naive}}-L_{\text{top1}}}\times 10^{2} , averaged over ten checkpoint pairs, where L_{\text{method}} , L_{\text{top1}} , and L_{\text{naive}} denote loss barriers of our methods, the best permutation, and the naive interpolation, respectively. Results. Table 2 shows both methods achieve low ranks and near-zero \hat{L} , indicating near-optimal matching. Full results are presented in Tables 7 (loss) and 8 (accuracy). Figure 2 visualizes the representative performance of our methods across all 24 permutations, highlighting the importance of correct expert order matching, as poor ordering can yield performance close to the naive interpolation. 6.4 Ablation Study on Number of Layers This study investigates the impact of the number of Transformer layers on LMC in MoE models on CIFAR-10. We evaluate models with 2, 4, and 6 layers, incorporating MoE at all possible layer positions. Employing Algorithm 1 , we compute ratios for four metrics–loss barrier, loss Area Under the Curve (AUC, relative to the straight line connecting the metrics of the two endpoint models), accuracy barrier, and accuracy AUC, relative to linear interpolation–across five repeated experiments, evaluating ten model pairs per configuration. Table 3 reports results for MoE, showing significant reduction ratios across all configurations. Corresponding results for SMoE and DeepSeekMoE are provided in Table 9 , with detailed loss and accuracy metrics presented in Tables 10 and 11 . Table 3: Ablation Study on varying the number of Transformer layers and MoE layer replacement in ViT for CIFAR-10. The ratios of metrics (loss barrier and AUC, accuracy barrier and AUC) compare Algorithm 1 to naive interpolation. For all MoE variants, refer to Tables 9 , 10 , and 11 . Layers Layer replaced Loss Barrier \downarrow Loss AUC \downarrow Accuracy Barrier \downarrow Accuracy AUC \downarrow Layers Layer replaced Loss Barrier \downarrow Loss AUC \downarrow Accuracy Barrier \downarrow Accuracy AUC \downarrow 2 1 8.54 \pm 1.53 8.73 \pm 1.68 8.39 \pm 1.42 8.01 \pm 1.39 6 1 4.16 \pm 1.47 4.96 \pm 2.31 3.64 \pm 1.00 4.13 \pm 1.55 2 9.33 \pm 2.04 8.94 \pm 1.92 9.10 \pm 2.19 8.83 \pm 2.22 2 6.78 \pm 4.80 7.71 \pm 3.09 7.27 \pm 5.73 8.98 \pm 2.94 4 1 8.29 \pm 1.63 8.08 \pm 1.59 7.41 \pm 1.45 6.43 \pm 1.09 3 9.69 \pm 4.92 8.62 \pm 5.46 9.03 \pm 3.31 8.14 \pm 3.57 2 10.19 \pm 3.43 10.19 \pm 3.16 10.21 \pm 5.26 9.08 \pm 5.33 4 10.83 \pm 5.61 12.01 \pm 6.42 8.92 \pm 2.21 10.00 \pm 7.30 3 8.50 \pm 1.82 7.83 \pm 1.89 9.82 \pm 2.58 8.70 \pm 2.58 5 9.56 \pm 4.52 11.72 \pm 5.59 8.72 \pm 2.89 11.01 \pm 3.83 4 5.12 \pm 0.67 4.60 \pm 0.65 4.17 \pm 1.68 4.58 \pm 1.01 6 6.90 \pm 3.54 9.67 \pm 6.59 7.16 \pm 3.98 9.69 \pm 3.11 7 Conclusion This paper presents a systematic analysis of LMC in MoE models. We show that functional equivalence via permutation symmetries is sufficient to construct low-loss linear paths–when they exist–between independently trained models. To enable this, we introduce an algorithm for identifying such permutations. Empirical evaluations confirm the presence of LMC across a wide range of MoE configurations–including dense, sparse, and shared-expert variants–under diverse hyperparameter settings and across datasets of varying scales and modalities. While our method does not provide theoretical bounds on the loss barrier, this remains a common limitation across prior LMC studies. Overall, our work offers a principled foundation for extending LMC analysis to other architectures, such as Transformers and State Space Models. Acknowledgments and Disclosure of Funding Use unnumbered first level headings for the acknowledgments. All acknowledgments go at the end of the paper before the list of references. Moreover, you are required to declare funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work). More information about this disclosure can be found at: https://neurips.cc/Conferences/2025/PaperInformation/FundingDisclosure . Do not include this section in the anonymized submission, only in the final paper. You can use the ack environment provided in the style file to automatically hide this section in the anonymized submission. Table of Notation \mathbb{R}^{d} d -dimensional Euclidean space \mathbb{S}^{d-1} (d-1) -dimensional hypersphere \|\cdot\|_{2} Euclidean norm \|\cdot\|_{F} Frobenius norm \mathcal{E} Expert function \mathcal{D} Mixture-of-Experts with Dense gating \mathcal{S} Mixture-of-Experts with Sparse gating Top- k(\cdot) The Top- k map \theta,\theta^{\prime} parameter of Expert functions \Theta weight space of Expert functions \phi,\phi^{\prime} parameter of Mixture-of-Experts \Phi(n) weight space of Mixture-of- n -Experts \text{S}_{n} permutation group of order n \tau element of permutation group c_{W},c_{b} elements of additive group of Euclidean space G,G(n) group act on weight space of Mixture-of- n -Experts \Omega,\Omega_{1},\Omega_{2} sets with specific purposes B barrier loss C,C_{ij} cost matrix and its entries A,B,u,v weight and bias of Expert functions P permutation matrix \partial boundary of a set in a topological space \sigma , ReLU rectifier activation function \mathcal{H} space of holomorphic functions \mathcal{F} space of meromorphic functions \mathbb{C}[x],\mathbb{C}[x_{1},\ldots,x_{n}] polynomial ring in complex variables \mathbb{C}(x),\mathbb{C}(x_{1},\ldots,x_{n}) field of rational functions p_{i},r_{i} polynomial Appendix of “On Linear Mode Connectivity of Mixture-of-Experts Architectures” Table of Contents Appendix A On the Weight Spaces of Mixture-of-Experts Architecture Denote the input token dimension as d . A.1 Weight Space of Mixture-of-Experts Expert functions. We study expert functions \mathcal{E}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} , which are realized by standard feedforward neural networks employing the ReLU activation function. Specifically, each expert map \mathcal{E}(\cdot;\theta) is represented as a composition of affine transformations and ReLU nonlinearities: \displaystyle\mathcal{E}(x;\theta)=f_{L}\circ\sigma\circ f_{L-1}\circ\cdots\circ\sigma\circ f_{1}(x), (12) where each affine layer is defined by \displaystyle f_{i}(x)=xW_{i}+b_{i},\quad\text{for }i=1,\dots,L. (13) Here, \sigma denotes the ReLU activation function, applied component-wise, and \theta=\{W_{i},b_{i}\}_{i=1}^{L} represents the full set of learnable parameters. The weight matrix W_{i}\in\mathbb{R}^{n_{i-1}\times n_{i}} and bias vector b_{i}\in\mathbb{R}^{n_{i}} parameterize the i -th layer, where n_{0}=n_{L}=d is the input and output dimensions, respectively. We have \displaystyle\theta\in\prod_{i=1}^{L}(\mathbb{R}^{n_{i-1}\times n_{i}}\times\mathbb{R}^{n_{i}})=\mathbb{R}^{e},~\text{ where }~e=\sum_{i=1}^{L}(n_{i-1}n_{i}+n_{i}). (14) Define the weight space of expert functions as: \displaystyle\Theta=\prod_{i=1}^{L}\left(\mathbb{R}^{n_{i-1}\times n_{i}}\times\mathbb{R}^{n_{i}}\right)=\mathbb{R}^{e}. (15) Dense Mixture-of-Experts. Given a positive integer n representing the number of experts, a Mixture-of-Experts with Dense gating (MoE) is defined as the function: \mathcal{D}~\colon~\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} defined by \displaystyle\mathcal{D}\left(x;\left\{W_{i},b_{i},\theta_{i}\right\}_{i=1}^{n}\right)=\sum_{i=1}^{n}\operatorname{softmax}_{i}\left(\left\{W_{i}x+b_{i}\right\}_{i=1}^{n}\right)\cdot\mathcal{E}\left(x;\theta_{i}\right). (16) The function \mathcal{D} is parameterized as \mathcal{D}(x;\phi) where \displaystyle\phi=\Big(\big(W_{i},b_{i}\big),\theta_{i}\Big)_{i=1,\ldots,n}\in\Big(\big(\mathbb{R}^{D}\times\mathbb{R}\big)\times\Theta\Big)^{n}. (17) Denote the weight space of a Mixture of n Experts as \displaystyle\Phi(n)=\Big(\big(\mathbb{R}^{D}\times\mathbb{R}\big)\times\Theta\Big)^{n}. (18) Varying the number of experts leads to a Mixture-of-Experts weight space that spans across expert sets of different sizes, denoted by \displaystyle\Phi=\bigsqcup_{n=1}^{\infty}\Phi(n)=\bigsqcup_{n=1}^{\infty}\Big(\big(\mathbb{R}^{D}\times\mathbb{R}\big)\times\Theta\Big)^{n}. (19) Sparse Mixture-of-Experts. Given a positive integer k\leq n , the \text{Top}\text{-}k map is defined by: for any vector z=(z_{1},\ldots,z_{n})\in\mathbb{R}^{n} , \displaystyle\text{Top-}k(z)=\{i_{1},\ldots,i_{k}\}, (20) where i_{1},\ldots,i_{k} are the indices corresponding to the k largest components of x . In the event of ties, we select smaller indices first. Using this, a Mixture-of-Experts with Sparse gating (SMoE) is the function \mathcal{S}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} defined by \displaystyle\mathcal{S}\left(x;\left\{W_{i},b_{i},\theta_{i}\right\}_{i=1}^{n}\right)=\sum_{i\in T(x)}\operatorname{softmax}_{i}\left(\left\{W_{i}x+b_{i}\right\}_{i\in T(x)}\right)\cdot\mathcal{E}\left(x;\theta_{i}\right). (21) where \displaystyle T(x)=T\left(x;\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right)=\text{Top-}k\Big(\big(W_{i}x+b_{i}\big)_{i=1}^{n}\Big). (22) The weight space for the Sparse Mixture-of-Experts is identical to that of the Dense Mixture-of-Experts, as the inclusion of the \text{Top-}k operator does not introduce any additional trainable parameters. However, the SMoE function \mathcal{S} is generally discontinuous, primarily due to the non-smooth nature of the \text{Top-}k selection–particularly in scenarios where multiple experts share identical gating scores (i.e., ties occur). To circumvent this issue and ensure well-defined behavior, we restrict our analysis to a subset of the input space \mathbb{R}^{d} where the top K gating scores are uniquely determined without ambiguity. In particular, for \{W_{i},b_{i}\}_{i=1}^{n}\in(\mathbb{R}^{d}\times\mathbb{R})^{n} , we define the subset of \mathbb{R}^{d} such that the gating scores of all experts are pairwise distinct, i.e., \displaystyle\Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right)=\left\{x\in\mathbb{R}^{d}~\colon~\left(W_{i}x+b_{i}\right)_{i=1}^{n}\text{ are pairwise distinct}\right\}. (23) We provide one result characterizing the structure of this domain, and another result describing the behavior of the SMoE map when it is restricted to inputs within this domain. Proposition A.1 . If \left\{W_{i},b_{i}\right\} are pairwise distinct for i=1,\ldots,n , then \Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right) is an open and dense subset of \mathbb{R}^{d} . Proof. We have \displaystyle\Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right) \displaystyle=\left\{x\in\mathbb{R}^{d}~\colon~W_{i}x+b_{i}~\text{ is pairwise distinct for all }i=1,\ldots,n\right\} \displaystyle=\bigcap_{1\leq iW_{2}x+b_{2}>\cdots>W_{n}x+b_{n}. (111) We can strengthen this to require: \displaystyle\begin{array}[]{lcl}(W_{1}x+b_{1})-(W_{2}x+b_{2})&=&1,\\ (W_{2}x+b_{2})-(W_{3}x+b_{3})&=&1,\\ &\vdots&\\ (W_{n-1}x+b_{n-1})-(W_{n}x+b_{n})&=&1.\end{array} This is equivalent to the following system of linear equations: \displaystyle\begin{array}[]{lcl}(W_{1}-W_{2})x&=&1-(b_{1}-b_{2}),\\ (W_{2}-W_{3})x&=&1-(b_{2}-b_{3}),\\ &\vdots&\\ (W_{n-1}-W_{n})x&=&1-(b_{n-1}-b_{n}).\end{array} Since the vectors \{W_{i-1}-W_{i}\}_{i=2}^{n} are linearly independent by assumption, this system has a unique solution. Hence, there exists x\in\mathbb{R}^{d} satisfying Equation ( C.1 ). ∎ Proposition C.3 will be invoked in the proof of Theorem C.4 . A justification for the linear independence assumption is provided in Remark C.8 . C.2 Functional Equivalence in Mixture-of-Experts with Sparse Gating We establish a functional equivalence theorem for the Sparse Mixture-of-Experts (SMoE) architecture, in parallel with the result for MoE given in Theorem B.5 . However, our analysis is limited to the case k>1 , as the k=1 setting leads to singularities that disrupt the general structure of equivalence. A detailed explanation for this exclusion is provided in Remark C.9 . Theorem C.4 (Functional equivalence in Mixture-of-Experts with Sparse Gating) . Suppose \phi,\phi^{\prime} define the same SMoE function, i.e. \mathcal{S}(\cdot;\phi)=\mathcal{S}(\cdot;\phi^{\prime}) . Assume that the following conditions hold: 1. Both \{\mathcal{E}(\cdot;\theta_{i})\}_{i=1}^{n} and \{\mathcal{E}(\cdot;\theta_{i}^{\prime})\}_{i=1}^{n^{\prime}} consist of pairwise strongly distinct functions; 2. Both \{W_{i-1}-W_{i}\}_{i=2}^{n} and \{W_{i-1}^{\prime}-W^{\prime}_{i}\}_{i=2}^{n^{\prime}} are linear independent subsets of \mathbb{R}^{d} . Then n=n^{\prime} , and there exists g=(c_{W},c_{b},\tau)\in G(n) such that for all i=1,\ldots,n , we have W_{i}^{\prime}=W_{\tau(i)}+c_{W} , b_{i}^{\prime}=b_{\tau(i)}+c_{b} , and \mathcal{E}(x;\theta_{i}^{\prime})=\mathcal{E}(x;\theta_{\tau(i)}) for all x\in\Omega(\{W_{i},b_{i}\}_{i=1}^{n}) such that \tau(i)\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n}) . Before presenting the proof of Theorem C.4 , we begin with two remarks. Remark C.5 . Suppose n=n^{\prime} and there exist \tau\in\textup{S}_{n} , c_{W}\in\mathbb{R}^{d} , and c_{b}\in\mathbb{R} such that for every i=1,\ldots,n , we have \displaystyle W^{\prime}_{i}=W_{\tau(i)}+c_{W},\quad b^{\prime}_{i}=b_{\tau(i)}+c_{b}. (120) Then the two sets \Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right) and \Omega\left(\left\{W^{\prime}_{i},b^{\prime}_{i}\right\}_{i=1}^{n}\right) coincide. Furthermore, for any x in this set, the condition \tau(i)\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n}) holds if and only if i\in\textup{Top-}k((W^{\prime}_{i}x+b^{\prime}_{i})_{i=1}^{n}) . Remark C.6 . It is easy to verify that Assumption 2 in Theorem C.4 implies Assumption 2 in Theorem B.5 . Proof. To enhance clarity, we begin by outlining the main steps of the proof at a high level: 1. Formulate the identity \mathcal{S}(\cdot;\phi)=\mathcal{S}(\cdot;\phi^{\prime}) explicitly and introduce simplified notation to streamline the presentation. 2. Partition the input space into regions where the \text{Top-}k operator selects the same index set, and within which all expert functions are affine. 3. Demonstrate the desired equivalence for a fixed subset of experts. The core technique is to invoke the MoE equivalence result given in Theorem B.5 . 4. Generalize the argument to establish the equivalence across all experts. We now proceed with the detailed derivation and proofs for each of the outlined steps. Step 1. Given that \mathcal{S}(\cdot;\phi)=\mathcal{S}(\cdot;\phi^{\prime}) , it follows that \displaystyle\sum_{i\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n})}\operatorname{softmax}_{i}\left(\left\{W_{i}x+b_{i}\right\}_{i\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n})}\right)\cdot\mathcal{E}\left(x;\theta_{i}\right) \displaystyle=~ \displaystyle\sum_{i\in\textup{Top-}k((W^{\prime}_{i}x+b^{\prime}_{i})_{i=1}^{n^{\prime}})}\operatorname{softmax}_{i}\left(\left\{W^{\prime}_{i}x+b^{\prime}_{i}\right\}_{i\in\textup{Top-}k((W^{\prime}_{i}x+b^{\prime}_{i})_{i=1}^{n^{\prime}})}\right)\cdot\mathcal{E}\left(x;\theta^{\prime}_{i}\right), (121) for all x\in\mathbb{R}^{d} . To simplify notation, define \displaystyle\mathcal{E}_{i}(\cdot)=\mathcal{E}\left(\cdot;\theta_{i}\right),~~\text{ and }~~\mathcal{E}^{\prime}_{i}(\cdot)=\mathcal{E}\left(\cdot;\theta^{\prime}_{i}\right). (122) Using this notation, we can rewrite Equation ( C.2 ) more compactly as: \displaystyle\sum_{i\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n})}\operatorname{softmax}_{i}\left(\left\{W_{i}x+b_{i}\right\}_{i\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n})}\right)\cdot\mathcal{E}_{i}(x) \displaystyle=~ \displaystyle\sum_{i\in\textup{Top-}k((W^{\prime}_{i}x+b^{\prime}_{i})_{i=1}^{n^{\prime}})}\operatorname{softmax}_{i}\left(\left\{W^{\prime}_{i}x+b^{\prime}_{i}\right\}_{i\in\textup{Top-}k((W^{\prime}_{i}x+b^{\prime}_{i})_{i=1}^{n^{\prime}})}\right)\cdot\mathcal{E}^{\prime}_{i}(x). (123) Step 2. We begin by highlighting two key observations: • Assumption 2 guarantees that the parameter pairs \left\{W_{i},b_{i}\right\} are pairwise distinct for i=1,\ldots,n , and likewise, \left\{W^{\prime}_{i},b^{\prime}_{i}\right\} are pairwise distinct for i=1,\ldots,n^{\prime} . By Proposition A.1 , the set \displaystyle\Omega_{1}=\Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right)\cap\Omega\left(\left\{W^{\prime}_{i},b^{\prime}_{i}\right\}_{i=1}^{n^{\prime}}\right), (124) is open and dense in \mathbb{R}^{d} . For any x\in\Omega_{1} , the values \{W_{i}x+b_{i}\} and \{W^{\prime}_{i}x+b^{\prime}_{i}\} are pairwise distinct. Moreover, for each x\in\Omega_{1} , there exists an open neighborhood around x in which both Top- k selections remain fixed. • From the analysis in Appendix B.1 , there exists a set \Omega_{2}\subset\mathbb{R}^{d} , also open and dense, such that for every x\in\Omega_{2} , all expert functions \mathcal{E}_{i} and \mathcal{E}^{\prime}_{j} are affine in a neighborhood of x . Taking the intersection \Omega=\Omega_{1}\cap\Omega_{2} , we obtain a set that is still open and dense. Within \Omega , both the Top- k selections and the expert functions remain locally constant and affine, respectively. Consequently, there exists a collection of open sets \{U_{i}\}_{i\in I} that cover \Omega , such that \displaystyle\Omega=\bigcup_{i\in I}U_{i}, (125) and on each U_{i} , the expert functions \mathcal{E}_{i} and \mathcal{E}^{\prime}_{j} are affine, and the Top- k selections do not vary. Step 3. Let U be an arbitrary open set from the cover described in Equation ( 125 ). Without loss of generality, we may relabel the indices such that both Top- k maps are constantly equal to \{1,\ldots,k\} throughout U . Under this reindexing, Equation ( C.2 ) simplifies to \displaystyle\sum_{i=1}^{k}\operatorname{softmax}_{i}\left(\left\{W_{i}x+b_{i}\right\}_{i=1}^{k}\right)\cdot\mathcal{E}_{i}(x) \displaystyle\hskip 80.0pt=\sum_{i=1}^{k}\operatorname{softmax}_{i}\left(\left\{W^{\prime}_{i}x+b^{\prime}_{i}\right\}_{i=1}^{k}\right)\cdot\mathcal{E}^{\prime}_{i}(x)\quad\text{for all }x\in U. (126) According to Assumption 1, the expert functions \mathcal{E}_{i} are strongly distinct, which implies that they remain distinct on the open set U . The same conclusion applies to the functions \mathcal{E}^{\prime}_{i} . Therefore, the assumptions of Theorem B.5 are satisfied on U , and Equation ( C.2 ) falls within its scope. As a consequence, there exist constants c_{W}\in\mathbb{R}^{d} and c_{b}\in\mathbb{R} such that, up to reindexing, we have for all i=1,\ldots,k , \displaystyle W^{\prime}_{i}=W_{i}+c_{W},\quad b^{\prime}_{i}=b_{i}+c_{b}, (127) and furthermore, \mathcal{E}_{i}=\mathcal{E}^{\prime}_{i} on U . Step 4. For any index m\in\{3,4,\ldots,n\} , we invoke Proposition C.3 to select an open set V_{1} from the cover in Equation ( 125 ) such that both indices 1 and k appear in T(V_{1}) . Restricting Equation ( C.2 ) to V_{1} and applying Theorem C.4 , we conclude that there exist indices 1\leq t_{1},s_{1}\leq n^{\prime} such that \displaystyle W_{1}-W_{m}=W^{\prime}_{t_{1}}-W^{\prime}_{s_{1}}. (128) Applying the same reasoning with indices 2 and m , we obtain 1\leq t_{2},s_{2}\leq n^{\prime} such that \displaystyle W_{2}-W_{m}=W^{\prime}_{t_{2}}-W^{\prime}_{s_{2}}. (129) Subtracting Equation ( 129 ) from Equation ( 128 ) yields \displaystyle W^{\prime}_{1}-W^{\prime}_{2}=W_{1}-W_{2}=(W_{1}-W_{m})-(W_{2}-W_{m})=(W^{\prime}_{t_{1}}-W^{\prime}_{s_{1}})-(W^{\prime}_{t_{2}}-W^{\prime}_{s_{2}}). (130) Due to the linear independence guaranteed by Assumption 2, this identity can only hold if t_{1}=1 , t_{2}=2 , and s_{1}=s_{2} . Denoting this shared index by \tau(m) , i.e., \tau(m)=s_{1}=s_{2} , we then have \displaystyle W_{1}-W_{m}=W^{\prime}_{1}-W^{\prime}_{\tau(m)}, (131) which implies \displaystyle W^{\prime}_{\tau(m)}-W_{m}=W^{\prime}_{1}-W_{1}=c_{W}. (132) Similarly, we deduce \displaystyle b^{\prime}_{\tau(m)}-b_{m}=b^{\prime}_{1}-b_{1}=c_{b}. (133) Since m ranges over \{3,4,\ldots,n\} , the corresponding values of \tau(m) must be distinct. If not, suppose that \tau(m)=\tau(m^{\prime}) for some m\neq m^{\prime} , which would imply \displaystyle W_{m}-W_{m^{\prime}}=W^{\prime}_{\tau(m)}-W^{\prime}_{\tau(m^{\prime})}=0, (134) contradicting Assumption 3. Applying a symmetric argument to the parameters of \mathcal{S}(\cdot;\phi^{\prime}) , we conclude that n=n^{\prime} . Therefore, there exists a permutation \tau of \{1,\ldots,n\} such that \displaystyle W^{\prime}_{i}=W_{\tau(i)}+c_{W},\quad b^{\prime}_{i}=b_{\tau(i)}+c_{b}. (135) Finally, the analysis above shows that for any x\in\Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right) with \tau(i)\in\textup{Top-}k((W_{i}x+b_{i})_{i=1}^{n}) –i.e., when index i is selected by the Top- k operator in \mathcal{S} –we have \displaystyle\mathcal{E}_{\tau(i)}(x)=\mathcal{E}^{\prime}_{i}(x). (136) This completes the proof of Theorem C.4 . ∎ Remark C.7 . While Theorem C.4 is conceptually analogous to Theorem B.5 , it is crucial to recognize that establishing the result for SMoE involves substantially greater technical complexity. The main challenge arises from the Top- k operator, which introduces discontinuities by dynamically changing the subset of active experts in a manner that depends intricately on the input. This input-dependent behavior complicates the analysis and makes the theoretical treatment significantly more delicate. Remark C.8 (Rationale behind the assumptions in Theorem C.4 ) . We begin by restating the two assumptions made in Theorem C.4 : 1. Both \{\mathcal{E}(\cdot;\theta_{i})\}_{i=1}^{n} and \{\mathcal{E}(\cdot;\theta_{i}^{\prime})\}_{i=1}^{n^{\prime}} consist of pairwise strongly distinct functions; 2. Both \{W_{i-1}-W_{i}\}_{i=2}^{n} and \{W_{i-1}^{\prime}-W^{\prime}_{i}\}_{i=2}^{n^{\prime}} are linear independent subsets of \mathbb{R}^{d} . These assumptions are strictly stronger than those required in Theorem B.5 . We now discuss their necessity and implications in greater detail. Assumption 1. This condition arises primarily from the behavior of the Top- k operator, which induces input-dependent expert selection. As a result, the functional contribution of an expert is restricted to regions where it is actively selected by the gating mechanism. Outside these regions, the expert can behave arbitrarily without influencing the output. Therefore, if the experts are merely pairwise distinct–rather than pairwise strongly distinct–it becomes possible for different sets of expert functions to yield identical overall behavior when restricted to their respective activation domains. This ambiguity underscores the necessity of strong distinctness to ensure functional identifiability in the SMoE setting. Assumption 2. In practice, the number of experts n is typically much smaller than the input (token) dimension D . As a result, the collections \{W^{(G,i-1)}-W_{i}\}_{i=2}^{n} and \{W^{\prime(G,i-1)}-W^{\prime}_{i}\}_{i=2}^{n^{\prime}} are generically linearly independent. However, when linear dependence arises, it can prevent certain expert pairs from ever being simultaneously selected by the gating mechanism across all possible inputs. This limitation introduces singular symmetries: different parameter configurations that yield functionally identical outputs but cannot be related via the equivalence structure defined in Theorem C.4 . To illustrate this phenomenon concretely, consider an example with n=4 and k=2 , and let \mathcal{E}_{1},\mathcal{E}_{2},\mathcal{E}_{3},\mathcal{E}_{4} denote arbitrary expert functions. Define two SMoE models \mathcal{S}_{1} and \mathcal{S}_{2} , whose gating logits are (-2x,-x,x,2x) and (-3x,-2x,2x,3x) , respectively. The resulting functions take the form: \displaystyle\mathcal{S}_{1}(x)=\begin{cases}\operatorname{softmax}_{1}(-2x,-x)\cdot\mathcal{E}_{1}(x)+\operatorname{softmax}_{2}(-2x,-x)\cdot\mathcal{E}_{2}(x)&\text{if }x<0,\\ \operatorname{softmax}_{1}(x,2x)\cdot\mathcal{E}_{3}(x)+\operatorname{softmax}_{2}(x,2x)\cdot\mathcal{E}_{4}(x)&\text{if }x>0,\end{cases} (137) and \displaystyle\mathcal{S}_{2}(x)=\begin{cases}\operatorname{softmax}_{1}(-3x,-2x)\cdot\mathcal{E}_{1}(x)+\operatorname{softmax}_{2}(-3x,-2x)\cdot\mathcal{E}_{2}(x)&\text{if }x<0,\\ \operatorname{softmax}_{1}(2x,3x)\cdot\mathcal{E}_{3}(x)+\operatorname{softmax}_{2}(2x,3x)\cdot\mathcal{E}_{4}(x)&\text{if }x>0.\end{cases} (138) It is easy to verify that \mathcal{S}_{1}(x)=\mathcal{S}_{2}(x) for all x\in\mathbb{R}\setminus\{0\} , where the gating logits are pairwise distinct and the Top- k selections remain constant. However, no transformation of the form specified in Theorem C.4 maps one configuration to the other. This demonstrates how singular symmetries can arise in the SMoE architecture, even when functional outputs coincide on an open dense subset of the input domain. Remark C.9 (The case of k=1 ) . In the special case where k=1 , the SMoE function in Equation ( 21 ) simplifies to \displaystyle\mathcal{S}\left(x;\left\{W_{i},b_{i},\theta_{i}\right\}_{i=1}^{n}\right)=\mathcal{E}\left(x;\theta_{i}\right), (139) where the index i is determined by \displaystyle i=\underset{i=1,\ldots,n}{\text{argmax}}\left(W_{i}x+b_{i}\right). (140) In this setting, the Top- 1 gating mechanism selects only the expert with the highest score, and the softmax reduces to a one-hot distribution with a single nonzero entry equal to 1 . Beyond the standard G(n) symmetry acting on the expert parameters, the SMoE architecture with k=1 exhibits an additional, nontrivial invariance under the action of the multiplicative group \mathbb{R}_{>0} . Specifically, for any scalar c>0 , we have \displaystyle\mathcal{S}\left(x;\left\{W_{i},b_{i},\theta_{i}\right\}_{i=1}^{n}\right)=\mathcal{S}\left(x;\left\{cW_{i},cb_{i},\theta_{i}\right\}_{i=1}^{n}\right), (141) as the argmax used to select the active expert remains invariant under uniform positive scaling: \displaystyle\underset{i=1,\ldots,n}{\text{argmax}}\left(W_{i}x+b_{i}\right)=\underset{i=1,\ldots,n}{\text{argmax}}\left(cW_{i}x+cb_{i}\right), (142) for all x\in\Omega\left(\left\{W_{i},b_{i}\right\}_{i=1}^{n}\right) . Furthermore, since only one expert is active at any given input, the architecture does not involve explicit interactions between experts. This leads to a more complex symmetry structure, including hidden and continuous transformations, which complicates theoretical analysis. For this reason, we exclude the case k=1 from our main results and leave its investigation to future work. Appendix D Technical Details for Sections 5 and 6 D.1 Proof for the Sufficiency of Permutation Invariance in LMC of MoE The following result establishes that translation transformations do not affect the barrier loss between two parameter configurations. Proposition D.1 . Let h=(c_{W},c_{b},\textup{id}_{n})\in G(n) , where \textup{id}_{n}\in\textup{S}_{n} denotes the identity permutation. For any \phi_{A},\phi_{B}\in\Phi(n) , the barrier loss remains invariant under translation: B(\phi_{A},\phi_{B})=B(\phi_{A},h\phi_{B}). (143) That is, the translation components do not contribute to the barrier loss as defined in Equation ( 1 ). Proof. Recall that B(\phi_{A},\phi_{B})=\sup_{t\in[0,1]}\left[\mathcal{L}(t\phi_{A}+(1-t)\phi_{B})-\left(t\mathcal{L}(\phi_{A})+(1-t)\mathcal{L}(\phi_{B})\right)\right]. (144) For t\in[0,1] , denote h_{t}=(tc_{W},tc_{b},\text{id}_{n})\in G(n) . For \phi_{A},\phi_{B}\in\Phi(n) such that \displaystyle\phi_{A} \displaystyle=(W^{A}_{i},b^{A}_{i},\theta^{A}_{i})_{i=1,\ldots,n} (145) \displaystyle\phi_{B} \displaystyle=(W^{B}_{i},b^{B}_{i},\theta^{B}_{i})_{i=1,\ldots,n}, (146) we have: \displaystyle t\phi_{A}+(1-t)(h\phi_{B}) \displaystyle=~ \displaystyle t(W^{A}_{i},b^{A}_{i},\theta^{A}_{i})_{i=1,\ldots,n}+(1-t)(W^{B}_{i}+c_{W},b^{B}_{i}+c_{b},\theta^{B}_{i})_{i=1,\ldots,n} \displaystyle=~ \displaystyle\big(tW^{A}_{i}+(1-t)W^{B}_{i}+(1-t)c_{W},tb^{A}_{i}+(1-t)b^{B}_{i}+(1-t)c_{b},t\theta^{A}_{i}+(1-t)\theta^{B}_{i}\big)_{i=1,\ldots,n} \displaystyle=~ \displaystyle\big(h_{1-t}\big)\big(t\phi_{A}+(1-t)\phi_{B}\big). (147) Since the group action of G(n) on \Phi(n) preserves the functionality of the MoE function \mathcal{D} , we have \displaystyle\mathcal{D}\Big(\cdot;\phi_{B}\Big)=\mathcal{D}\Big(\cdot;h\phi_{B}\Big), (148) and \displaystyle\mathcal{D}\Big(\cdot;t\phi_{A}+(1-t)(h\phi_{B})\Big) \displaystyle\hskip 70.0pt=\mathcal{D}\Big(\cdot;\big(h_{1-t}\big)\big(t\phi_{A}+(1-t)\phi_{B}\big)\Big) \displaystyle\hskip 180.0pt=\mathcal{D}\Big(\cdot;t\phi_{A}+(1-t)\phi_{B}\Big). (149) These observations lead to \displaystyle\mathcal{L}(\phi_{B}) \displaystyle=\mathcal{L}(h\phi_{B}), (150) \displaystyle\mathcal{L}\Big(t\phi_{A}+(1-t)(h\phi_{B})\Big) \displaystyle=\mathcal{L}\Big(t\phi_{A}+(1-t)\phi_{B}\Big). (151) Thus, for t\in[0,1] , we have \displaystyle~B(\phi_{A},h\phi_{B}) \displaystyle= \displaystyle\sup_{t\in[0,1]}\Big[\mathcal{L}\big(t\phi_{A}+(1-t)(h\phi_{B})\big)-\big(t\mathcal{L}(\phi_{A})+(1-t)\mathcal{L}(h\phi_{B})\big)\Big] \displaystyle= \displaystyle\sup_{t\in[0,1]}\Big[\mathcal{L}\big(t\phi_{A}+(1-t)\phi_{B}\big)-\big(t\mathcal{L}(\phi_{A})+(1-t)\mathcal{L}(\phi_{B})\big)\Big] \displaystyle= \displaystyle~B(\phi_{A},\phi_{B}). (152) Therefore, the proof is finished. ∎ D.2 Proof of the Permutation-Invariant Property for Equation ( 11 ) Proposition D.2 . The cost function defined in Method 2 of Section 5.2 is permutation-invariant with respect to the hidden units of the experts in the MoE models. Proof. For each expert i in the first Mixture-of-Experts (MoE) model \phi , let A_{i}\in\mathbb{R}^{h\times d} and u_{i}\in\mathbb{R}^{h} be the weight matrix and bias for the first layer, and B_{i}\in\mathbb{R}^{d\times h} and v_{i}\in\mathbb{R}^{d} be the weight matrix and bias for the second layer. Define the augmented matrices \widetilde{A}_{i}=[A_{i},u_{i}]\in\mathbb{R}^{h\times(d+1)} and \widetilde{B}_{i}=[B_{i},v_{i}]\in\mathbb{R}^{d\times(h+1)} . Similarly, for each expert j in the second MoE model \phi^{\prime} , define A^{\prime}_{j} , u^{\prime}_{j} , B^{\prime}_{j} , v^{\prime}_{j} , and the augmented matrices \widetilde{A}^{\prime}_{j} , \widetilde{B}^{\prime}_{j} accordingly. We define the Gram matrices (\widetilde{A}_{i})^{\top}\widetilde{A}_{i}\in\mathbb{R}^{(d+1)\times(d+1)} and \widetilde{B}_{i}(\widetilde{B}_{i})^{\top}\in\mathbb{R}^{d\times d} for each expert i in \phi , and similarly for \phi^{\prime} . To establish permutation invariance, consider a permutation matrix P\in\mathbb{R}^{h\times h} applied to the hidden units of expert i in \phi . For the first layer, the permuted augmented weight matrix is P\widetilde{A}_{i} . For the second layer, the permutation affects the columns of B_{i} within \widetilde{B}_{i}=[B_{i},v_{i}] , so the transformed matrix is \widetilde{B}_{i}\begin{bmatrix}P^{\top}&0\\ 0&1\end{bmatrix} , leaving the bias v_{i} unchanged. We now verify the invariance of the Gram matrices under this permutation. For the first layer: (P\widetilde{A}_{i})^{\top}(P\widetilde{A}_{i})=(\widetilde{A}_{i})^{\top}P^{\top}P\widetilde{A}_{i}=(\widetilde{A}_{i})^{\top}I\widetilde{A}_{i}=(\widetilde{A}_{i})^{\top}\widetilde{A}_{i}, (153) since P^{\top}P=I , the h\times h identity matrix. For the second layer: \displaystyle\left(\widetilde{B}_{i}\begin{bmatrix}P^{\top}&0\\ 0&1\end{bmatrix}\right)\left(\widetilde{B}_{i}\begin{bmatrix}P^{\top}&0\\ 0&1\end{bmatrix}\right)^{\top} \displaystyle=\widetilde{B}_{i}\begin{bmatrix}P^{\top}&0\\ 0&1\end{bmatrix}\begin{bmatrix}P&0\\ 0&1\end{bmatrix}\widetilde{B}_{i}^{\top} \displaystyle=\widetilde{B}_{i}\begin{bmatrix}P^{\top}P&0\\ 0&1\end{bmatrix}\widetilde{B}_{i}^{\top} \displaystyle=\widetilde{B}_{i}\begin{bmatrix}I&0\\ 0&1\end{bmatrix}\widetilde{B}_{i}^{\top} \displaystyle=\widetilde{B}_{i}\widetilde{B}_{i}^{\top}, (154) where \begin{bmatrix}I&0\\ 0&1\end{bmatrix} is the (h+1)\times(h+1) identity matrix. This confirms that both ( \widetilde{A}_{i})^{\top}\widetilde{A}_{i} and \widetilde{B}_{i}(\widetilde{B}_{i})^{\top} are invariant under permutations of the hidden units. The cost matrix C\in\mathbb{R}^{n\times n} for the Linear Assignment Problem (LAP) is C=\{C_{i,j}\}_{1\leq i\leq n,1\leq j\leq n} , where C_{i,j}=\left(\left\|(\widetilde{A}_{i})^{\top}\widetilde{A}_{i}-(\widetilde{A}^{\prime}_{j})^{\top}\widetilde{A}^{\prime}_{j}\right\|_{F}^{2}+\left\|\widetilde{B}_{i}(\widetilde{B}_{i})^{\top}-\widetilde{B}^{\prime}_{j}(\widetilde{B}^{\prime}_{j})^{\top}\right\|_{F}^{2}\right)^{\frac{1}{2}}. (155) Since the Gram matrices (\widetilde{A}_{i})^{\top}\widetilde{A}_{i} and \widetilde{B}_{i}(\widetilde{B}_{i})^{\top} (and their counterparts in \phi^{\prime} ) are permutation-invariant, their differences and the Frobenius norms of these differences are also invariant. Consequently, C_{ij} remains unchanged under permutations of the hidden units in either \phi or \phi^{\prime} . Thus, the cost function is permutation-invariant with respect to the hidden units of the experts, completing the proof. ∎ D.3 Weight Matching Algorithm for Mixture-of-Experts Input: MoE model weights \phi=(W_{i},b_{i},\theta_{i})_{i=1,\ldots,n} , \phi^{\prime}=(W^{\prime}_{i},b^{\prime}_{i},\theta^{\prime}_{i})_{i=1,\ldots,n} Output: Permutation \tau for experts, and permutations \{P_{i}\}_{i=1}^{n} for hidden units % Step 1: Match experts’ order using two methods for method in {gate, expert} do Compute cost matrix C Solve LAP to obtain expert permutation \tau_{\text{method}} end for % The two candidate expert orderings \tau_{\text{gate}} and \tau_{\text{expert}} are obtained % Step 2: Align internal weights of matched expert pairs for method in {gate, expert} do for i=1 to n do Compute P_{i} by applying Weight Matching to \theta_{i} and \theta^{\prime}_{\tau_{\text{method}}(i)} end for end for return \left(\tau_{\text{gate}},\left(\{P_{i}\}_{i=1}^{n}\right)_{\text{gate}}\right) , \left(\tau_{\text{expert}},\left(\{P_{i}\}_{i=1}^{n}\right)_{\text{expert}}\right) Algorithm 1 Weight Matching for Mixture-of-Experts D.4 Formal formulation of DeepSeekMoE For completeness, we present the notation and formulations of three variants of MoE models: Dense MoE, Sparse MoE (SMoE), and DeepSeekMoE. Expert. Given d denoting the input dimension. We define an Expert as a function \mathcal{E}(\cdot;\theta)\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} , parameterized by \theta\in\mathbb{R}^{e} with e\in\mathbb{N} is the total number of trainable parameters of \mathcal{E} . In this work, we consider each expert \mathcal{E}(\cdot;\theta) to be a feedforward neural network with ReLU activation functions. Unless stated otherwise, all experts are assumed to share the same architecture. Mixture of Expert with dense gating. Given n denoting the number of experts, we define Mixture-of-Experts with dense gating as a function \mathcal{D}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} such that \displaystyle\mathcal{D}\big(x;\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n}\big)=\sum_{i=1}^{n}\text{softmax}_{i}\big(s_{1}(x),\ldots,s_{n}(x)\big)\mathcal{E}(x;\theta_{i}), (156) where s_{i}(x)=W_{i}x+b_{i} determines the contribution of each expert to the final output. Here, \theta_{i} are the parameters of the i^{\text{th}} expert. The function s=(s_{1},\ldots,s_{n}) is called the gating score , and is parameterized as s(\cdot;\{W_{i},b_{i}\}_{i=1}^{n}) with (W_{i},b_{i})\in\mathbb{R}^{d}\times\mathbb{R} are the corresponding gating parameters. Mixture-of-Experts with sparse gating. Given a positive integer k\leq n denoting the number of activated experts, define the \mathrm{Top}\text{-}k map by \text{Top-}k(z)=\{i_{1},\ldots,i_{k}\} for z=(z_{1},\ldots,z_{n})\in\mathbb{R}^{n} , where i_{1},\ldots,i_{k} are the indices corresponding to the k largest components of x . In the event of ties, we select smaller indices first. We define a Mixture-of-Experts with sparse gating (SMoE) as a function \mathcal{S}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} as follows. For x\in\mathbb{R}^{d} , let T(x)=\text{Top-}k(s(x)) , then define: \displaystyle\mathcal{S}\big(x;\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n}\big)=\sum_{i\in T(x)}\operatorname{softmax}_{i}\Big(\big(s_{i}(x)\big)_{i\in T(x)}\Big)\cdot\mathcal{E}(x;\theta_{i}) (157) In other words, the Top- k selects the k highest-scoring experts used to compute the output. DeepseekMoE with shared and routed experts. The DeepseekMoE architecture extends the sparse Mixture-of-Experts by incorporating both shared experts that are always active and routed experts that are sparsely activated based on the input. Given positive integers n_{s} and n_{r} denoting the number of shared and routed experts, respectively ( n=n_{s}+n_{r} as the total number of experts), and a positive integer k_{r}\leq n_{r} denoting the number of activated routed experts, the DeepseekMoE layer is defined as a function \mathcal{M}\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} such that \displaystyle\mathcal{M}\big(x;\{\theta_{i}^{(s)}\}_{i=1}^{n_{s}},\{W_{i},b_{i},\theta_{i}\}_{i=1}^{n_{r}}\big)=\sum_{i=1}^{n_{s}}\mathcal{E}_{i}^{(s)}(x;\theta_{i}^{(s)})+\sum_{i=1}^{n_{r}}g_{i}(x)\mathcal{E}(x;\theta_{i}^{(r)}), (158) where: • \mathcal{E}_{i}^{(s)}(\cdot;\theta_{i}^{(s)})\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} are the shared experts, parameterized by \theta_{i}^{(s)} , • \mathcal{E}_{i}^{(r)}(\cdot;\theta_{i}^{(r)})\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} are the routed experts, parameterized by \theta_{i}^{(r)} , • The gating values g_{i}(x) are computed as follows: \displaystyle\tilde{s}(x) \displaystyle=\text{softmax}\big(s_{1}(x),\ldots,s_{n_{r}}(x)\big)\in\mathbb{R}^{n_{r}}, \displaystyle g_{i}(x) \displaystyle=\begin{cases}\tilde{s}_{i}(x),&\text{if }i\in\text{Top-}k_{r}(\tilde{s}(x))\\ 0,&\text{otherwise}\end{cases} This formulation allows the model to leverage a combination of always-active shared experts and input-dependent routed experts, enhancing efficiency and performance in large-scale neural networks. Appendix E Impact of Feedforward Reinitialization on Pretrained Transformer Performance We empirically investigate the sensitivity of pretrained Transformer models to targeted parameter reinitialization within their Feedforward Network (FFN) modules. The FFN, a critical component for non-linear transformations and feature representation within each Transformer block, was selectively reinitialized layer by layer to assess its functional contribution and the robustness of the overall model architecture. This analysis aims to identify the extent to which performance is dependent on the learned parameters of individual FFNs and to potentially highlight layers that are more critical to maintaining task proficiency. To this end, experiments were conducted on two representative model-task pairs: 1. Image Classification using a Vision Transformer (ViT-Base, patch size 16, image resolution 224) pretrained on ImageNet. 2. Language Modeling using a GPT-2 model (12-layer, hidden size 768) pretrained on WikiText103. For each model, the parameters of the FFN subcomponent within a single Transformer block were reinitialized using a standard initialization scheme (e.g., variance scaling), while all other model parameters, including attention weights and the FFNs in other layers, were kept frozen from the pretrained state. Following reinitialization, the models were evaluated on their respective tasks without any subsequent fine-tuning. This approach isolates the immediate effect of disrupting a single FFN on pretrained performance. Figures 3 and 4 illustrate the performance degradation observed when reinitializing the FFN at different layers for the ViT and GPT-2 models, respectively. A striking observation is the significantly larger performance drop (increase in loss, decrease in accuracy) when reinitializing the FFN in the first layer (Layer 0) compared to any subsequent layer. While reinitializing FFNs in layers beyond the first does result in performance degradation, this impact is markedly less severe than at Layer 0. The results further reveal a non-uniform sensitivity profile among these subsequent layers, with FFN reinitialization in middle layers tending to induce a more pronounced performance decrement than in later layers. This pattern suggests a unique functional importance or sensitivity associated with the initial layer’s FFN, while the network’s architecture, particularly the presence of skip connections, appears to enable a greater degree of resilience to disruption in FFNs at deeper layers. Figure 3: Performance degradation in ViT-Base on ImageNet due to FFN reinitialization at different layers. Figure 4: Effect of FFN reinitialization on GPT-2 perplexity across layers on WikiText103. A key observation from these experiments, particularly for deeper and more complex models with skip connections, is that reinitializing a single FFN (except potentially the very first one) often does not lead to a catastrophic performance collapse. This phenomenon can be partially attributed to the presence of skip connections, which facilitate the unimpeded flow of information across layers, allowing features learned by other parts of the network to bypass the disrupted FFN. Furthermore, standard weight initialization schemes typically employ relatively small scales centered around zero, meaning the reinitialized FFN parameters introduce a perturbation that is initially small and balanced compared to the potentially large and specialized weights learned during pretraining. This combination of architectural properties (skip connections) and initialization characteristics helps the model maintain functionality, suggesting that the reinitialized state may remain within or close to the original optimization basin found during pretraining. Previous work [ 53 ] has highlighted how skip connections contribute to flatter minima and improve training stability in deep networks, which aligns with the observation that reinitialization causes less disruption than might intuitively be expected. To further investigate the generality of this observation, we also conducted FFN reinitialization experiments on tasks commonly used for evaluating model robustness and optimization landscape properties, including text classification tasks (IMDB Review [ 57 ] , AGNEWS [ 86 ] , DBPedia [ 8 ] ) and additional language modeling tasks (EnWik8 [ 38 ] , Penn Treebank (Word Level) [ 58 ] ). Table 4 presents a summary of the performance changes on these datasets after reinitializing an arbitrary single FFN layer compared to the original pretrained performance, and also shows performance after subsequent fine-tuning. Table 4: Effect of single FFN reinitialization on various tasks. "Pretrained" indicates performance of the original model. "Reinitialize" shows performance after reinitializing a single FFN layer. "Finetune" shows performance after fine-tuning the reinitialized model. "Naive Loss Barrier" refers to the loss barrier observed between two fine-tuned models initialized with different random seeds and trained with different batch orders. Dataset Pretrained Pretrained Reinitialize Reinitialize Finetune Finetune Naive Loss Loss Accuracy Loss Accuracy Loss Accuracy Barrier IMDB Review 0.3724 87.50 0.6452 68.75 0.4284 87.50 0.0019 AGNEWS 0.2849 90.50 0.6006 76.64 0.2986 90.73 0.0007 DBPedia 0.1896 93.75 0.3017 87.50 0.2036 94.47 0.0003 EnWik8 0.9684 - 1.3053 - 0.9506 - 0.0082 Penn Treebank 4.5246 - 6.9699 - 4.4639 - -0.0454 As shown in Table 4 , while reinitializing a single FFN does lead to performance degradation on these tasks (e.g., loss increases by approximately 0.1 to 0.3), the magnitude of this change is relatively modest compared to the significant loss increases observed in experiments involving more widespread reinitialization or on larger tasks like GPT-2 on WikiText103 or ViT on ImageNet (where full model reinitialization can increase loss by several units). The relatively small performance drop suggests that for these specific datasets and this focused reinitialization strategy, the model’s state remains close to its original pretrained optimum. Consequently, metrics used to assess mode connectivity between the original and reinitialized states, such as the Naive Loss Barrier (shown in Table 4 ), yield very low values. This indicates that a simple linear path in parameter space between the original and reinitialized single-FFN models exhibits low loss values. However, due to the minor performance impact of the reinitialization itself, this low barrier value may primarily reflect the fact that the reinitialized state is already highly functional and located within the same or a very accessible optimization basin, rather than necessarily implying a broadly flat landscape between drastically different high-performing modes. Therefore, while consistent with observations of flat minima facilitated by skip connections [ 53 ] , these specific reinitialization experiments on these datasets may not provide deep insights into complex mode connectivity far from the original optimum. In summary, these findings highlight the heterogeneous functional roles of FFNs across different layers in Transformer models and demonstrate varying degrees of sensitivity to parameter reinitialization. The relative robustness observed for many layers, especially in deep models, is consistent with the structural benefits of skip connections and the properties of standard initialization. This layer-wise sensitivity analysis contributes to a better understanding of Transformer architecture and has potential implications for model compression techniques (e.g., identifying less critical FFNs), efficient fine-tuning strategies, and the design of conditional computation mechanisms like Mixture-of-Experts (MoE) routing. Appendix F Experimental Details and Hyperparameters We conduct a comprehensive evaluation of Linear Mode Connectivity (LMC) across a broad suite of vision and language modeling benchmarks. For vision tasks, our study includes MNIST, CIFAR-10, CIFAR-100, ImageNet-1k, as well as transfer learning scenarios from ImageNet-21k to CIFAR-10 and CIFAR-100. For language modeling, we utilize WikiText103 and the One Billion Word dataset. All experiments are performed using pretrained Transformer-based architectures, wherein all original model parameters are frozen and only the inserted Mixture-of-Experts (MoE) layers are subject to fine-tuning. For vision tasks, we adopt Vision Transformer (ViT) backbones, while GPT-2 serves as the backbone for language modeling. Model configurations–including architecture depth, width, and tokenization context–are adjusted per dataset to reflect task-specific complexity and scale. Each expert module is architecturally aligned with the original feedforward network (MLP) layers of the pretrained model. To promote stable convergence, learning rates are independently tuned for each setting within the range of 10^{-4} to 10^{-1} . MNIST We utilize the original MNIST grayscale images, each resized into a grid of non-overlapping patches with a patch size of 7. A lightweight Vision Transformer (ViT) model is employed, configured with an embedding dimension of 32 and a depth of 1 to 2 Transformer layers, depending on the specific setup. Each self-attention layer uses 4 attention heads to process the patch embeddings. Dropout is applied at a rate of 0.0 and the activation function throughout the network is gelu . The model is optimized using stochastic gradient descent (SGD) during both the pretraining and fine-tuning stages. CIFAR-10. We utilize the original CIFAR-10 images with a patch size of 4. The ViT model is employed, with an embedding dimension of 128, 8 self-attention heads and 2 to 6 transformer layers. Dropout is applied at a rate of 0.0, and the gelu activation function is used. The SGD optimizer is employed during both pretraining and fine-tuning. CIFAR-100. We resize CIFAR-100 images to 224x224, with a patch size of 16. The ViT model is employed, with an embedding dimension of 384, 6 self-attention heads and 6 to 12 transformer layers. Dropout is applied at a rate of 0.0, and the gelu activation function is used. The SGD optimizer is employed during fine-tuning. Imagenet21k \rightarrow CIFAR10. We adopt the ViT-Small-Patch16-224 model [ 75 ] , pretrained on ImageNet-21k and subsequently fine-tuned on CIFAR-10. The model consists of 12 layers, a hidden size of 384, an MLP size of 1536, and 6 attention heads, resulting in approximately 22.2M parameters. It employs a patch size and stride of 16. Dropout is disabled (set to 0.0), and the activation function is gelu . Stochastic Gradient Descent (SGD) is employed during fine-tuning. Imagenet21k \rightarrow CIFAR100. We adopt the ViT-Small-Patch16-224 model [ 75 ] , pretrained on ImageNet-21k and subsequently fine-tuned on CIFAR-100. The model consists of 12 layers, a hidden size of 384, an MLP size of 1536, and 6 attention heads, resulting in approximately 22.2M parameters. It employs a patch size and stride of 16. The attention probability dropout and hidden layer dropout are set to 0.0, and the activation function is gelu . The SGD optimizer is employed during fine-tuning. ImageNet-1k. We use the ViT-Base-Patch16-224 model [ 20 ] , consisting of 12 Transformer layers, each with a hidden size of 768, MLP hidden dimension of 3072, and 12 self-attention heads. The total parameter count is approximately 86M. The encoder uses a patch size and stride of 16, and the gelu activation function is applied throughout. Both the attention dropout and hidden dropout rates are set to 0.0. Optimization is performed using the Adam optimizer during both pretraining and MoE-layer fine-tuning. WikiText103. We adopt a GPT-2 model architecture consisting of 12 layers with 12 attention heads and an embedding dimension of 768 . The context length and maximum position embeddings are set to 1024. The dropout rate is set to 0.1, and the activation function is gelu . The vocabulary size is 50,257. The model uses the standard GPT-2 initialization and layer normalization settings. The Adam optimizer is employed during both pretraining and fine-tuning. One Billion Word. Similar to WikiText103, the GPT-2 model consists of 12 layers, 12 attention heads, and an embedding dimension of 768 , but with a reduced context and position length of 256. The dropout rate and activation function remain consistent with the WikiText103 setup. The vocabulary size is extended to 793,470 to accommodate the dataset’s linguistic diversity. The Adam optimizer is employed during both pretraining and fine-tuning. All experiments are executed on a single NVIDIA H100 GPU with 80GB of memory, except for the One Billion Word task, which utilizes two H100 GPUs. Due to the use of the JAX framework, approximately 75% of GPU memory (around 60GB) is pre-allocated by default. The number of CPU workers used in data loading and preprocessing is kept less than or equal to 10 across all experiments. For smaller-scale tasks such as MNIST, CIFAR-10, CIFAR-100, and transfer learning from ImageNet-21k, each experiment completes in under 30 minutes. For larger-scale tasks, the fine-tuning durations are approximately 15 hours for ImageNet-1k, 3.5 hours for WikiText103, and 4 hours for One Billion Word. Appendix G Experimental Results G.1 Verification of Linear Mode Connectivity across diverse configurations To rigorously evaluate the robustness and generality of Linear Mode Connectivity (LMC) in expert-based Transformer architectures, we conducted systematic experiments across a wide range of Mixture-of-Experts (MoE) configurations. Our study includes three representative variants–vanilla MoE, SMoE, and DeepSeekMoE–which differ in both architectural structure and expert routing mechanisms. In all cases, we substituted the feed-forward network (FFN) of the first Transformer layer with an MoE module, while leaving the rest of the model architecture unchanged. All models were fine-tuned from identical random initializations to ensure consistency in comparison. Table 5 summarizes the experimental conditions, covering diverse model depths (2, 6, 12 layers), task domains (including vision benchmarks such as MNIST, CIFAR-10, ImageNet21k \rightarrow CIFAR-100, and language modeling with WikiText103), and expert configurations. SMoE employs a sparsity level of k=2 active experts per token, whereas DeepSeekMoE further incorporates a shared expert ( s=1 ) to encourage parameter reuse and stability. Across nearly all configurations, LMC consistently emerges: linear interpolation between independently fine-tuned models (initialized identically) yields smooth loss trajectories, with no significant barriers. This suggests that expert-based models, even with dynamic routing and varying capacities, tend to converge to connected optima under matched training setups. The corresponding loss curves, linked in Table 5 , reinforce this observation. These results provide strong empirical evidence that LMC is a robust and general phenomenon in expert architectures, supporting the hypothesis that expert specialization and routing do not disrupt the connected geometry of the optimization landscape when alignment in initialization and architecture is preserved. Table 5: Experimental configurations for LMC analysis across MoE, SMoE, and DeepSeekMoE variants, evaluated on multiple datasets and settings. Datasets of the form A\rightarrow B denote pretraining on A and finetuning on B . SMoE uses k=2 active experts, while DeepSeekMoE uses k=2 with an additional shared expert ( s=1 ). In all cases, the MLP in the first Transformer layer is replaced with an MoE module. Method Dataset No. layers No. experts Figure MoE MNIST 1 [2, 4] [ 5 , 6 ] 2 [2, 4] [ 7 , 8 ] CIFAR-10 2 [2, 4, 6] [ 9 , 10 , 11 ] 6 [2, 4, 6] [ 12 , 13 , 14 ] CIFAR-100 6 [2, 4, 6] [ 15 , 16 , 17 ] ImageNet-21k \rightarrow CIFAR-10 12 [2, 4, 6] [ 18 , 19 , 20 ] ImageNet-21k \rightarrow CIFAR-100 12 [2, 4, 6] [ 21 , 22 , 23 ] ImageNet-1k 12 [2, 4, 6, 8] [ 24 , 25 , 26 , 27 ] WikiText103 12 [2, 4, 6, 8] [ 28 , 29 , 30 , 31 ] One Billion Word 12 [2, 4, 6, 8] [ 32 , 33 , 34 , 35 ] SMoE ( k=2 ) MNIST 1 [4] [ 36 ] 2 [4] [ 37 ] CIFAR-10 2 [4, 8] [ 38 , 39 ] 6 [4, 8] [ 40 , 41 ] CIFAR-100 6 [4, 8] [ 42 , 43 ] ImageNet-21k \rightarrow CIFAR-10 12 [4, 8] [ 44 , 45 ] ImageNet-21k \rightarrow CIFAR-100 12 [4, 8] [ 46 , 47 ] ImageNet-1k 12 [4, 8, 16] [ 48 , 49 , 50 ] WikiText103 12 [4, 8, 16] [ 51 , 52 , 53 ] One Billion Word 12 [4, 8, 16] [ 54 , 55 , 56 ] DeepSeekMoE MNIST 1 [4] [ 57 ] ( k=2,s=1 ) 2 [4] [ 58 ] CIFAR-10 2 [4, 8] [ 59 , 60 ] 6 [4, 8] [ 61 , 62 ] CIFAR-100 6 [4, 8] [ 63 , 64 ] ImageNet-21k \rightarrow CIFAR-10 12 [4, 8] [ 65 , 66 ] ImageNet-21k \rightarrow CIFAR-100 12 [4, 8] [ 67 , 68 ] ImageNet-1k 12 [4, 8, 16] [ 69 , 70 , 71 ] WikiText103 12 [4, 8, 16] [ 72 , 73 , 74 ] One Billion Word 12 [4, 8, 16] [ 75 , 76 , 77 ] G.1.1 Dense Mixture-of-Experts Figure 5: Linear Mode Connectivity for ViT-MoE on MNIST with 1 layer and 2 experts Figure 6: Linear Mode Connectivity for ViT-MoE on MNIST with 1 layer and 4 experts Figure 7: Linear Mode Connectivity for ViT-MoE on MNIST with 2 layers and 2 experts Figure 8: Linear Mode Connectivity for ViT-MoE on MNIST with 2 layers and 4 experts Figure 9: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 2 layers and 2 experts Figure 10: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 2 layers and 4 experts Figure 11: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 2 layers and 6 experts Figure 12: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 2 experts Figure 13: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 4 experts Figure 14: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 6 experts Figure 15: Linear Mode Connectivity for ViT-Moe on CIFAR-100 with 6 layers and 2 experts Figure 16: Linear Mode Connectivity for ViT-Moe on CIFAR-100 with 6 layers and 4 experts Figure 17: Linear Mode Connectivity for ViT-Moe on CIFAR-100 with 6 layers and 6 experts Figure 18: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 2 experts Figure 19: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts Figure 20: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 6 experts Figure 21: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 2 experts Figure 22: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts Figure 23: Linear Mode Connectivity for ViT-MoE with ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 6 experts Figure 24: Linear Mode Connectivity for ViT-MoE on ImageNet-1k with 12 layers and 2 experts Figure 25: Linear Mode Connectivity for ViT-MoE on ImageNet-1k with 12 layers and 4 experts Figure 26: Linear Mode Connectivity for ViT-MoE on ImageNet-1k with 12 layers and 6 experts Figure 27: Linear Mode Connectivity for ViT-MoE on ImageNet-1k with 12 layers and 8 experts Figure 28: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 2 experts Figure 29: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 4 experts Figure 30: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 6 experts Figure 31: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 8 experts Figure 32: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and 2 experts Figure 33: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and 4 experts Figure 34: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and 6 experts Figure 35: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and 8 experts G.1.2 Sparse Mixture-of-Experts Figure 36: Linear Mode Connectivity for ViT-SMoE (k=2) on MNIST with 1 layer and 4 experts Figure 37: Linear Mode Connectivity for ViT-SMoE (k=2) on MNIST with 2 layers and 4 experts Figure 38: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-10 with 2 layers and 4 experts Figure 39: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-10 with 2 layers and 8 experts Figure 40: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-10 with 6 layers and 4 experts Figure 41: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-10 with 6 layers and 8 experts Figure 42: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-100 with 6 layers and 4 experts Figure 43: Linear Mode Connectivity for ViT-SMoE (k=2) on CIFAR-100 with 6 layers and 8 experts Figure 44: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts Figure 45: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 8 experts Figure 46: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts Figure 47: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 8 experts Figure 48: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-1k with 12 layers and 4 experts Figure 49: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-1k with 12 layers and 8 experts Figure 50: Linear Mode Connectivity for ViT-SMoE (k=2) on ImageNet-1k with 12 layers and 16 experts Figure 51: Linear Mode Connectivity for GPT2-SMoE (k=2) on Wikitext103 with 12 layers and 4 experts Figure 52: Linear Mode Connectivity for GPT2-SMoE (k=2) on Wikitext103 with 12 layers and 8 experts Figure 53: Linear Mode Connectivity for GPT2-SMoE (k=2) on Wikitext103 with 12 layers 16 experts Figure 54: Linear Mode Connectivity for GPT2-SMoE (k=2) on One Billion Word with 12 layers and 4 experts Figure 55: Linear Mode Connectivity for GPT2-SMoE (k=2) on One Billion Word with 12 layers and 8 experts Figure 56: Linear Mode Connectivity for GPT2-SMoE (k=2) on One Billion Word with 12 layers and 16 experts G.1.3 DeepSeek Mixture-of-Experts Figure 57: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on MNIST with 1 layer and 4 experts Figure 58: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on MNIST with 2 layers and 4 experts Figure 59: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 2 layers and 4 experts Figure 60: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 2 layers and 8 experts Figure 61: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 6 layers and 4 experts Figure 62: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 6 layers and 8 experts Figure 63: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-100 with 6 layers and 4 experts Figure 64: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-100 with 6 layers and 8 experts Figure 65: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts Figure 66: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 8 experts Figure 67: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts Figure 68: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 8 experts Figure 69: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-1k with 12 layers and 4 experts Figure 70: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-1k with 12 layers and 8 experts Figure 71: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-1k with 12 layers and 16 experts Figure 72: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on Wikitext103 with 12 layers and 4 experts Figure 73: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on Wikitext103 with 12 layers and 8 experts Figure 74: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on Wikitext103 with 12 layers and 16 experts Figure 75: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on One Billion Word with 12 layers and 4 experts Figure 76: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on One Billion Word with 12 layers and 8 experts Figure 77: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on One Billion Word with 12 layers and 16 experts G.2 Linear Mode Connectivity Analysis: Last Layer We investigate Linear Mode Connectivity (LMC) in settings where the feed-forward network (FFN) of the last Transformer layer is replaced with a Mixture-of-Experts (MoE) module. This extends our earlier analysis of first -layer substitutions by examining connectivity at the terminal depth of the architecture, offering a complementary view on model plasticity and the modular structure of expert-based designs. Following the procedure outlined in Section 6.1 , all pretrained weights were frozen, and only the MoE module in the last layer was fine-tuned. For each configuration, three independent fine-tuning runs were conducted with different random seeds. LMC was assessed by evaluating the loss along linear interpolation paths between pairs of fine-tuned models, providing insight into the connectivity and overlap of their respective solution basins. Table 6 summarizes the experimental settings for last -layer FFN replacement across dense MoE, SMoE, and DeepSeekMoE variants. Unless otherwise stated, all figure references correspond to last -layer experiments. Table 6: In all configurations, the MLP component of the last Transformer layer is replaced by an MoE component. Method Dataset Number of layers Number of experts Figure MoE CIFAR-10 6 [2, 4] [ 78 , 79 ] ImageNet-21k \rightarrow CIFAR-10 12 [2, 4] [ 80 , 81 ] ImageNet-21k \rightarrow CIFAR-100 12 [2, 4] [ 82 , 83 ] SMoE ( k=2 ) CIFAR-10 6 [4, 8] [ 84 , 85 ] ImageNet-21k \rightarrow CIFAR-10 12 [4, 8] [ 86 , 87 ] ImageNet-21k \rightarrow CIFAR-100 12 [4, 8] [ 88 , 89 ] DeepSeekMoE CIFAR-10 6 [4, 8] [ 90 , 91 ] ( k=2,s=1 ) ImageNet-21k \rightarrow CIFAR-10 12 [4, 8] [ 92 , 93 ] ImageNet-21k \rightarrow CIFAR-100 12 [4, 8] [ 94 , 95 ] Figure 78: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 2 experts. Figure 79: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 4 experts. Figure 80: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 2 experts. Figure 81: Linear Mode Connectivity for ViT-MoE on on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts. Figure 82: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 2 experts. Figure 83: Linear Mode Connectivity for ViT-MoE on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts. Figure 84: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on CIFAR-10 with 6 layers and 4 experts. Figure 85: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on CIFAR-10 with 6 layers and 8 experts. Figure 86: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts. Figure 87: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 8 experts. Figure 88: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts. Figure 89: Linear Mode Connectivity for ViT-SMoE ( k=2 ) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 8 experts. Figure 90: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 6 layers and 4 experts. Figure 91: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on CIFAR-10 with 6 layers and 8 experts. Figure 92: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 4 experts. Figure 93: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-10 with 12 layers and 8 experts. Figure 94: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 4 experts. Figure 95: Linear Mode Connectivity for ViT-DeepSeekMoE (k=2,s=1) on ImageNet-21k \rightarrow CIFAR-100 with 12 layers and 8 experts. G.3 Expert Matching Method Table 7: Comparison of expert matching methods (Expert Weight Matching and Gate Weight Matching) across three ViT-MoE model variants evaluated on the CIFAR-10 and CIFAR-100 test sets, measuring loss . Metrics reported include rank and \hat{L} , defined in Section 6.3 , computed across 24 permutations for 10 checkpoint pairs. All models consist of 12 layers and 4 experts. Method Dataset Layer replaced Expert Weight Matching Gate Weight Matching Rank \downarrow \hat{L}\downarrow Rank \hat{L}\downarrow MoE CIFAR-10 1 2.50 \pm 1.50 2.12 \pm 0.42 3.00 \pm 1.00 3.42 \pm 0.55 4 2.10 \pm 0.50 1.04 \pm 0.32 2.60 \pm 0.92 1.54 \pm 0.44 8 2.70 \pm 0.46 0.60 \pm 0.27 2.90 \pm 0.30 0.74 \pm 0.17 12 4.60 \pm 2.00 0.13 \pm 0.05 3.80 \pm 1.66 0.09 \pm 0.03 CIFAR-100 1 2.80 \pm 0.40 3.17 \pm 0.25 2.90 \pm 0.70 2.73 \pm 1.03 4 3.60 \pm 1.20 1.15 \pm 0.55 2.70 \pm 1.00 2.03 \pm 0.93 8 3.30 \pm 0.78 0.67 \pm 0.14 3.20 \pm 0.87 1.13 \pm 0.55 12 3.40 \pm 0.92 0.07 \pm 0.03 4.20 \pm 0.89 0.11 \pm 0.04 SMoE ( k=2 ) CIFAR-10 1 3.00 \pm 1.00 3.80 \pm 1.18 3.20 \pm 0.98 3.46 \pm 2.94 4 2.80 \pm 0.98 1.73 \pm 0.49 2.40 \pm 1.56 1.64 \pm 0.86 8 2.60 \pm 0.49 0.40 \pm 0.46 2.60 \pm 0.43 0.36 \pm 0.83 12 4.00 \pm 2.45 0.06 \pm 0.04 2.80 \pm 1.66 0.22 \pm 0.19 CIFAR-100 1 2.80 \pm 0.40 3.29 \pm 3.18 2.10 \pm 0.70 2.00 \pm 2.02 4 2.60 \pm 1.20 1.20 \pm 0.43 3.20 \pm 1.47 1.91 \pm 1.05 8 3.10 \pm 0.83 0.13 \pm 0.07 2.70 \pm 0.90 0.11 \pm 0.09 12 2.40 \pm 0.92 0.07 \pm 0.13 2.00 \pm 0.89 0.03 \pm 0.12 DeepSeekMoE CIFAR-10 1 3.10 \pm 2.12 5.06 \pm 1.22 3.60 \pm 0.83 4.28 \pm 2.92 (k=2,s=1) 4 2.60 \pm 0.77 3.32 \pm 0.38 3.10 \pm 1.52 2.80 \pm 0.88 8 2.30 \pm 0.51 0.39 \pm 0.48 2.40 \pm 0.37 0.78 \pm 1.00 12 3.60 \pm 3.55 0.08 \pm 0.04 3.60 \pm 1.48 0.23 \pm 0.17 CIFAR-100 1 4.20 \pm 0.60 4.45 \pm 3.40 3.70 \pm 0.45 4.21 \pm 2.60 4 2.50 \pm 0.78 1.34 \pm 0.46 3.10 \pm 1.91 2.98 \pm 1.46 8 1.90 \pm 0.60 0.13 \pm 0.08 3.10 \pm 1.02 0.13 \pm 0.16 12 3.30 \pm 1.26 0.04 \pm 0.11 2.70 \pm 0.89 0.02 \pm 0.20 Table 8: Comparison of expert matching methods (Expert Weight Matching and Gate Weight Matching) across three ViT-MoE model variants evaluated on the CIFAR-10 and CIFAR-100 test sets, measuring accuracy . Metrics reported include rank and \hat{L} , defined in Section 6.3 , computed across 24 permutations for 10 checkpoint pairs. All models consist of 12 layers and 4 experts. Method Dataset Layer replaced Expert Weight Matching Gate Weight Matching Rank \downarrow \hat{L}\downarrow Rank \hat{L}\downarrow MoE CIFAR-10 1 4.90 \pm 2.59 1.80 \pm 0.78 4.40 \pm 1.80 2.00 \pm 0.44 4 3.20 \pm 0.40 0.95 \pm 0.97 2.90 \pm 0.70 1.64 \pm 0.98 8 3.00 \pm 1.80 0.03 \pm 0.05 3.80 \pm 1.83 0.05 \pm 0.05 12 4.00 \pm 1.45 0.02 \pm 0.03 2.80 \pm 1.40 0.04 \pm 0.06 CIFAR-100 1 3.20 \pm 1.47 2.66 \pm 1.03 4.20 \pm 1.00 1.51 \pm 1.16 4 2.70 \pm 0.78 1.87 \pm 1.17 2.90 \pm 0.83 1.39 \pm 1.06 8 4.70 \pm 1.42 0.08 \pm 0.02 3.80 \pm 1.47 0.05 \pm 0.03 12 2.40 \pm 0.92 0.05 \pm 0.02 3.20 \pm 0.87 0.02 \pm 0.02 SMoE ( k=2 ) CIFAR-10 1 2.30 \pm 2.06 3.09 \pm 1.21 3.20 \pm 2.75 2.24 \pm 0.90 4 2.90 \pm 0.59 1.16 \pm 1.00 2.80 \pm 0.83 1.75 \pm 1.00 8 3.40 \pm 1.47 0.07 \pm 0.03 3.00 \pm 1.74 0.04 \pm 0.02 12 3.60 \pm 1.21 0.12 \pm 0.08 4.70 \pm 2.05 0.07 \pm 0.03 CIFAR-100 1 2.10 \pm 1.85 3.52 \pm 2.26 2.40 \pm 0.93 2.05 \pm 2.95 4 2.00 \pm 0.67 1.81 \pm 1.28 2.90 \pm 0.73 1.02 \pm 2.41 8 4.30 \pm 1.60 0.09 \pm 0.03 3.10 \pm 1.02 0.11 \pm 0.08 12 2.60 \pm 1.03 0.11 \pm 0.08 2.70 \pm 0.89 0.03 \pm 0.06 DeepSeekMoE CIFAR-10 1 2.80 \pm 2.20 2.95 \pm 1.15 3.30 \pm 2.40 2.60 \pm 0.85 (k=2,s=1) 4 3.10 \pm 0.80 1.40 \pm 1.13 2.90 \pm 0.90 1.60 \pm 1.30 8 3.60 \pm 1.32 0.09 \pm 0.04 3.40 \pm 1.10 0.07 \pm 0.03 12 3.20 \pm 1.00 0.06 \pm 0.02 3.00 \pm 0.87 0.05 \pm 0.02 CIFAR-100 1 2.54 \pm 1.90 3.20 \pm 2.00 2.80 \pm 2.00 2.90 \pm 1.80 4 2.90 \pm 0.70 1.60 \pm 1.20 3.10 \pm 0.80 1.40 \pm 1.00 8 3.80 \pm 1.43 0.10 \pm 0.03 3.50 \pm 1.20 0.08 \pm 0.04 12 2.70 \pm 0.90 0.07 \pm 0.02 2.90 \pm 0.70 0.06 \pm 0.03 Figure 96: Accuracy curves for 12-layer ViT-MoE models with a 4-expert MoE replacement in either the first layer (subplots 1, 3) or the last layer (subplots 2, 4), on CIFAR-10 and CIFAR-100. The curves compare two Expert Order Matching methods across 24 permutations, with Weight Matching applied post-reordering to all permutations. Corresponding loss metrics are presented in Figure 2 G.4 Ablation Study on Number of Layers Table 9: Ablation Study on varying the number of Transformer layers and all MoE variants Integration in ViT for CIFAR-10. The ratios of metrics (loss barrier, loss AUC, accuracy barrier, and accuracy AUC) compare Algorithm 1 to naive interpolation. For full table results (non-raio), kindly refer to Tables 10 and 11 Method Number of layers Layer replaced Loss barrier ratio (%) \downarrow Loss AUC ratio (%) \downarrow Acc. barrier ratio (%) \downarrow Acc. AUC ratio (%) \downarrow MoE 2 1 8.54 \pm 1.53 8.73 \pm 1.68 8.39 \pm 1.42 8.01 \pm 1.39 2 9.33 \pm 2.04 8.94 \pm 1.92 9.10 \pm 2.19 8.83 \pm 2.22 4 1 8.29 \pm 1.63 8.08 \pm 1.59 7.41 \pm 1.45 6.43 \pm 1.09 2 10.19 \pm 3.43 10.19 \pm 3.16 10.21 \pm 5.26 9.08 \pm 5.33 3 8.50 \pm 1.82 7.83 \pm 1.89 9.82 \pm 2.58 8.70 \pm 2.58 4 5.12 \pm 0.67 4.60 \pm 0.65 4.17 \pm 1.68 4.58 \pm 1.01 6 1 4.16 \pm 1.47 4.96 \pm 2.31 3.64 \pm 1.00 4.13 \pm 1.55 2 6.78 \pm 4.80 7.71 \pm 3.09 7.27 \pm 5.73 8.98 \pm 2.94 3 9.69 \pm 4.92 8.62 \pm 5.46 9.03 \pm 3.31 8.14 \pm 3.57 4 10.83 \pm 5.61 12.01 \pm 6.42 8.92 \pm 2.21 10.00 \pm 7.30 5 9.56 \pm 4.52 11.72 \pm 5.59 8.72 \pm 2.89 11.01 \pm 3.83 6 6.90 \pm 3.54 9.67 \pm 6.59 7.16 \pm 3.98 9.69 \pm 3.11 SMoE ( k=2 ) 2 1 8.93 \pm 0.92 9.34 \pm 1.23 9.36 \pm 1.33 8.84 \pm 1.40 2 7.55 \pm 1.39 6.62 \pm 1.35 9.10 \pm 2.88 7.29 \pm 2.78 4 1 8.96 \pm 1.54 8.83 \pm 1.64 8.66 \pm 2.07 8.21 \pm 1.91 2 9.62 \pm 2.68 10.07 \pm 3.50 9.41 \pm 3.04 9.67 \pm 3.52 3 13.31 \pm 1.49 12.44 \pm 2.06 11.37 \pm 3.01 13.63 \pm 3.17 4 9.17 \pm 1.40 9.36 \pm 1.48 10.97 \pm 2.77 12.52 \pm 2.57 6 1 2.37 \pm 1.92 2.65 \pm 1.34 2.25 \pm 2.71 4.67 \pm 2.38 2 10.09 \pm 3.11 9.26 \pm 3.19 8.35 \pm 3.37 8.36 \pm 2.38 3 11.22 \pm 6.25 10.17 \pm 8.52 12.10 \pm 7.04 12.80 \pm 10.77 4 8.76 \pm 2.06 12.16 \pm 2.22 13.09 \pm 5.49 10.85 \pm 7.21 5 14.39 \pm 7.82 14.66 \pm 9.10 11.82 \pm 9.19 12.45 \pm 7.85 6 6.38 \pm 5.01 9.56 \pm 3.79 11.96 \pm 11.93 10.93 \pm 15.06 DeepSeekMoE 2 1 8.94 \pm 1.95 9.12 \pm 2.10 7.96 \pm 1.59 7.88 \pm 1.90 ( k=2,r=1 ) 2 9.25 \pm 0.37 8.27 \pm 0.44 12.00 \pm 1.35 11.14 \pm 1.73 4 1 8.65 \pm 1.01 8.56 \pm 1.19 7.17 \pm 0.99 6.46 \pm 0.88 2 10.80 \pm 3.77 10.88 \pm 3.69 10.45 \pm 4.91 9.92 \pm 4.83 3 7.90 \pm 1.79 8.82 \pm 1.87 10.21 \pm 2.44 9.02 \pm 2.52 4 8.35 \pm 1.20 8.29 \pm 1.39 9.29 \pm 2.75 7.04 \pm 3.14 6 1 6.29 \pm 5.00 6.17 \pm 3.48 8.07 \pm 5.34 8.35 \pm 5.65 2 8.03 \pm 1.79 9.16 \pm 2.10 7.88 \pm 3.66 8.54 \pm 3.93 3 9.27 \pm 4.25 9.29 \pm 3.19 12.93 \pm 2.18 11.56 \pm 5.58 4 10.80 \pm 2.18 12.72 \pm 1.52 7.34 \pm 3.64 8.99 \pm 5.10 5 13.77 \pm 2.27 13.03 \pm 7.01 11.14 \pm 5.47 13.28 \pm 3.32 6 9.82 \pm 2.99 8.38 \pm 4.81 8.92 \pm 4.22 8.14 \pm 3.48 Table 10: Ablation study results on CIFAR-10 for LMC in ViT with layer replacements of all MoE variants, reporting test set loss . We evaluate the loss metrics for linear interpolation using our proposed Algorithm 1 against naive linear interpolation. Metrics include the loss barrier and loss Area Under the Curve (AUC), with AUC computed relative to the straight line connecting the two model endpoints. Each experiment is repeated five times, and LMC is performed on ten model pairs. Additionally, we report loss values across five models to provide enhanced context. To improve readability, all metrics are scaled up by a factor of 10^{2} . Method Number of layers Layer replaced WM loss barrier \downarrow Naive loss barrier \downarrow WM loss AUC \downarrow Naive loss AUC \downarrow Loss value \downarrow MoE 2 1 3.14 \pm 0.55 34.22 \pm 1.25 1.45 \pm 0.28 16.11 \pm 0.74 99.04 \pm 0.92 2 2.84 \pm 0.44 33.54 \pm 1.12 1.38 \pm 0.15 17.53 \pm 0.57 104.52 \pm 0.81 4 1 9.48 \pm 1.36 115.56 \pm 10.19 4.25 \pm 0.68 53.07 \pm 4.54 97.00 \pm 0.59 2 6.59 \pm 2.12 64.73 \pm 2.05 3.71 \pm 1.11 36.39 \pm 0.96 91.74 \pm 0.87 3 4.20 \pm 0.90 48.23 \pm 0.92 2.21 \pm 0.53 28.23 \pm 0.50 88.06 \pm 1.30 4 2.76 \pm 0.38 53.92 \pm 0.80 1.42 \pm 0.22 31.01 \pm 0.54 86.15 \pm 0.33 6 1 1.42 \pm 0.44 37.47 \pm 8.47 0.74 \pm 0.30 18.73 \pm 3.30 90.35 \pm 1.15 2 0.47 \pm 0.32 7.09 \pm 0.60 0.27 \pm 0.10 3.53 \pm 0.30 88.73 \pm 0.39 3 0.33 \pm 0.17 3.42 \pm 0.22 0.13 \pm 0.08 1.60 \pm 0.14 88.26 \pm 0.50 4 0.14 \pm 0.08 1.36 \pm 0.33 0.07 \pm 0.04 0.64 \pm 0.16 86.48 \pm 0.17 5 0.14 \pm 0.06 1.68 \pm 0.44 0.08 \pm 0.03 0.83 \pm 0.21 85.92 \pm 0.46 6 0.23 \pm 0.11 2.46 \pm 0.29 0.12 \pm 0.04 1.26 \pm 0.21 88.61 \pm 0.33 SMoE 2 1 2.95 \pm 0.24 33.17 \pm 1.01 1.39 \pm 0.16 14.95 \pm 0.51 98.88 \pm 0.86 (k=2) 2 2.52 \pm 0.46 33.46 \pm 0.67 1.20 \pm 0.25 18.12 \pm 0.48 105.59 \pm 0.37 4 1 10.45 \pm 1.99 116.54 \pm 8.34 4.75 \pm 1.01 53.65 \pm 3.97 96.73 \pm 0.85 2 8.31 \pm 0.83 86.88 \pm 3.25 4.73 \pm 0.57 47.40 \pm 1.91 90.80 \pm 1.02 3 4.73 \pm 0.62 35.49 \pm 1.34 2.59 \pm 0.49 20.78 \pm 0.80 86.29 \pm 0.24 = 4 4.48 \pm 0.70 48.84 \pm 0.68 2.68 \pm 0.44 28.62 \pm 0.44 82.33 \pm 0.37 6 1 2.12 \pm 0.67 34.23 \pm 7.22 1.28 \pm 0.35 12.59 \pm 2.67 90.25 \pm 1.00 2 0.75 \pm 0.29 7.28 \pm 0.83 0.34 \pm 0.15 3.59 \pm 0.43 88.99 \pm 0.65 3 0.34 \pm 0.19 3.28 \pm 0.30 0.15 \pm 0.06 1.76 \pm 0.13 87.69 \pm 0.52 4 0.26 \pm 0.14 2.78 \pm 0.38 0.14 \pm 0.07 1.35 \pm 0.19 86.08 \pm 0.40 5 0.22 \pm 0.11 2.50 \pm 0.04 0.11 \pm 0.05 1.25 \pm 0.02 86.08 \pm 0.34 6 0.23 \pm 0.18 3.63 \pm 0.17 0.13 \pm 0.08 1.93 \pm 0.10 88.38 \pm 0.36 DeepSeekMoE 2 1 3.00 \pm 0.61 33.63 \pm 1.30 1.37 \pm 0.27 15.14 \pm 0.73 98.41 \pm 1.06 (k=2,s=1) 2 3.07 \pm 0.21 33.16 \pm 1.26 1.50 \pm 0.11 18.18 \pm 0.72 105.92 \pm 0.53 4 1 10.57 \pm 1.57 122.13 \pm 10.50 4.91 \pm 0.84 57.32 \pm 4.89 97.71 \pm 1.26 2 5.85 \pm 2.14 54.13 \pm 2.27 2.90 \pm 1.26 30.17 \pm 1.35 90.89 \pm 0.52 3 2.67 \pm 0.65 33.36 \pm 1.47 1.35 \pm 0.40 19.48 \pm 0.89 86.37 \pm 0.43 4 4.11 \pm 0.61 49.18 \pm 0.95 2.37 \pm 0.41 28.49 \pm 0.55 82.26 \pm 0.39 6 1 2.25 \pm 0.82 39.42 \pm 6.23 1.14 \pm 0.38 18.88 \pm 3.32 91.32 \pm 0.25 2 2.88 \pm 0.57 28.53 \pm 1.26 1.56 \pm 0.32 15.97 \pm 0.69 89.50 \pm 1.05 3 0.56 \pm 0.12 7.90 \pm 0.18 0.26 \pm 0.06 3.90 \pm 0.12 88.22 \pm 0.57 4 0.24 \pm 0.16 3.07 \pm 0.34 0.12 \pm 0.03 1.53 \pm 0.14 86.09 \pm 0.40 5 0.32 \pm 0.17 3.39 \pm 0.31 0.16 \pm 0.08 1.63 \pm 0.13 86.10 \pm 0.51 6 0.23 \pm 0.08 2.61 \pm 0.53 0.14 \pm 0.04 1.43 \pm 0.32 89.27 \pm 0.57 Table 11: Ablation study results on CIFAR-10 for LMC in ViT with layer replacements of all MoE variants, reporting test set accuracy . We evaluate the accuracy metrics for linear interpolation using our proposed Algorithm 1 against naive linear interpolation. Metrics include the accuracy barrier and accuracy Area Under the Curve (AUC), with AUC computed relative to the straight line connecting the two model endpoints. Each experiment is repeated five times, and LMC is performed on ten model pairs. Additionally, we report accuracy values across five models to provide enhanced context. To improve readability, all metrics are scaled up by a factor of 10^{2} . Method Number of layers Layer replaced WM acc. barrier \downarrow Naive acc. barrier \downarrow WM acc. AUC \downarrow Naive acc. barrier \downarrow Acc. value \uparrow MoE 2 1 0.91 \pm 0.28 12.33 \pm 1.28 0.44 \pm 0.15 5.89 \pm 0.54 66.53 \pm 3.30 2 0.76 \pm 0.21 7.85 \pm 2.31 0.33 \pm 0.08 3.77 \pm 1.11 63.65 \pm 0.47 4 1 2.10 \pm 0.42 28.36 \pm 1.85 0.86 \pm 0.16 13.43 \pm 0.99 72.48 \pm 0.68 2 1.41 \pm 0.71 13.83 \pm 0.53 0.75 \pm 0.41 7.88 \pm 0.29 73.43 \pm 0.61 3 0.67 \pm 0.17 6.84 \pm 0.24 0.35 \pm 0.09 4.00 \pm 0.12 73.64 \pm 0.20 4 0.23 \pm 0.09 5.48 \pm 0.19 0.14 \pm 0.03 3.05 \pm 0.14 73.58 \pm 0.19 6 1 0.50 \pm 0.13 13.95 \pm 2.20 0.24 \pm 0.07 5.71 \pm 0.93 74.29 \pm 0.35 2 0.35 \pm 0.10 2.49 \pm 0.36 0.12 \pm 0.04 1.23 \pm 0.21 74.65 \pm 0.44 3 0.14 \pm 0.05 1.48 \pm 0.28 0.05 \pm 0.02 0.72 \pm 0.14 74.72 \pm 0.46 4 0.05 \pm 0.08 1.33 \pm 0.43 0.03 \pm 0.05 0.68 \pm 0.24 74.46 \pm 0.12 5 0.05 \pm 0.03 0.59 \pm 0.05 0.02 \pm 0.01 0.28 \pm 0.02 74.62 \pm 0.08 6 0.06 \pm 0.04 0.61 \pm 0.12 0.03 \pm 0.02 0.30 \pm 0.05 74.25 \pm 0.21 SMoE 2 1 1.07 \pm 0.15 11.42 \pm 0.22 0.46 \pm 0.07 5.20 \pm 0.09 65.27 \pm 0.69 (k=2) 2 0.58 \pm 0.17 6.41 \pm 0.24 0.26 \pm 0.10 3.63 \pm 0.13 62.58 \pm 0.16 4 1 2.48 \pm 0.66 28.55 \pm 1.55 1.12 \pm 0.29 13.52 \pm 0.81 72.26 \pm 0.60 2 1.78 \pm 0.23 19.00 \pm 0.71 0.97 \pm 0.15 10.16 \pm 0.36 73.12 \pm 0.40 3 0.87 \pm 0.17 4.98 \pm 0.32 0.38 \pm 0.10 2.78 \pm 0.22 73.58 \pm 0.33 4 0.79 \pm 0.14 5.29 \pm 0.31 0.40 \pm 0.09 3.15 \pm 0.15 73.60 \pm 0.44 6 1 0.30 \pm 0.38 12.61 \pm 2.27 0.28 \pm 0.11 5.11 \pm 0.86 74.09 \pm 0.28 2 0.29 \pm 0.16 2.85 \pm 0.37 0.13 \pm 0.08 1.48 \pm 0.22 74.81 \pm 0.47 3 0.18 \pm 0.14 1.36 \pm 0.18 0.10 \pm 0.06 0.95 \pm 0.10 74.97 \pm 0.30 4 0.18 \pm 0.07 1.35 \pm 0.05 0.08 \pm 0.02 0.76 \pm 0.03 74.60 \pm 0.24 5 0.19 \pm 0.09 1.45 \pm 0.08 0.09 \pm 0.04 0.77 \pm 0.05 74.72 \pm 0.24 6 0.19 \pm 0.08 1.60 \pm 0.14 0.08 \pm 0.02 0.89 \pm 0.09 74.15 \pm 0.28 DeepSeekMoE 2 1 0.92 \pm 0.18 11.64 \pm 0.48 0.42 \pm 0.10 5.29 \pm 0.25 65.43 \pm 0.30 (k=2,s=1) 2 0.78 \pm 0.11 6.45 \pm 0.31 0.41 \pm 0.08 3.67 \pm 0.21 62.55 \pm 0.27 4 1 2.07 \pm 0.31 28.85 \pm 1.64 0.90 \pm 0.13 13.93 \pm 0.91 71.57 \pm 0.84 2 1.77 \pm 0.56 12.14 \pm 0.50 0.67 \pm 0.32 6.74 \pm 0.24 73.70 \pm 0.14 3 1.01 \pm 0.14 9.80 \pm 0.28 0.55 \pm 0.08 4.91 \pm 0.17 73.72 \pm 0.20 4 0.52 \pm 0.15 5.60 \pm 0.14 0.23 \pm 0.10 3.22 \pm 0.11 73.89 \pm 0.30 6 1 2.43 \pm 0.73 30.80 \pm 2.43 0.99 \pm 0.31 14.06 \pm 0.90 74.26 \pm 0.19 2 0.95 \pm 0.19 12.63 \pm 0.43 0.47 \pm 0.10 7.37 \pm 0.18 74.51 \pm 0.14 3 0.39 \pm 0.13 4.85 \pm 0.14 0.20 \pm 0.06 2.44 \pm 0.09 74.98 \pm 0.19 4 0.14 \pm 0.07 2.29 \pm 0.08 0.10 \pm 0.04 1.13 \pm 0.03 74.70 \pm 0.16 5 0.12 \pm 0.03 1.39 \pm 0.08 0.08 \pm 0.02 0.66 \pm 0.04 74.62 \pm 0.14 6 0.16 \pm 0.07 1.74 \pm 0.12 0.06 \pm 0.04 0.88 \pm 0.05 74.31 \pm 0.49 Appendix H Broader Impact The investigation into Linear Mode Connectivity (LMC) within Mixture-of-Experts (MoE) architectures presents both potential positive and negative societal implications. On the positive side, this work could lead to more efficient training methods for MoE models, enhancing their scalability and computational efficiency. Such advancements may democratize access to advanced AI technologies, enabling researchers and developers with limited resources to leverage powerful models for innovative applications. Furthermore, the insights gained into the optimization dynamics and functional landscape of neural networks could contribute to the development of models that generalize more effectively, thereby improving the reliability and robustness of AI systems in critical domains such as healthcare, finance, and autonomous systems. Additionally, this research enriches the broader AI community’s understanding of neural network loss landscapes, fostering further theoretical and practical advancements. It is important to note that while these potential impacts are significant, the primary contribution of this work lies in its theoretical and foundational insights. The actual societal ramifications will largely depend on how these insights are applied and governed in practical settings. As such, responsible development and deployment practices, coupled with ongoing research into bias mitigation and ethical AI, are essential to maximizing the benefits and minimizing the risks associated with advancements in MoE architectures.