S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning

Giang Do, Hung Le, Truyen Tran · 2025

Read S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning by Giang Do, Hung Le, Truyen Tran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23007.txt.

Introduction

Giang Do, Hung Le, and Truyen Tran make mixture-of-experts models steadier by training them with deliberate noise, in a 2025 paper. Sparse expert models route each token to a few specialist subnetworks, which saves compute in giant language models. Training often collapses, though, as experts blur into sameness or starve.

Their design, S2MoE, learns from both clean and noisy inputs, a stance they call learning under uncertainty. Wider expert embeddings and stochastic routing keep experts distinct. Across tasks the model matched rival routers while cutting inference cost by twenty-eight percent. Cheaper, steadier experts matter as language models keep growing.

Catalog record

CREATORGiang Do, Hung Le, Truyen Tran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning Giang Do, Hung Le, Truyen Tran Abstract Sparse Mixture of Experts (SMoE) enables efficient training of large language models by routing input tokens to a select number of experts. However, training SMoE remains challenging due to the issue of representation collapse. Recent studies have focused on improving the router to mitigate this problem, but existing approaches face two key limitations: (1) expert embeddings are significantly smaller than the model’s dimension, contributing to representation collapse, and (2) routing each input to the Top-K experts can cause them to learn overly similar features. In this work, we propose a novel approach called Robust Sparse Mixture of Experts via Stochastic Learning (S2MoE), which is a mixture of experts designed to learn from both deterministic and non-deterministic inputs via Learning under Uncertainty. Extensive experiments across various tasks demonstrate that S2MoE achieves performance comparable to other routing methods while reducing computational inference costs by 28% . 1 Introduction Sparse Mixture of Experts (SMoE) models have achieved notable success in natural language processing (NLP) and visual representation learning tasks ( Du et al., 2022 ; Fedus et al., 2022 ; Riquelme et al., 2021a ; Shen et al., 2023 ) . These advancements build on the Transformer architecture ( Vaswani et al., 2017 ) and its variants ( Child et al., 2019 ; Dai et al., 2019b ) , which leverage large datasets and significant compute resources. However, training large Transformer models can be prohibitively expensive, requiring extensive compute hours ( Kaddour et al., 2023 ) . To address this, SMoE models activate only a subset of experts for each input, reducing inference time compared to dense models ( Shazeer et al., 2017 ; Zoph et al., 2022 ; Artetxe et al., 2022 ; Krajewski et al., 2024 ) . Figure 1: BPC ( Bits-per-character ) on the Text8 dataset with varying numbers of experts used for inference. S2MoE requires the activation of only one expert to achieve comparable performance with other routing methods, resulting in a savings of 28% in computational inference costs. All methods have the same FLOPs. Despite promising results, SMoE models face the challenge of representation collapse , where either a few experts dominate the routing or all experts learn similar representations Chi et al. (2022a) ; Chen et al. (2022) . To address this, research has focused on improving router policies Chi et al. (2022b) ; Chen et al. (2023a) ; Do et al. (2023a) . One solution, SMoE-Dropout Chen et al. (2023b) , freezes a randomly initialized router throughout training and gradually increases the number of active experts. However, these existing approaches have two limitations: (1) the expert embeddings are much smaller than the model dimension, leading to representation collapse, and (2) routing each input to the Top-K experts can cause them to learn similar features. To address these limitations, this work proposes a novel approach called Robust Sparse Mixture of Experts via Stochastic Learning (S2MoE) to enhance expert knowledge and prevent overlap in their learning. Instead of feeding the same input to the Top-K experts, S2MoE utilizes a Gaussian noise to enhance feature learning prior to expert selection, a concept that has been validated in the vision domain Luisier et al. (2011) ; Russo (2003) ; Chen et al. (2024) . By doing this, S2MoE can enhance expert learning efficiency during training and reduce the representation collapse issue. To showcase its effectiveness, we perform comprehensive evaluations across various NLP tasks, comparing S2MoE with several state-of-the-art SMoE routing strategies. Additionally, S2MoE reaches the same performance levels with fewer experts during inference, greatly improving the efficiency of deploying LLMs in real-world applications. Figure 1 demonstrates that S2MoE requires the activation of only one expert to achieve comparable performance with other routing methods, resulting in a savings of 28% in computational inference costs. 2 Related Work Sparse Mixture of Experts (SMoE). Sparse Mixure of Experts (SMoE) building on the Mixture of Experts framework ( Jacobs et al., 1991 ; Jordan and Jacobs, 1994 ) , gained traction with large language models and has since been applied in various fields, including computer vision and speech recognition ( Zhou et al., 2022 ; Riquelme et al., 2021b ) . However, SMoE encounters the challenge of representation collapse, where experts produce similar outputs. To combat this, various methods have emerged, such as XMoE, which uses low-dimensional routing scores ( Chi et al., 2022b ) , and SMoE-dropout, which activates more experts gradually ( Chen et al., 2023a ) . Other strategies include HyperRouter ( Do et al., 2023a ) and StableMoE ( Dai et al., 2022a ) , both aimed at improving the stability and robustness of routers. Despite these innovations, representation collapse remains a persistent issue ( Pham et al., 2024 ) . Our approach differs by emphasizing enhanced feature learning, which helps expand experts’ knowledge and reduce the collapse issue. Learning under Uncertainty. Learning under Uncertainty have a long history that consists well-known research topic in Machine Learning such as Dropout Srivastava et al. (2014) , Bayesian neural networks Friedman et al. (1997) and noise regularized learning Noh et al. (2017) . Some studies Moreno-Barea et al. (2018) ; Maharana et al. (2022) ; Hu et al. (2018) have applied learning under uncertainty in data augmentation, a common technique in the vision domain that helps models improve robustness and reduce overfitting. Additionally, Chen et al. (2024) enhanced feature learning for vision models by incorporating Gaussian noise generation. 3 Methodology We propose a novel model, the Stochastic Sparse Mixture of Experts (S2MoE), which is a mixture of experts designed to learn from both deterministic and non-deterministic inputs. As illustrated in Figure 2 , our method consists of two parts: (1) learning with the original input and (2) learning with noise-added input. To regulate the quality of the noise generation process, we introduce uncertainty loss, as shown in the Equation 4 . 3.1 Preliminaries Sparse Mixture of Experts. The Sparse Mixture of Experts (SMoE) is typically a transformer architecture that replaces the multi-layer perceptron (MLP) layers in standard transformers with Mixture of Experts (MoE) layers, inspired by ( Shazeer et al., 2017 ) . Given \mathbf{x}\in\mathbb{R}^{n\times d} as the output of the multi-head attention (MHA) layer, the result of the SMoE with N experts is a weighted sum of each expert’s computation, E_{i}(x) , weighted by the router function \mathcal{S}(x) : \displaystyle f_{\mathrm{SMoE}}(\boldsymbol{x})=\sum_{i=1}^{N}\mathcal{S}(\boldsymbol{x})_{i}\cdot E_{i}(\boldsymbol{x}) (1) Where \mathcal{S}(x) is computed by TopK function as below the Equation 2 , and W_{e} is a learnable experts embeddings. \mathcal{S}(\boldsymbol{x})=\operatorname{TopK}(\operatorname{softmax}(W_{e}\cdot x),k) (2) \operatorname{TopK}(\boldsymbol{v},k)=\begin{cases}\boldsymbol{v_{i}}&\text{if }\boldsymbol{v_{i}}\text{ is in the top }k\text{ largest of }\boldsymbol{v},\\ -\infty&\text{otherwise}.\end{cases} 3.2 Robust Sparse Mixture of Experts via Stochastic Learning (S2MoE) Uncertainty Modeling. Inspired by Chen et al. (2024) , we introduce the Gaussian Noise Module as Figure 2 , which directly applies to the representation space to enhance model feature learning. Given a representation \mathbf{x}\in\mathbb{R}^{n\times d} and \mu_{x} , \sigma_{x} representing the mean and standard deviation of the Gaussian noise calculated per batch from the feature x , the noise-augmented input \hat{x} is defined by the following formula: \hat{x}=N_{1}\cdot x+N_{2} , where N_{1} , N_{2} are two noise vectors that are sampled from two Gaussian distribution (

Read the complete text