MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
Zehua Liu, Han Wu, Ruifeng She, Xiaojin Fu, Xiongwei Han, Tao Zhong, Mingxuan Yuan · 2025
Read MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models by Zehua Liu, Han Wu, Ruifeng She, Xiaojin Fu, Xiongwei Han, Tao Zhong, Mingxuan Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23100.txt.
Introduction
Liu and co-authors present a 2025 paper on making mixture-of-experts language models cheaper to run. Such models switch on only a subset of their parts for each word piece, which saves effort. Still, they eat memory and need heavy data traffic in training. The authors propose a new layout, called MoLAE, built on a shared low-size latent space.
Each expert works through that shared space first, then applies its own small transform. The design cuts parameter counts sharply where model width is large. Math analysis gives rules for converting trained models into the new form in two steps. Tests show performance close to the standard layout at much lower cost, the authors report.
Catalog record
| CREATOR | Zehua Liu, Han Wu, Ruifeng She, Xiaojin Fu, Xiongwei Han, Tao Zhong, Mingxuan Yuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models Zehua Liu, Han Wu, Ruifeng She, Xiaojin Fu, Xiongwei Han, Tao Zhong, Mingxuan Yuan Abstract Mixture of Experts (MoE) has become a key architectural paradigm for efficiently scaling Large Language Models (LLMs) by selectively activating a subset of parameters for each input token. However, standard MoE architectures face significant challenges, including high memory consumption and communication overhead during distributed training. In this paper, we introduce Mixture of Latent Experts (MoLAE), a novel parameterization that addresses these limitations by reformulating expert operations through a shared projection into a lower-dimensional latent space, followed by expert-specific transformations. This factorized approach substantially reduces parameter count and computational requirements, particularly in existing LLMs where hidden dimensions significantly exceed MoE intermediate dimensions. We provide a rigorous mathematical framework for transforming pre-trained MoE models into MoLAE architecture, characterizing conditions for optimal factorization, and developing a systematic two-step algorithm for this conversion. Our comprehensive theoretical analysis demonstrates that MoLAE significantly improves efficiency across multiple dimensions while preserving model capabilities. Experimental results confirm that MoLAE achieves comparable performance to standard MoE with substantially reduced resource requirements. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks ( Bommasani et al., 2021 ; Zhuang et al., 2020 ) , from text generation ( Achiam et al., 2023 ; Dubey et al., 2024 ) to complex reasoning ( Guo et al., 2025 ) . As these models scale to increasingly larger parameter spaces, the Mixture of Experts (MoE) architecture ( Jacobs et al., 1991 ; Jordan and Jacobs, 1994 ) has emerged as a promising paradigm for efficiently scaling model capacity without proportionally increasing computational costs. By selectively activating only a subset of parameters for each input token, MoE models achieve parameter efficiency while maintaining manageable inference latency. Despite their theoretical and empirical advantages, standard MoE architectures ( Dai et al., 2024 ) face significant practical limitations that inhibit broader deployment. These models require substantial memory resources to store parameters across numerous expert modules in Feed-Forward Network (FFN) layers and create communication bottlenecks during distributed training due to all-to-all data transfers. These challenges become increasingly pronounced as models scale to hundreds of experts, potentially limiting their applicability in resource-constrained environments. Through systematic investigation of parameter utilization in MoE architectures, we discover substantial redundancy within the FFN layers of current MoE models. By analyzing Qwen1.5-MoE-A2.7B ( Team, 2024 ) , we reveal that a significant proportion of parameters in its FFN layers can be effectively approximated through lower-dimensional representations without compromising model performance. This empirical observation motivates a fundamental rethinking of expert parameterization in neural architectures. Figure 1: Architectural comparison between MoE and MoLAE in the FFN layer. In both diagrams, N_{r} denotes the number of routed experts. MoLAE extends the conventional MoE architecture by introducing latent mappings B_{\text{up}} , B_{\text{gate}} , and B_{\text{down}} that capture shared information across experts. Expert-specific information is encapsulated in the mappings A_{\text{up}}^{i} , A_{\text{down}}^{i} , and A_{\text{gate}}^{i} for each expert i . In this work, we introduce M ixture o f LA tent E xperts ( MoLAE ), a novel parameterization paradigm that addresses the core inefficiencies of traditional MoE architectures. Our key insight is that expert modules in standard MoE models contain significant redundancy and operate in unnecessarily high-dimensional spaces. MoLAE reformulates each expert operation through a mathematically principled two-phase transformation: (1) a shared projection into a compressed latent space, followed by (2) expert-specific transformations within this lower-dimensional manifold. Formally, MoLAE implements this insight by factorizing each expert’s weight matrix W^{i}\in\mathbb{R}^{m\times n} into the product A^{i}B , where A^{i}\in\mathbb{R}^{m\times m} represents expert-specific transformations and B\in\mathbb{R}^{m\times n} represents a shared projection into a latent space of dimension. This factorization yields substantial parameter reduction, particularly in contemporary LLM architectures where the hidden dimension n significantly exceeds the MoE intermediate dimension m . Our contributions are as follows: 1) We propose MoLAE, a parameter-efficient architecture that achieves competitive performance with standard MoE models while requiring significantly fewer parameters and reduced computational overhead. 2) We develop a theoretically grounded framework for transforming pre-trained MoE models into the MoLAE architecture, including a mathematical characterization of optimal factorization conditions and an efficient two-stage algorithm incorporating low-rank approximation techniques. 3) Through comprehensive empirical evaluation on multiple benchmark datasets, we demonstrate that MoLAE preserves or enhances model capabilities across diverse language tasks while substantially improving parameter efficiency, thereby enabling more economical scaling of large language models. 2 Related Works Finer-Grained Mixture of Experts . Mixture of Experts (MoE), initially introduced by Jacobs et al. (1991) and Jordan and Jacobs (1994) , has garnered significant attention in recent years ( Aljundi et al., 2017 ; Collobert et al., 2001 ; Deisenroth and Ng, 2015 ; Eigen et al., 2013 ; Rasmussen and Ghahramani, 2001 ; Shahbaba and Neal, 2009 ; Theis and Bethge, 2015 ) . Lepikhin et al. (2020) pioneered the integration of MoE technology into transformer architectures, enabling substantial parameter scaling while maintaining computational efficiency. Subsequently, numerous studies have advanced MoE algorithms, particularly focusing on replacing feed-forward network (FFN) layers with MoE layers in transformer-based Large Language Models (LLMs) ( Dai et al., 2024 ; Du et al., 2022 ; Fedus et al., 2022 ; Xue et al., 2024 ; Zoph et al., 2022 ) . However, conventional GShard models exhibit limitations in capturing domain-specific expertise due to their relatively small number of experts. To address this constraint and enhance expert specialization, finer-grained MoE architectures were proposed by Dai et al. (2024) and subsequently implemented in several state-of-the-art models ( Guo et al., 2025 ; Liu et al., 2024 ; Team, 2024 ) . In contrast to traditional GShard MoE designs, finer-grained variants incorporate substantially more experts, each with reduced parameter counts, enabling greater specialization in domain-specific knowledge representation and processing. This approach not only refines the decomposition of knowledge across experts, facilitating more precise learning, but also enhances the flexibility of expert activation combinations, allowing for more specialized and targeted knowledge capture. Algorithmic Design of MoE . The introduction of expert modules in LLMs introduces several algorithmic challenges that must be addressed during both training and inference phases. A critical aspect of MoE design is the gating function, which orchestrates the engagement of expert computations and the combination of their respective outputs. The gating mechanisms can be broadly categorized into three types: sparse, which activates a subset of experts; dense, which activates all experts; and soft, which encompasses fully-differentiable approaches including input token merging and expert merging ( Pan et al., 2024 ; Zadouri et al., 2023 ; Puigcerver et al., 2022 ) . Sparse token-choice gating, where the gating function selects top-k experts for each input token, is the most prevalent approach ( Fedus et al., 2022 ; Lepikhin et al., 2020 ; Zoph et al., 2022 ) . This method is often augmented with auxiliary loss functions to promote balanced expert utilization ( Lepikhin et al., 2020 ; Fedus et al., 2022 ; Du et al., 2022 ) . Alternative approaches include expert-choice gating, where each expert selects the top-k tokens they will process ( Zhou et al., 2022 ; Zhou et al., 2023 ) , and non-trainable gating mechanisms that use predetermined routing