CoSMoEs : Compact Sparse Mixture of Experts

Patrick Huber, Akshat Shrivastava, Ernie Chang, Chinnadhurai Sankar, Ahmed Aly, Adithya Sagar · 2025

Read CoSMoEs : Compact Sparse Mixture of Experts by Patrick Huber, Akshat Shrivastava, Ernie Chang, Chinnadhurai Sankar, Ahmed Aly, Adithya Sagar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00245.txt.

Introduction

Patrick Huber and co-authors published this model design study in 2025. Mixture-of-experts models split work among many specialist sub-networks, activating only a few per input. They shine at huge scale but are little explored in small models for phones and devices. On-device use squeezes three things: quality, memory, and speed. The team set out to make sparse expert models practical under those limits.

First, in fair comparisons that remove confounding factors, their expert models beat dense models of equal computing cost at device scale. A trick that splits expert weights into shared and specific parts improved quality. Second, they reworked how experts are loaded from storage during use. Smarter offloading cut memory pressure and response delay. The authors present the package, called CoSMoEs, as evidence sparse designs deserve a place in small on-device systems, not only in giant data-center models.

Catalog record

CREATORPatrick Huber, Akshat Shrivastava, Ernie Chang, Chinnadhurai Sankar, Ahmed Aly, Adithya Sagar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoSMoEs : Compact Sparse Mixture of Experts Patrick Huber, Akshat Shrivastava, Ernie Chang, Chinnadhurai Sankar, Ahmed Aly, Adithya Sagar Abstract Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mixture of Experts (CoSMoEs) for on-device inference. Specifically, we tackle the three main on-device dimensions: Quality, Memory and Latency. Along the quality axis, we show that in a fair evaluation (removing confounding factors) MoE architectures outperform FLOP-aligned dense models at on-device scale. We introduce weight-decomposed experts, further improving the MoE model performance. Regarding model memory and latency, we significantly improve model offloading efficiency and, in turn, reduce model inference latency. † † date: August 24, 2026 † † correspondence: Patrick Huber at [email protected] and Adithya Sagar at [email protected] † † Code: Model code and checkpoints will be released soon! Figure 1 : Server-side training environment (left) compared to the memory-constraint inference environment (right), showing deployment restrictions for parameter heavy MoEs and large dense models on edge devices. 1 Introduction Mixture of Experts (short: MoEs) have been a popular extension of the transformer architecture Vaswani et al. (2023) , introducing the idea that each token of the input sequence is not fed through a single, dense network per layer, but a set of sub-networks, or “experts”. To allow every input token to utilize a mixture of expert, the sub-networks are usually combined with a gating mechanism, which determines the contribution of each expert. In the general MoE setting, going back to Jacobs et al. (1991) and Jordan and Jacobs (1993) , all experts are used to compute the final layer output. Building on top of this general architecture, sparse Mixture of Expert models have been proposed as a more compute-efficient alternative, only allowing a subset of experts to be activated for each token Cai et al. (2024) . Recently, many foundational models have adopted the MoE approach, such as Qwen Bai et al. (2023) ; Yang et al. (2024) , OLMoE Muennighoff et al. (2024) , Mixtral Jiang et al. (2024) , Deepseek ( 2024 ) , inter alia . In comparison to large-scale foundational Mixture of Expert models, optimized for highly parallelized server-side inference, in this work, we focus on small-scale foundational MoEs models deployed on edge devices 1 1 1 We focus on two model sizes: “Wearables-sized” models at 200M active parameters and “Phone-sized” models at 1.4B active parameters. . As such, this comes with a set of challenges around single-sample, on-device inference, which can be classified into three categories: Quality, Memory and Latency. Quality: We tackle the fundamental research question if Mixture-of-Expert models can improve language modeling abilities over dense models at on-device scale. In comparison to previous work (e.g. Jiang et al. (2024) ), we set up a truly fair comparison between MoEs and dense models. Here, we define a “fair comparison” of an MoE model against its dense counterpart by aligning for both, the same number of active parameters (i.e. FLOP aligned, short: FA ) and total parameters (i.e. parameter aligned, short: PA ). We further assume that a “fair comparison” between models should reduce confounding factors. Along those lines, we normalize models for training datasets, recipes, and architectures wherever possible. This way, we can make a clear performance attribution to the MoE component in isolation. In our evaluation, we show that MoE-style architectures improve the average language modeling performance by at least 2.35% absolute across on-device model sizes. Based on these results, we propose a novel MoE model extension following the core intuition of “expert specialization”. Using weight-decomposed experts, we show up to an additional 1.1% language modeling improvements. Memory/Latency: For server-side models, language modeling ability presents the main dimension for model improvements. In the on-device context, however, we face two additional hard constraints: Memory and latency. As depicted in Figure 1 , models trained in server environments, with loose memory and latency restrictions, face additional constraints for inference on edge-devices. While these restrictions are architecture independent, MoE-style models with a high total parameter count are more impacted. Luckily, the sparsity property of MoE architectures allows to circumvent this restriction by offloading unused experts, effectively reducing the model size in memory to the active parameter count (see Figure 1 ). Reducing the model memory through expert offloading, however, comes at the cost of 4-20x increased inference latency, since experts might need to be offloaded for every single token in the output sequence Xue et al. (2024) . To relax this memory/latency trade-off, we propose a novel “block-wise expert selection” loss, reducing expert offloads by 6x and, in turn, improving inference latency by 50% compared to default offloaded MoEs. 2 CoSMoEs Models 2.1 Sparse Mixture-of-Experts At the core of this work is the sparse Mixture-of-Expert (MoE) architecture, popularized by works such as GShard Lepikhin et al. (2020) and Switch Transformers Fedus et al. (2022) . While MoEs can generally be implemented for different parts of the architecture, the most common approach is to replace the single dense feed-forward layer with a router component and multiple experts (see Figure 2 ). Selecting a discrete subset of experts at each step, sparse MoE models can be defined by their active parameters (FLOPs) and total parameters (model size in memory). The resulting FLOP-to-parameter ratio directly translates to increased training and inference efficiency, without sacrificing model performance. To find a suitable subset of experts, different expert routing paradigms have been established, either selecting experts per token (token choice or “TC”) Shazeer et al. (2017) or per expert (expert choice or “EC”) Zhou et al. (2022) . Here, we use the token choice expert routing paradigm (illustrated in Figure 2 ) following the findings in OLMoE Muennighoff et al. (2024) , showing that EC does not bring clear improvements for text-only models. Please note that from here on out, we will refer to sparse MoEs as solely “MoEs” for brevity. However, all evaluated models in this paper are sparse versions of Mixture-of-Expert models. Figure 2 : Sparse Mixture-of-Experts architecture with Token Choice (TC) Routing and k=2 2.2 Weight-Decomposed Experts To reduce the naturally large total parameter count of MoE-style models, we propose a lightweight definition of experts using matrix weight decompositions (“WD”) similar in spirit to Low Ranking (“LoRa”) adapters Hu et al. (2021) . Intuitively, individual experts are intended to “specialize” towards a subset of, ideally, \frac{1}{\#Experts} tokens. Based on this intuition, we replace expert matrices of shape n\times m with weight decompositions of shape n\times r and r\times m as shown in Figure 3 and defined in Equation 1 : M_{n\times m}\approx L_{n\times r}\times R_{r\times m} (1) Here, the original matrix M is replaced by L and R , with r\ll n and r\ll m . In preliminary experiments, we test multiple reduction factors for r and find that a decomposition of half the hidden dimension results in the best trade-off between parameter reduction and model performance. Weight decomposed models are from here on out prefix with a WD term. To ensure a paramter-aligned comparison, we adjust the number of heads and layers as further elaborated on in section 3.1 . 2.3 Block-wise Expert Selection We now explore the second restrictive dimension of

Read the complete text