DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu · 2026
Read DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices by Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10933.txt.
Introduction
Chenyang Song and five colleagues design DECO, a sparse mixture-of-experts architecture meant to match dense transformers on devices with tight storage and memory. Expert models scale capacity cheaply in compute, but their huge total parameter count chokes end-side deployment.
DECO routes with a flexible ReLU-based scheme plus learnable per-expert scaling that balances routed and shared experts, and adds NormSiLU, an activation that steadies expert usage and raises sparsity. Activating only a fifth of routed experts, it matches dense performance and beats established expert baselines, with a custom kernel running nearly three times faster than dense code on Jetson hardware. Edge teams get expert scale without the storage penalty.
Catalog record
| CREATOR | Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50314 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10933
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu Abstract While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO , a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93 × \times speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO . 1 Introduction The scale of large language models (LLMs) has grown rapidly to achieve consistent performance gains across diverse tasks. The rising training and deployment costs for massive LLMs have made mixture-of-experts (MoE) an increasingly prominent model architecture. The key property of MoE is the sparse activation, namely, activating a small subset of expert modules from a large pool of parameters. Therefore, MoE retains high capacity and strong performance while substantially reducing computation costs. As a research hotspot, MoE has been extensively studied, from architecture design 17 ; 4 to scaling laws and compute-optimal settings 15 ; 29 . Prior work primarily pursues two objectives: high performance and low computation cost . However, when it comes to end-side deployment, a third non-negligible objective emerges: small storage overhead . Concretely, MoE with a huge number of total parameters demands substantial storage space. More critically, large MoE models may incur high memory-access costs when transferring experts between GPU high-bandwidth memory and shared memory 16 , or when moving offloaded parameters from disk/flash storage to GPU/NPU memory of end-side devices. Such latency can erode the efficiency gains afforded by sparse computation. Figure 1: The “ideal triangle” of end-side MoE. Beyond the high performance and reduced computational cost of sparse MoE, the model should maintain a minimal storage footprint, achieving high performance within dense-comparable total parameter budgets. Figure 2: The overall architecture of DECO. For router design, we adopt ReLU-based routing enhanced by learnable expert-wise router scaling. For expert design, we propose NormSiLU as a better routed-expert activation function and employ non-gated MLP experts. For precise sparsity control, we employ adaptive sparsity regularization. Therefore, as shown in Figure 1 , an ideal MoE model for end-side deployment should satisfy the above three objectives. To pursue this “ideal triangle”, we pose the question: Can a sparse MoE model achieve performance comparable to a dense model, given the same total parameter budget and the same number of training tokens? A closely related study by 16 identifies the optimal settings of DeepSeek-V3-style MoE architectures that enable them to surpass dense models under matched total parameters and computation budget. However, due to the low per-token computation of sparse MoE, in that work, MoE settings are trained on substantially more tokens under the same computation budget. We adopt a stricter setting that requires exactly the same number of training tokens. To achieve this goal, we propose DECO (Figure 2 ), a sparse MoE architecture that achieves DE nse- CO mparable performance through a fundamental revision of MoE design. For router design, conventional MoE models generally adopt TopK routing, which is non-differentiable and enforces a uniform activation ratio across all tokens. To overcome this issue, we adopt ReLU-based routing , a differentiable paradigm that enables flexible token-dependent activation ratios. Moreover, to mitigate output scale imbalances between shared and routed experts, while simultaneously accounting for potential expert heterogeneity, we introduce learnable expert-wise router scaling . This mechanism involves learnable scaling factors to calibrate the contribution of individual routed experts. The expert design is similarly optimized for stability and efficiency. Empirical analysis reveals that coupling ReLU-based routing with vanilla SiLU-activated experts results in two critical problems: a surging routed-expert activation ratio (Figure 6 ) and vanishing SiLU output magnitudes (Figure 6 ). To resolve these issues, we propose NormSiLU , which applies dual-stage normalization prior to the SiLU operator. NormSiLU stabilizes activation trends and reduces the activation ratio, alleviating the need for aggressive sparsity regularization. It also produces more stable and significant SiLU output magnitudes, promoting better utilization of expert parameters. Beyond the activation function, we employ non-gated MLP experts rather than standard gated variants, as they exhibit superior empirical compatibility with ReLU-based routing. Finally, to precisely control the activation ratio, we design an adaptive sparsity regularization that auto-scales the regularization strength. As shown in Section 4 , at the activation ratio of 20%, DECO demonstrates performance comparable to dense models when matched for total parameters and training tokens. DECO also surpasses established MoE baselines of the same scale and activation ratio. Finally, we implement a tailored acceleration kernel for DECO to test its practical inference acceleration value. Based on CUTLASS 28 , the kernel leverages tensor cores to improve computational throughput and reduces memory-access overhead by exploiting the sparse activation. Overall, the kernel achieves a speedup of 2.93 \times compared with vanilla dense inference on the real edge device of Jetson AGX Orin. 2 Related Works To achieve high performance while curbing computational growth, MoE has recently risen as the mainstream architecture. An MoE typically comprises three components: a router, a set of experts, and an auxiliary training objective. Router design. The router computes weights assigned to each expert and selects which experts to activate. It generally consists of a linear projection, an activation function, and post-processing of router scores. The activation function controls the expert selection pattern. Many MoE designs use TopK, which forces each token to activate a fixed number of experts 13 ; 6 . However, TopK is criticized for its inflexibility (an input-invariant number of active experts) and non-differentiability. TopP 12 selects experts by a threshold p , activating experts until their cumulative router score reaches at least p , thereby permitting token-dependent activation ratios. MoE++ 14 retains TopK but introduces zero-computation experts, which indirectly allows variable computation cost. To improve differentiability, ReMoE 32 and BlockFFN 26 adopt ReLU for expert selection. Since ReLU naturally produces considerable zero outputs while remaining differentiable, it enables smoothly learnable activation ratios and delivers performance advantages. Post-processing of router scores primarily normalizes expert weights to preserve a consistent output scale. Most designs use Softmax as the score normalizer. DeepSeek-V3 17 instead applies element-wise Sigmoid followed by unit-sum normalization, which helps mitigate extremely skewed score distributions. Notably, DeepSeek-V3 also introduces a scalar scaling factor applied to router scores, which helps balance contributions between shared and routed experts. In this work, DECO adopts ReLU-based routing, and replaces the fixed scalar scaling factor with learnable expert-wise router scaling factors,