BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen · 2025
Read BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity by Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08771.txt.
Introduction
Chenyang Song and co-authors reworked a sparse network design in 2025 so small devices can run large language models. In a mixture-of-experts model, each word lights only a few expert blocks. That saves work per word, yet a run of words tends to light most blocks in turn, which kills the gain on weak gear. Their design, BlockFFN, routes words with a smooth switch that can be trained end to end.
Training goals push both lone words and runs of words to stay sparse. New run code joins that sparsity with a guess-ahead trick for the first time. On real pocket devices, their build ran up to near four times faster than a dense model. Most experts stayed dark even across runs of words.
Catalog record
| CREATOR | Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen Abstract To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing of vanilla MoE hurts model performance. Moreover, while each token activates only a few parameters, these sparsely-activated architectures exhibit low chunk-level sparsity, indicating that the union of multiple consecutive tokens activates a large ratio of parameters. Such a sparsity pattern is unfriendly for acceleration under low-resource conditions (e.g., end-side devices) and incompatible with mainstream acceleration techniques (e.g., speculative decoding). To address these challenges, we introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques. Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing. Next, to promote both token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly. Finally, we implement efficient acceleration kernels, combining activation sparsity and speculative decoding for the first time. The experimental results demonstrate the superior performance of BlockFFN over other MoE baselines, achieving over 80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67 × \times speedup on real end-side devices than dense models. All codes and checkpoints are available publicly 1 1 1 https://github.com/thunlp/BlockFFN . 1 Introduction To reduce the high costs of training and deploying large language models (LLMs), various efficient LLM architectures are proposed ( Wan et al. 2023 ) . A popular paradigm is designing architectures with activation sparsity , indicating that a considerable part of LLM parameters contribute weakly to LLM outputs given specific inputs, and thus can be skipped (i.e., not activated) in the forward and backward computation. Mixture-of-experts (MoE) is an outstanding representative and has been adopted by many recent models such as Mixtral-8 \times 22B ( Jiang et al. 2024 ) and DeepSeek-V3 ( Liu et al. 2024b ) . Based on MoE, techniques including load balancing ( Wang et al. 2024a ) and expert parallelism ( He et al. 2021 ) are adopted to achieve remarkable efficiency on cloud-side servers. However, few efforts explore sparsely-activated architectures under low-resource conditions (e.g., end-side devices), where it is challenging to deploy huge MoE models and highly-distributed frameworks with expert parallelism. For end-side MoE models, which generally serve only a few users, some typical issues are not required to be considered (e.g., load balancing, see Appendix A ), while raising the following two challenges. Performance compromise caused by imperfect routing. Existing MoE models generally compromise performance due to two significant routing drawbacks: non-differentiability and inflexibility. Specifically, most mainstream MoE models adopt a TopK router ( Fedus et al. 2022 ; Jiang et al. 2024 ) with discrete and non-differentiable computation. Consequently, only activated parameters have complete gradients and are well updated at each step, which harms the convergence efficiency of MoE models ( Liu et al. 2024c ) . Moreover, TopK makes each token activate the same number of experts, enforcing an inflexible activation pattern, which may weaken model performance ( Huang et al. 2024 ) . Few works can well alleviate both drawbacks, see Section 2.1 . Acceleration unfriendliness caused by low chunk-level sparsity (CLS). To make a sparsely-activated architecture more friendly for acceleration, just increasing the ratio of weakly-contributed experts for each token, namely the token-level sparsity (TLS), is not enough. Instead, the ratio of weakly-contributed experts for multiple consecutive tokens, namely the chunk-level sparsity (CLS), is critical for practical acceleration. (a) (b) (c) Figure 1 : (a) For models with high TLS (except BlockFFN-1.2B), CLS quickly collapses to a lower level as a chunk contains more consecutive tokens. (b) Smoothed training curves of the BlockFFN-1.2B and other MoE baselines with the same total and activated parameters. (c) The speed of different decoding methods, where “Ours (1-Tok)” and “Ours (32-Tok)” are our token-level and chunk-level sparsity-based acceleration kernels, respectively. Specifically, a low CLS can eliminate the value of activation sparsity when combined with speculative decoding, a mainstream acceleration method that requires LLMs to process multiple consecutive tokens at the same time ( Leviathan et al. 2023 ) . Besides, other important resource-saving techniques, such as offloading, also become more challenging to implement due to the large differences in activation patterns within a specific chunk, leading to frequent GPU-CPU communication overheads. Unfortunately, existing works mainly focus on improving TLS ( Mirzadeh et al. 2023 ; Song et al. 2024 ; Song et al. 2025 ) , but low CLS values still exist in most sparse architectures (see Figure 1(a) ). To address the above challenges, we introduce the following novel MoE architecture, as well as its training techniques and efficient end-side deployment. For model architectures, we propose BlockFFN , a novel MoE paradigm that minimizes performance compromise. Specifically, its router module integrates ReLU and RMSNorm. ReLU computes differentiable and flexible activation patterns, enabling each token to determine the number of activated experts adaptively. RMSNorm generates learnable magnitudes of activation values. This separation of activation patterns and magnitudes alleviates the disturbance on activation magnitudes induced by regularization (e.g., the shrinkage of activation magnitudes caused by L1 ( Rajamanoharan et al. 2024 ) ), as regularization applies solely to the ReLU activation pattern. Through experiments, we demonstrate the better performance of BlockFFN compared to other MoE variants (Figure 1(b) , Table 2 and 3 ). For training techniques, we introduce CLS-aware training objectives to improve the CLS of BlockFFN, which can make BlockFFN more friendly for acceleration, including the activation locality loss and the chunk sparsification loss. The former is to increase the similarity of activation patterns between neighbor tokens, helping reduce the gap between TLS and CLS. The latter is to increase the overall sparsity level. While existing sparsification objectives such as L1 ( Song et al. 2025 ) are applied to each token independently, our chunk sparsification loss directly minimizes the probability that a specific expert is activated by at least one token within the chunk. In experiments, we obtain average TLS values higher than 80% and 8-token CLS values higher than 70% (Table 2 ). For end-side deployment, we implement efficient acceleration kernels for BlockFFN, combining activation sparsity and speculative decoding for the first time, and demonstrate its practical effectiveness on real end-side devices such as NVIDIA Jetson Orin NX. To enhance the efficiency of verifying multiple tokens in speculative sampling, we leverage the high activation similarity across multiple tokens induced by the high CLS level. This enables merging memory accesses to the same expert across different tokens, reducing the memory access volume to the union of experts activated by these tokens (Figure 2 ). Additionally, we implement the kernels based on CUTLASS ( Thakkar et al. 2023 ) and utilize tensor cores to boost computational efficiency. Overall, the kernel achieves an acceleration ratio of 3.67 \times , compared to the baseline auto-regressive (AR) decoding (Figure 1(c) and Table 6 ). 2 Preliminaries and Related Works Model Activation values \mathbf{A}(\mathbf{x}) Single expert outputs E_{i}(\mathbf{x}) Neuron-level activation sparsity Non-gated FFN \sigma(\mathbf{W}_{up}^{T}\mathbf{x}) \mathbf{w}_{down} \mathbf{W}_{up}\in\mathbb{R}^{d_{h}\times N_{e}} \mathbf{w}_{down}\in\mathbb{R}^{d_{h}} Gated FFN \sigma(\mathbf{W}_{gate}^{T}\mathbf{x}) (\mathbf{w}_{up}^{T}\mathbf{x})\cdot\mathbf{w}_{down} ( Shazeer 2020 ) \mathbf{W}_{gate}\in\mathbb{R}^{d_{h}\times N_{e}} \mathbf{w}_{up},\mathbf{w}_{down}\in\mathbb{R}^{d_{h}} dReLU \mathrm{ReLU}(\mathbf{W}_{gate}^{T}\mathbf{x}) \mathrm{ReLU}(\mathbf{w}_{up}^{T}\mathbf{x})\cdot\mathbf{w}_{down} ( Song et al.