ReMoE: Fully Differentiable Mixture-of- Experts with ReLU Routing
Ziteng Wang, Jun Zhu, Jianfei Chen · 2024
Read ReMoE: Fully Differentiable Mixture-of- Experts with ReLU Routing by Ziteng Wang, Jun Zhu, Jianfei Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14711.txt.
Introduction
Ziteng Wang, Jun Zhu, and Jianfei Chen present a new form of mixture-of-experts model for large neural nets. Their paper is from two thousand twenty-four. Such models hold many expert parts but switch on only a few for each token, so size can grow with no rise in compute cost. The classic router that picks experts learns in jumps, which caps its gains.
Their design, ReMoE, uses a ReLU router that can be trained in a smooth way from end to end. They add ways to keep the router sparse and to share load evenly among experts. Tests across sizes and expert counts show steady gains over the classic router. Growth with more experts is also stronger.
Catalog record
| CREATOR | Ziteng Wang, Jun Zhu, Jianfei Chen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14711
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReMoE: Fully Differentiable Mixture-of- Experts with ReLU Routing Ziteng Wang, Jun Zhu, Jianfei Chen Abstract Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to scale up model capacity without increasing the computation budget. However, vanilla TopK routers are trained in a discontinuous, non-differentiable way, limiting their performance and scalability. To address this issue, we propose ReMoE, a fully differentiable MoE architecture that offers a simple yet effective drop-in replacement for the conventional TopK+Softmax routing, utilizing ReLU as the router instead. We further propose methods to regulate the router’s sparsity while balancing the load among experts. ReMoE’s continuous nature enables efficient dynamic allocation of computation across tokens and layers, while also exhibiting domain specialization. Our experiments demonstrate that ReMoE consistently outperforms vanilla TopK-routed MoE across various model sizes, expert counts, and levels of granularity. Furthermore, ReMoE exhibits superior scalability with respect to the number of experts, surpassing traditional MoE architectures. The implementation based on Megatron-LM is available at https://github.com/thu-ml/ReMoE . 1 Introduction Transformer models ( Vaswani, 2017 ) consistently improve performance as the number of parameters increases ( Kaplan et al., 2020 ) . However, scaling these models is constrained by computation resources. Sparsely activated Mixture-of-Experts (MoE) ( Shazeer et al., 2017 ) mitigates this challenge by employing a sparse architecture that selectively activates a subset of parameters during both training and inference. This conditional computation allows MoE models to expand model capacity without increasing computational costs, offering a more efficient alternative to dense models. The key component in MoE is the routing network, which selects the experts to activate for each token. Various routing methods ( Shazeer et al., 2017 ; Lewis et al., 2021 ; Roller et al., 2021 ; Zhou et al., 2022 ) have been proposed, with TopK routing ( Shazeer et al., 2017 ) being the most commonly adopted. However, the vanilla TopK router introduces a discrete and non-differentiable training objective ( Shazeer et al., 2017 ; Zoph et al., 2022 ) , limiting the performance and scalability. Recent works on fully-differentiable MoE aim to overcome this limitation. Soft MoE ( Puigcerver et al., 2023 ) introduces token merging, while SMEAR ( Muqeeth et al., 2023 ) proposes expert merging. However, both approaches break token causality, making them unsuitable for autoregressive models. Lory ( Zhong et al., 2024 ) improves upon SMEAR and is applicable to autoregressive models. But it underperforms vanilla MoE with TopK routing. In this work, we address the discontinuities by introducing ReMoE, an MoE architecture that incorporates ReLU routing as a simple yet effective drop-in replacement for TopK routing. Unlike TopK routing, which computes a softmax distribution over the experts and calculates a weighted sum of the largest K experts, ReLU routing directly controls the active state of each expert through a ReLU gate. The number of active experts is determined by the sparsity of the ReLU function. To maintain the desired sparsity, we propose adding a load-balancing refined L_{1} regularization to the router outputs, with an adaptively tuned coefficient. This approach ensures that ReMoE maintains the same computational costs as TopK-routed MoE. Compared to TopK routing, ReLU routing is continuous and fully differentiable, as the ReLU function can smoothly transition between zero and non-zero values, indicating inactive and active. Besides, ReLU routing manages the “on/off” state of each expert independently, offering greater flexibility. Moreover, the number of activated experts can vary across tokens and layers, enabling a more efficient allocation of computational resources. Further analysis reveals that ReMoE effectively learns to allocate experts based on token frequency and exhibits stronger domain specialization. Our experiments on mainstream LLaMA ( Touvron et al., 2023 ) architecture demonstrate that ReLU routing outperforms existing routing methods including TopK routing and fully-differentiable Lory. Through an extensive investigation across model structures, we find that ReMoE consistently outperforms TopK-routed MoE across a broad range of active model sizes (182M to 978M), expert counts (4 to 128), and levels of granularity (1 to 64) ( Krajewski et al., 2024 ) . Notably, in terms of scaling behavior, we observe that ReMoE exhibits a steeper performance improvement as the number of experts scales up, surpassing traditional MoE models. Figure 1: Compute flows of vanilla MoE with TopK routing and ReMoE with ReLU routing. Positive values are shown in orange, and negative values in blue, with deeper colors representing larger absolute values. Zeros, indicating sparsity and computation savings, are shown in white. The red dash arrows in TopK routing indicate discontinuous operations. Compared with TopK routing MoE, ReMoE uses ReLU to make the compute flow fully differentiable. 2 Preliminaries 2.1 MoE for Decoder-Only Transformer A typical decoder-only Transformer model consists of L layers, each containing a Self-Attention module and a Feed-Forward Network (FFN) module. MoE modifies this structure by replacing each FFN module with an MoE module, which comprises a small router and several experts \text{FFN}_{1},\dots,\text{FFN}_{E} , where each expert is equivalent to the original FFN and E denotes the number of experts. Given the input {\bm{x}}^{l}=({\bm{x}}^{l}_{t})_{t=1}^{T}\in\mathbb{R}^{T\times d} of the layer l , where T is the number of tokens in a batch and d is the hidden size, the output {\bm{y}}^{l}=({\bm{y}}^{l}_{t})_{t=1}^{T} is computed as: \displaystyle{\bm{y}}^{l}_{t}=\sum_{e=1}^{E}R({\bm{x}}^{l}_{t})_{e}\text{FFN}_{e}({\bm{x}}^{l}_{t};d_{ffn}) (1) Here, R(\cdot) represents the routing function, and d_{ffn} is the intermediate size of the FFN, typically set to d_{ffn}=4d . 2.2 TopK Routing TopK routing ( Shazeer et al., 2017 ; Lepikhin et al., 2020 ; Fedus et al., 2022 ) is the most commonly used method for defining the routing function R(\cdot) . It introduces sparsity in the MoE computation by forcibly zeroing out smaller elements: \displaystyle R({\bm{x}}^{l}_{t})=\text{TopK}(\text{Softmax}({\bm{x}}^{l}_{t}{\bm{W}}_{l}),k) (2) where {\bm{W}}_{l}\in\mathbb{R}^{d\times E} is the router’s weight matrix, and \text{TopK}(\cdot,k) retains the top k largest values while setting the rest to zero. This mechanism allows for skipping the computation of the \text{FFN}_{e} functions corresponding to the zeroed-out R({\bm{x}}^{l}_{t})_{e} values in both the forward and backward passes. 3 Our Method: ReMoE 3.1 Motivation: From TopK to ReLU For a given token {\bm{x}}=(x_{e})_{e=1}^{E} after Softmax, TopK introduces a jump discontinuity at the k -th largest value, denoted as x_{[k]} , by zeroing out the values smaller than x_{[k]} . This can be expressed as: \displaystyle\text{TopK}({\bm{x}},k)_{e}=x_{e}\cdot\bm{1}\{x_{e}\geq t({\bm{x}},k)\},\quad t({\bm{x}},k)=x_{[k]} (3) Figure 2: Comparison between TopK and ReLU. where \bm{1}\{\cdot\} is the indicator function, returning 1 if the condition is met and 0 otherwise. As shown in Figure 2 , the jump discontinuity can be eliminated by setting the breakpoint t({\bm{x}},k)\equiv 0 , which actually corresponds to the ReLU function: \displaystyle\text{ReLU}({\bm{x}})_{e}=x_{e}\cdot\bm{1}\{x_{e}\geq 0\} (4) At a high level, ReLU improves upon TopK by aligning the breakpoints of all inputs and setting them to 0. This ensures that the output is continuous at 0, where the experts transition between active and inactive. As a result, the training pipeline becomes fully differentiable. 3.2 Differentiable ReLU Routing We define the ReLU routing function as follows: \displaystyle R({\bm{x}}^{l}_{t})=\text{ReLU}({\bm{x}}^{l}_{t}{\bm{W}}_{l}) (5) with (1-\frac{k}{E}) being the desired sparsity of ReLU, where k is the number of active experts and E is the total number of experts. This ensures that the computational cost remains equivalent to that of TopK routing. In vanilla TopK routers, the Softmax outputs sum to 1, representing the probabilities of selecting each expert, after which TopK eliminates those with lower probabilities. In contrast, ReLU