UMoE: Unifying Attention and FFN with Shared Experts

Yuanhang Yang, Chaozheng Wang, Jing Li · 2025

Read UMoE: Unifying Attention and FFN with Shared Experts by Yuanhang Yang, Chaozheng Wang, Jing Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07260.txt.

Introduction

Yuanhang Yang, Chaozheng Wang, and Jing Li merge the two big building blocks of transformer networks so they can share one pool of experts, in a 2025 paper. Mixture-of-experts designs sprinkle specialized sub-networks through a model to scale it cheaply. They work well in feed-forward layers, but bolting them onto attention layers has needed custom machinery and given weaker results. The team's insight is a reformulation showing attention hides a feed-forward-like structure inside it.

Their architecture, UMoE, builds expert layers for attention in the same mold as for feed-forward blocks. Parameters can then be shared between the two, cutting waste. In their tests the unified design beat earlier attention-expert attempts. The authors argue a single expert mechanism everywhere makes sparse models simpler to build and tune. They present the reformulation as the conceptual key.

Catalog record

CREATORYuanhang Yang, Chaozheng Wang, Jing Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UMoE: Unifying Attention and FFN with Shared Experts Yuanhang Yang, Chaozheng Wang, Jing Li Abstract Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward network (FFN) layers, recent studies have explored extending the MoE paradigm to attention layers to enhance model performance. However, existing attention-based MoE layers require specialized implementations and demonstrate suboptimal performance compared to their FFN-based counterparts. In this paper, we aim to unify MoE designs in attention and FFN layers by introducing a novel reformulation of the attention mechanism, that reveals an underlying FFN-like structure within attention modules. Our proposed architecture, UMoE, achieves superior performance through attention-based MoE layers while enabling efficient parameter sharing between FFN and attention components. 1 Introduction Scaling plays a crucial role in advancing the capabilities of large language models [ 1 , 2 , 3 ] . However, this scaling advantage comes with substantial computational costs, making continued scaling increasingly impractical. Sparse Mixture-of-Experts (MoE) architectures have emerged as a promising solution by selectively activating only a subset of model parameters—termed experts—for each input [ 4 , 5 , 6 , 7 ] . This approach effectively decouples model size from computational cost, enabling efficient scaling with minimal overhead. Recent work has demonstrated the effectiveness of MoE in Transformer architectures [ 8 , 9 , 10 , 5 , 11 ] , particularly when applied to feed-forward neural network (FFN) layers. Building on this success, several studies have explored extending MoE to attention layers [ 12 , 13 , 14 ] , indicating potential for performance gains through attention scaling. Despite the potential, we find that existing MoE attention layers demonstrate suboptimal performance compared to FFN-MoE approaches, when provided with similar computational and parametric budgets. This performance gap challenges the practical utility of attention-MoE architectures, as parameters allocated to scaling attention layers might be more effectively utilized for scaling FFNs instead. We identify two distinctions between attention-MoE and FFN-MoE implementations that likely account for the observed performance differential: (1) the different expert design between attention and FFN layers, and (2) attention-MoE’s necessity to compromise the expressiveness of vanilla attention mechanisms to accommodate sparse computation [ 12 ] . Motivated by these observations, we investigate a compelling question: can we reformulate attention to reveal an underlying structure compatible with the same expert design as FFN layers, without compromising the expressive power of the attention mechanism? This is a challenging question due to the inherent complexity of attention mechanisms, including multiple projections and softmax calculations, which fundamentally differ from the straightforward two-matrix multiplication pattern of FFNs. To bridge this structural gap, we reformulate the attention mechanism to reveal its underlying FFN-like structure. Our reformulation decomposes attention into two sequential operations: token mixing and token-wise expert processing. The token-wise expert processing, consisting of two consecutive matrix multiplications, can be implemented as an FFN with a small intermediate size. This implementation naturally aligns with recent advances in fine-grained FFN expert design [ 6 , 15 , 3 ] , enabling unified expert architectures and parameter sharing across both attention and FFN layers. Figure 1: Illustration of a UMoE layer, which incorporates MoE into both FFN and attention modules with shared experts. The primary distinction between attention-MoE and FFN-MoE lies in an additional token mixing operation. Figure 2: Two formulations of the multi-head attention mechanism. (a) Vanilla attention interleaves mixing operations with value and output projections. (b) Pre-mixing attention performs token mixing prior to projections. Based on this insight, we introduce UMoE, a unified MoE architecture that abstracts Transformer layers into three fundamental components: experts , token mixing operations , and routers , as shown in Fig. 2 . The experts, implemented as standard two-layer FFNs, serve as the primary components for token processing and knowledge storage. The token mixing operations facilitate contextual information exchange through weighted summation of tokens. Routers are employed to dynamically dispatch tokens to the most relevant experts to enable sparse computation. In UMoE, the distinction between the FFN and attention layers lies solely in the expert inputs: FFN layers process tokens independently, while attention layers process tokens simultaneously through weighted summation. This unified design not only simplifies the architecture but also enables parameter-efficient scaling through expert sharing between attention and FFN components. To evaluate the effectiveness of UMoE, we conduct extensive experiments across various model sizes and tasks, including pre-training and zero-shot evaluations. With the reformulated attention mechanism, the attention-based MoE layers of UMoE match or exceed the performance of previous FFN-based MoE layers. Moreover, by sharing parameters across attention and FFN modules, UMoE achieves superior performance in fully MoE architectures while maintaining the same parameter count. We also present a detailed routing analysis of UMoE, revealing expert specialization patterns across modules, with higher-ranked experts demonstrating interpretable attention patterns. Our code is available at https://github.com/ysngki/UMoE . 2 Related Work Sparse Mixture-of-Experts (MoE). Sparse Mixture-of-Experts (MoE) models have gained increasing attention for their ability to scale model capacity while maintaining computational efficiency [ 4 , 9 , 10 , 5 , 11 ] . The core component of these models is the sparsely activated MoE sub-layer, which selectively activates different parameter subsets for different inputs. In recent Transformer-based implementations, MoE architectures primarily replace feed-forward network (FFN) layers with MoE sub-layers. Each MoE layer consists of a collection of experts, denoted as \{\text{E}_{i}\}_{i=1}^{N} , where each expert \text{E}_{i} is implemented as an FFN. Tokens are routed to a subset of experts through a routing mechanism, with the top-k router [ 4 ] being the most prevalent approach. Despite advances in routing mechanisms [ 16 , 17 , 18 , 19 , 15 ] , the top-k router remains widely adopted due to its simplicity and robust performance [ 9 ] . For a given token \boldsymbol{x}\in\mathbb{R}^{d} , where d is the hidden dimension, and a trainable weight matrix \mathbf{W}_{r}\in\mathbb{R}^{N\times d} , the top-k router computes the probability distribution over experts as: \displaystyle\boldsymbol{p} \displaystyle=\texttt{softmax}(\mathbf{W}_{r}\boldsymbol{x}). (1) The set of top- k experts \mathcal{T} is then selected based on \boldsymbol{p} , where |\mathcal{T}|=k . Each expert processes the token independently and the final output of the MoE layer is computed as the weighted combination of these k experts’ outputs: \displaystyle\boldsymbol{y}=\sum_{i\in\mathcal{T}}\boldsymbol{p}_{i}\text{E}_{i}(\boldsymbol{x}), (2) where each expert is implemented as an FFN with two matrices and a non-linear activation function. MoE for Attention. Several recent approaches have explored extending the MoE paradigm to attention layers in Transformers [ 12 , 13 ] , with a primary focus on expert design. Because attention layers lack the consecutive matrix multiplication pattern found in FFNs, these approaches necessitate expert designs that differ from FFN-MoE models. The Mixture-of-Attention (MoA) [ 12 ] propose to conceptualize individual attention heads as experts, scaling attention layers by increasing the number of attention heads. However, introducing sparsity into attention layers presents a significant challenge: query vectors computed by a specific expert (or head) require corresponding key and value vectors from the same expert, necessitating identical expert activation across all tokens. To address this constraint, MoA implements distinct query and output projections per head while maintaining shared key and value projections across attention heads. SwitchHead [ 13 ] presents an alternative approach to implementing the

Read the complete text