EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts

Anzhe Cheng Shukai Duan Shixuan Li Chenzhong Yin Mingxi Cheng Shahin Nazarian Paul Thompson Paul Bogdan · 2026

Read EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts by Anzhe Cheng Shukai Duan Shixuan Li Chenzhong Yin Mingxi Cheng Shahin Nazarian Paul Thompson Paul Bogdan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12137.txt.

Introduction

Anzhe Cheng, Shukai Duan, Shixuan Li, and their co-authors fix two flaws in mixture-of-experts models. Such models split work among expert sub-nets to save compute. One flaw is load imbalance, where a few experts get most work in a rich-get-richer loop.

The second flaw is that experts learn near the same traits, which wastes their point. Their method, named EMoE in the work, routes work by an eigenbasis guide. Better routing could keep experts both busy and distinct. That would make large models cheaper to run well. More even load also cuts wait time in large runs.

Catalog record

CREATORAnzhe Cheng Shukai Duan Shixuan Li Chenzhong Yin Mingxi Cheng Shahin Nazarian Paul Thompson Paul Bogdan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts Anzhe Cheng Shukai Duan Shixuan Li Chenzhong Yin Mingxi Cheng Shahin Nazarian Paul Thompson Paul Bogdan Abstract The relentless scaling of deep learning models has led to unsustainable computational demands, positioning Mixture-of-Experts (MoE) architectures as a promising path towards greater efficiency. However, MoE models are plagued by two fundamental challenges: 1) a load imbalance problem known as the“rich get richer” phenomenon, where a few experts are over-utilized, and 2) an expert homogeneity problem, where experts learn redundant representations, negating their purpose. Current solutions typically employ an auxiliary load-balancing loss that, while mitigating imbalance, often exacerbates homogeneity by enforcing uniform routing at the expense of specialization. To resolve this, we introduce the Eigen-Mixture-of-Experts (EMoE), a novel architecture that leverages a routing mechanism based on a learned orthonormal eigenbasis. EMoE projects input tokens onto this shared eigenbasis and routes them based on their alignment with the principal components of the feature space. This principled, geometric partitioning of data intrinsically promotes both balanced expert utilization and the development of diverse, specialized experts, all without the need for a conflicting auxiliary loss function. Our code is publicly available at https://github.com/Belis0811/EMoE . † † address: University of Southern California, Los Angeles, CA, USA 1 Introduction Deep learning has emerged as a cornerstone of modern artificial intelligence, driving state-of-the-art advances in computer vision and beyond [ 2 , 9 , 5 ] . This success is largely attributed to a straightforward scaling principle: training larger models on more data consistently leads to improved performance [ 6 ] . However, this paradigm has led to an exponential and unsustainable growth in model size and computational cost. The compute required to train frontier AI models has been doubling approximately every six months since 2010, far outpacing improvements in hardware and incurring prohibitive financial and environmental costs [ 17 , 11 , 16 ] . One promising solution to this scaling challenge is the Mixture-of-Experts (MoE) architecture [ 18 ] , which uses conditional computation to activate only a small subset of specialized ”expert” sub-networks for each input. By doing so, MoE models can dramatically increase their total parameter count while keeping the inference cost (FLOPs) nearly constant, effectively decoupling model capacity from runtime cost. Figure 1 : Overview of EMoE. (a) Training schematic: the eigenbasis is recomputed from routers with frozen ViT weight (b) EMoE-augmented ViT block: input tokens convert into eigenbasis tokens, then pass through LayerNorm and multi-head self-attention, followed by a MoE feed-forward layer. Despite their potential, MoE models often suffer from degenerative behaviors that limit their practical effectiveness [ 3 , 13 ] . The most prominent issue is routing collapse – a“rich-get-richer” effect where the gating mechanism routes a disproportionate number of inputs to a few popular experts [ 21 , 10 ] . This severe load imbalance creates computational bottlenecks: throughput is dictated by the busiest expert, leaving much of the model’s capacity underutilized. The common remedy is to add an auxiliary Load-Balancing Loss (LBL) [ 19 ] to the training objective, encouraging a more uniform distribution of inputs across experts. However, recent analyses have shown that enforcing uniform routing in this way can conflict with the goal of expert specialization [ 4 , 12 ] . By forcing the router to prioritize even load over content, LBL may cause experts to learn redundant, homogeneous representations (with pairwise similarities reported as high as 99%). This undermines the intended divide-and-conquer principle of the MoE approach. In essence, current MoE training techniques face a trade-off: they achieve either balanced loads or highly specialized experts, but not both simultaneously. To resolve this tension between computational balance and representational diversity, we propose the Eigenbasis-Guided Routing for Mixture-of-Experts (EMoE). EMoE fundamentally reimagines the routing mechanism by eliminating the conventional learned gating network and its auxiliary loss. Instead, routing decisions are made by projecting input features onto an eigenbasis derived from the data (the principal components of the feature distribution). This geometric partitioning of the feature space naturally assigns coherent clusters of inputs to different experts. By design, our approach promotes both balanced expert utilization and specialized expert representations without the need for a conflicting auxiliary objective. We evaluate EMoE on the ImageNet image classification benchmark, comparing it against state-of-the-art MoE models including V-MoE [ 13 ] , a single-gated MoE variant [ 14 ] , and DeepMoE [ 20 ] . EMoE attains accuracy on par with these models while achieving significantly better load balance across experts, demonstrating that balanced routing and expert specialization can be achieved simultaneously for large-scale image classification. 2 Method Overview of the architecture. Our proposed method integrates a sparse MoE layer into the ViT backbone with eigenbasis guide routers. Each MoE branch consists of an Eigen Router and a set of 8 expert MLP networks. The branch operates in parallel with the ViT’s feed-forward layer; the patch token outputs are fed into the Eigen Router, which dynamically selects one expert MLP per token. The chosen expert transforms the token’s features, and the expert’s output is added back to the token’s representation. In practice, the router selects top expert per token, sends the token through that expert’s MLP, optionally scales by the gate score, and then adds the result back through the residual stream. Importantly, only the patch tokens pass through the MoE branch, i.e., the class token bypasses the router and continues along the standard transformer pathway unchanged. Fig. 1 a illustrates this architecture, where each ViT block in the augmented set has a parallel EMoE route comprising the eigenbasis-guided router and multiple expert MLPs. Eigen Router. The Eigen Router is a novel gating mechanism that routes tokens to experts based on the covariance structure of their features. As shown in Fig. 1 b, at each forward pass, the router collects the patch token features output by the ViT block. Conceptually, given a batch of images, let H\in\mathbb{R}^{N\times D} be the matrix of patch token features(with N tokens across the batch and D dimensional embeddings per token), and the router maintains a learnable eigenbasis matrix \mathbf{U}\in\mathbb{R}^{D\times r} ( r\ll D ) that is optimized to align with the dominant eigenspace of the empirical covariance \mathbf{C}=\frac{1}{N}H^{\top}H . In other words, \mathbf{U} spans the top- r principal components of the patch tokens’ feature distribution. We enforce this through an orthonormality regularizer that keeps \mathbf{U} close to orthonormal, which encourages \mathbf{U}^{\top}\mathbf{U}\approx\mathbf{I}_{r} . This learned eigenbasis is updated continuously by reorthognal the input tokens during each step of training, allowing the router to adapt to the feature statistics of each batch. Given the basis \mathbf{U} , the router projects each patch token’s feature vector h_{t}\in\mathbb{R}^{D} into the r -dimensional subspace. Let z_{t}=h_{t}^{\top}\mathbf{U}\in\mathbb{R}^{r} denote the coordinates of token t in the eigenbasis. We compute the normalized energy of h_{t} along each basis direction j as the squared projection, normalized to form a distribution: e_{t,j}=\frac{z^{2}_{t,j}}{\sum_{k=1}^{r}z^{2}_{t,k}+\epsilon}, (1) where \epsilon is a small constant for numerical stability. The vector e_{t}=(e_{t,1},\dots,e_{t,r}) of dimension r represents the fraction of the token’s feature energy captured by each eigenvector. Intuitively, e_{t} tells us how “aligned” the token is with each principal direction of variance. Next, the router transforms these per-token basis-coefficient vectors into expert logits for gating. We associate each

Read the complete text