ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping[1ex] Tamoghna Chattopadhyay, Sophia I Thomopoulos, Shahin Nazarian, Paul Thompson, Paul Bogdan[1ex] University of Southern California[1ex] · 2025

Read ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization by Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping[1ex] Tamoghna Chattopadhyay, Sophia I Thomopoulos, Shahin Nazarian, Paul Thompson, Paul Bogdan[1ex] University of Southern California[1ex] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10971.txt.

Introduction

Anzhe Cheng, Shukai Duan, and their colleagues present ERMoE, a sparse model design that sends each input to only a few expert parts. In older designs the router that picks experts can clash with how each expert works inside, which wastes experts and slows training. Extra balancing losses help, but they can blur what each expert is good at.

ERMoE rebuilds each expert in a learned set of axes and scores matches by how well the input lines up with those axes. Routing then follows content, stays steadier, and needs no balancing loss. Tests on image sorting and image-text search gave top scores with more even expert loads.

Catalog record

CREATORAnzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping[1ex] Tamoghna Chattopadhyay, Sophia I Thomopoulos, Shahin Nazarian, Paul Thompson, Paul Bogdan[1ex] University of Southern California[1ex]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10971

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping[1ex] Tamoghna Chattopadhyay, Sophia I Thomopoulos, Shahin Nazarian, Paul Thompson, Paul Bogdan[1ex] University of Southern California[1ex] Abstract Mixture-of-Experts (MoE) architectures expand model capacity by sparsely activating experts, but suffer from two core challenges: misalignment between router logits and each expert’s internal structure leads to unstable routing and expert underutilization, and load imbalances create straggler bottlenecks. Standard solutions, such as auxiliary load-balancing losses, can reduce load disparities but often weaken expert specialization and hurt downstream performance. To address these issues, we propose ERMoE, a sparse MoE transformer that reparameterizes each expert in a learned orthonormal eigenbasis and replaces learned gating logits with an Eigenbasis Score—the cosine similarity between input features and an expert’s basis. This content-aware routing ties token assignments directly to experts’ representation spaces, inherently stabilizing utilization and promoting interpretable specialization without sacrificing sparsity. Crucially, ERMoE eliminates the need for explicit balancing losses and avoids the interfering gradients they introduce. We demonstrate that ERMoE achieves state-of-the-art accuracy on ImageNet classification and cross-modal image-text retrieval benchmarks (e.g., COCO, Flickr30K), while naturally producing flatter expert load distributions. Moreover, a 3D MRI variant (ERMoE- ba ) improves brain age prediction accuracy by over 7% and yields anatomically interpretable expert specializations. ERMoE thus introduces a new architectural principle for sparse expert models, directly addressing core routing instabilities and enabling improved performance with scalable, interpretable specialization. 1 Introduction Deep learning has emerged as a cornerstone of modern artificial intelligence, delivering striking advances across core vision problems 4 ; 25 ; 18 . A dominant driver of this progress has been a straightforward scaling rule: train larger models on more data to steadily reduce error 19 . Yet the same paradigm carries clear limits: ever-larger dense networks imply escalating compute, memory, and energy budgets that are difficult to reconcile with practical training and deployment constraints 39 ; 38 . To respond to this tension, Mixture-of-Experts (MoE) 40 architectures were introduced. By activating only a small subset of experts per input, MoE layers decouple parameter count from per-token FLOPs, preserving throughput while expanding capacity. Despite these potential, MoE systems repeatedly encounter structural drawbacks. The most significant issue is the misalignment between the router and the experts. With noisy or imbalanced gating, routers may route tokens to suboptimal experts, leading to incorrect predictions and weakening expert specialization 50 ; 22 . Direct evidence comes from R2-T2 27 , which shows that test-time re-routing, without updating experts, recovers accuracy by moving tokens away from the originally chosen experts toward better ones, implying the initial router frequently selects the wrong expert. Moreover, routing fluctuations sometimes occur 6 . Under noisy gates, the same input is being sent to different experts during training, while only one is used at inference, which wastes updates and reflects unstable, error-prone assignment under noisy gates. At the representation level, misaligned routing precipitates representation collapse: tokens cluster around expert centroids and experts become redundant, harming performance and blurring roles 9 ; 5 . Beyond the misalignment of router experts, routing collapse further constrains MoE progress: token-choice gates amplify a “rich-get-richer” dynamic, in which a few experts attract disproportionate traffic, starving others and creating straggler bottlenecks where the busiest expert dictates end-to-end latency, while overall capacity remains idle 53 ; 29 . In response, mainstream practice adds an auxiliary load-balancing loss (LBL) 44 or switches to balance-by-construction routers 53 ; 41 . While these strategies improve utilization, they do not address content alignment and can introduce side effects: large LBL coefficients inject interference gradients that impair task performance. Multiple analyses report that strong balancing fosters over-uniform routing and expert overlap, weakening the very divide-and-conquer specialization that MoE seeks to realize 16 ; 34 . Figure 1 : Overview of the Eigen-Reparameterized Mixture-of-Experts(ERMoE) framework. a) A ViT backbone tokenizes the image; at each ERMoE block, the router computes an eigenbasis score per expert, selects the top- k experts whose scores exceed a threshold T , and aggregates their outputs with score-normalized weights for the classification head. b) The details of the Eigenbasis score. For a given expert, the input token and its attention-weighted context are projected into that expert’s eigenbasis; the score is the cosine similarity between the two projections. c) a 3D ViT tokenizes volumetric T1 scans; routing operates over region experts and free experts, and their weighted outputs drive the brain-age estimator. To address these issues, we propose the Eigen-Reparameterized Mixture-of-Experts (ERMoE) - a novel MoE whose experts are parameterized in an orthonormal eigenbasis and whose gate scores experts based on feature–basis alignment rather than free router logits. Specifically, each expert’s weight is reparameterized into an eigenbasis, and the router computes the energy captured by each expert’s basis, routing based on these alignment scores. This design ties routing to the experts’ intrinsic representational subspaces, eliminating the need for auxiliary load-balancing losses while directly targeting the root cause of misassignment. By enforcing near-orthogonality within experts, ERMoE reduces feature redundancy and makes expert directions separable, thereby stabilizing specialization and mitigating representation collapse. Using alignment instead of loss-regularized logits improves content-aware assignment without sacrificing sparsity or introducing interference gradients from balancing terms. Additionally, to test whether ERMoE’s eigenbasis routing maintains strong router–expert alignment beyond natural images, we instantiate a 3D neuroimaging variant, ERMoE- ba , that scales the 2D ViT backbone in ERMoE to a 3D ViT for brain-age prediction from T1 MRI—an application where brain age is a clinically relevant biomarker. The primary contributions of this research can be summarized as follows: • We introduced Eigen-Reparameterized Mixture-of-Experts(ERMoE), a novel sparse architecture that parameterizes expert weights in an orthonormal eigenbasis. This fundamentally alters the gating mechanism, replacing standard router logits with a content-aware score based on feature-based alignment. • Critically, ERMoE’s design resolves the core structural flaws in standard MoEs without compromising sparsity. It eliminates the need for auxiliary load-balancing losses (LBL), thereby avoiding LBL-induced interference gradients and over-uniform expert assignments. • We proposed ERMoE- ba , a ERMoE variant specifically designed for 3D neuroimaging. We demonstrate its effectiveness on the challenging task of brain-age prediction from T1 MRI, a clinically relevant application where standard transformers are computationally prohibitive. 2 Related Work Vision MoEs. V-MoE 35 introduced sparse experts to ViT backbones, demonstrating that conditional computation can match or exceed dense models in large-scale image recognition with comparable inference costs. In parallel, single-gated MoE variants 36 revisit a simplified training recipe that stabilizes routing via a base branch, asynchronous pipelines, and clustering-based initialization. Beyond transformers, DeepMoE 45 explores deep dynamic routing and per-example channel sparsification inside CNNs, demonstrating that conditional computation benefits extend to convolutional pipelines as well. Routing beyond standard token-choice. A line of recent work targets the misalignment and instability by modifying the routing rather than the experts. Soft MoE 33 makes the layer fully differentiable, replacing hard top- k with soft assignments to reduce token drops while maintaining sparse compute at inference. Meanwhile, optimal-transport routers formulate token–expert matching as a balanced assignment, improving utilization and stability in vision MoEs 42 ; 29 . Another thread, BASE layers 26 , enforces balance by construction via a linear assignment of tokens to experts. While these approaches relieve load

Read the complete text