Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap

Feilong Liu · 2026

Read Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap by Feilong Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16349.txt.

Introduction

Liu presents a unified Jacobian-PCA-Grassmann tool for mapping expert focus in mixture-of-experts layers. Across trained MoE models, experts show strong function split, with near-zero cross-expert Jacobian match, while their routed codes occupy distinct but partly shared spaces. Function split and code overlap thus coexist rather than coincide. Test trials show top-k routing sharpens function split and space drift, while fully soft routing yields more tangled expert structure.

Catalog record

CREATORFeilong Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36881
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap Feilong Liu Abstract Mixture-of-Experts (MoE) architectures achieve scalable capacity through sparse routing, yet the geometric structure of expert specialization remains poorly understood. We introduce a unified Jacobian-PCA-Grassmann framework for analyzing MoE layers in both function space and representation space. Across pretrained MoE Transformers (Mistral, Qwen), we find a consistent structural asymmetry: experts exhibit strong functional decorrelation—consistently low (near-zero) cross-expert Jacobian alignment—while their routed representations occupy distinct but partially overlapping subspaces. This indicates that functional decorrelation and representation overlap coexist rather than coincide in MoE specialization. Controlled routing experiments further indicate that routing sparsity appears to be a key factor shaping this geometry: Top-k routing induces sharper functional separation and larger subspace divergence, whereas fully-soft routing yields more entangled expert structure. Together, these results suggest a geometric interpretation in which MoE layers may be viewed as implementing locally decorrelated operators over overlapping submanifolds on a shared representation manifold, and provide a general diagnostic framework for studying conditional computation in modern Transformer architectures. 1 Introduction Mixture-of-Experts (MoE) architectures have become a central mechanism for scaling Transformer models, enabling large parameter counts through sparse routing while maintaining fixed computational cost ( 19 ; 14 ; 7 ) . Despite their widespread adoption in modern language models, the geometric structure of expert specialization remains poorly understood. Existing analyses primarily focus on routing statistics, load balancing, or system-level behavior ( 14 ; 22 ; 5 ) , leaving open fundamental questions about how experts differ in function space and how routed tokens organize in representation space. While routing statistics provide a coarse view of expert behavior, a common intuition is that MoE layers partition the input distribution, with each expert acting as a local function approximator ( 10 ; 11 ) . However, empirical studies have shown that experts often operate on overlapping token regimes ( 13 ; 3 ) , and that specialization does not always correspond to clean input-space clusters. This raises a deeper question: what geometric structure does routing induce inside a pretrained MoE Transformer? To move beyond input-space intuitions, we study MoE specialization through a unified geometric lens 1 1 1 Code and data: https://drive.google.com/drive/folders/1X1nP8A2s62JrVu8Pb_Bgrw9cvP776iSl?usp=sharing . We introduce a Jacobian-PCA-Grassmann framework that jointly analyzes (i) functional geometry, via expert-local Jacobians (following prior Jacobian-based analyses of neural sensitivity; e.g., ( 17 ; 18 ) ), and (ii) representation geometry, via routed PCA and subspace distances (building on representation-space analyses in Transformers; ( 6 ; 20 ) ). Applying this framework to pretrained MoE Transformers—including Mistral and Qwen—we uncover a consistent structural asymmetry: experts exhibit strong functional decorrelation (near-zero cross-expert Jacobian alignment) while their routed representations occupy distinct but partially overlapping subspaces. This suggests that functional decorrelation and representation overlap coexist rather than coincide, providing evidence that is inconsistent with the assumption that MoE layers implement disjoint partitions of the input space. To probe the mechanism underlying this asymmetry, we conduct controlled routing experiments and find that routing sparsity appears to be a key factor shaping the MoE geometry. Top- k routing induces sharper functional separation and larger Grassmannian distances, whereas fully-soft routing yields more entangled expert geometry. These results indicate that routing sharpness modulates the degree of specialization, providing a mechanistic explanation for expert diversity in conditional computation ( 19 ; 7 ) . Taken together, these findings support a view of MoE layers as implementing a soft partitioning 2 2 2 We use “soft partitioning” informally to denote overlapping but distinct submanifolds, not a strict mathematical partition. of a shared representation manifold, where experts act as locally decorrelated operators over overlapping regimes. Beyond characterizing pretrained models, our framework provides a general diagnostic tool for analyzing expert specialization in conditional computation architectures. Why this matters for scaling. Modern large-scale MoE models—including Mixtral, DeepSeek-MoE ( ( 3 ) ), and Qwen-MoE—derive their efficiency from sparse routing, yet their internal specialization remains poorly understood. As these architectures scale to hundreds of experts and trillions of parameters, understanding how experts differentiate, when they collapse, and what geometric structure routing induces becomes critical for both training stability and inference reliability. Our results indicate that expert specialization is governed by a consistent geometric pattern—functional decorrelation paired with overlapping representation subspaces—and that routing sharpness directly controls this structure. This provides actionable insight for designing, scaling, and debugging MoE systems at frontier model sizes. 1.1 Contributions This paper makes the following contributions: • A unified geometric diagnostic framework for analyzing MoE specialization: We introduce a unified Jacobian—PCA—Grassmann framework to jointly characterize MoE specialization in function space (cross-expert Jacobian alignment) and representation space (routing-induced subspace structure via PCA and Grassmannian distance). This provides a systematic diagnostic for studying expert specialization beyond routing statistics or activation patterns. • Evidence of strong functional decorrelation across experts: Across pretrained MoE Transformers (Mistral, Qwen), we show that cross—expert Jacobian alignment is consistently near zero, indicating that experts implement locally distinct transformations rather than redundant variants of the same function. • Discovery of overlapping but distinct representation subspaces: Despite functional decorrelation, routed representations occupy distinct yet partially overlapping subspaces, as quantified by Grassmannian distances. This suggests a structural asymmetry, where MoE layers appear to separate functions more strongly than input clusters. • A geometric interpretation of MoE layers as soft partitioning of a shared manifold: These findings support a geometric interpretation of MoE layers as soft partitioning of a shared manifold, where experts act as locally decorrelated operators over overlapping regimes. 1.2 Relation to prior work Mixture—of—Experts routing and scaling: Early MoE work established sparse routing as an effective strategy for scaling neural networks ( 19 ; 14 ; 7 ) . Subsequent studies refined routing mechanisms, load-balancing objectives, and training stability ( 5 ; 12 ) . These works primarily analyze routing behavior and system-level performance, but do not characterize the geometric structure of expert specialization inside pretrained MoE Transformers. Expert specialization and redundancy: Beyond routing behavior, several works examine expert redundancy, merging, and pruning ( 15 ; 16 ; 21 ; 9 ) . These studies suggest that experts may be partially redundant, but rely on performance-based or parameter-based criteria. They do not analyze functional geometry (Jacobian structure) or representation subspaces, leaving an open question: when do experts represent genuinely distinct functions, and when do they reflect redundant parameterization? Jacobian analysis in deep networks: Orthogonal to MoE—specific studies, Jacobian—based analyses have been used to study sensitivity, curvature, and robustness in dense networks ( 2 ) . However, these analyses focus on monolithic architectures and do not address conditional computation. In particular, prior work does not examine cross - expert Jacobian alignment or functional decorrelation in MoE layers. Representation geometry and subspace structure: Complementary to functional analyses, representation-space studies examine representation anisotropy and layer geometry in Transformers ( 6 ; 20 , e.g.,) . Grassmannian distances ( 4 ; 1 ; 8 ) have been used to compare learned subspaces in multitask or modular networks, but not in the context of MoE routing. To our knowledge, no prior work jointly analyzes Jacobian geometry and routed representation subspaces in pretrained MoE Transformers. Our contribution in context: Prior work on MoE models has primarily examined routing behavior, load balancing, or expert redundancy, but has not analyzed

Read the complete text