Geometric Regularization in MoEs: The Disconnect Between Weights and Activations

Hyunjun Kim · 2026

Read Geometric Regularization in MoEs: The Disconnect Between Weights and Activations by Hyunjun Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00457.txt.

Introduction

Hyunjun Kim reports a negative result about a popular trick for mixture-of-experts models. These models route each input to a few specialist sub-networks, and a geometric penalty called orthogonality loss is meant to push the specialists apart so they diversify. Kim applied that penalty and measured what actually changed in weights and in activations.

Across seven penalty strengths, weight overlap did not fall and sometimes grew sharply, while activation overlap stayed high no matter what. Effects on language modeling quality were small and inconsistent across three text datasets, and weight and activation measures barely correlated. Kim concludes that weight-space penalties neither reach their geometric goal nor reliably help, so they are a poor tool for expert diversity.

Catalog record

CREATORHyunjun Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00457

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometric Regularization in MoEs: The Disconnect Between Weights and Activations Hyunjun Kim Abstract Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high ( ∼ \sim 0.6) regardless of regularization, and effects on performance are inconsistent —marginal improvement on WikiText-103 ( − 0.9 % -0.9\% ), slight degradation on TinyStories ( + 0.9 % +0.9\% ), and highly variable results on PTB (std > > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation ( r = − 0.293 r=-0.293 , p = 0.523 p=0.523 ) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity. 1 Introduction Mixture-of-Experts (MoE) models scale efficiently by activating only a subset of parameters per input Shazeer et al. 2017 ; Fedus et al. 2022 . A common assumption is that expert representations should be orthogonal to minimize interference Chen et al. 2022 ; Chen et al. 2023 . This intuition stems from linear algebra: orthogonal vectors are maximally distinguishable and their outputs do not interfere when combined. Hypothesis. Orthogonality regularization should improve expert diversity and reduce perplexity. Finding. It does not—and is unreliable. Across three datasets (TinyStories, WikiText-103, PTB), geometric regularization yields inconsistent results: marginal improvement on WikiText-103 ( -0.9\% ), slight degradation on TinyStories ( +0.9\% ), and high variance on PTB (std > 1.0). Why? We identify a Weight-Activation Gap : weight-space orthogonality (MSO \approx 10^{-4} ) does not translate to activation-space orthogonality (MSO \approx 0.6 ). Across 7 regularization strengths, we find no significant correlation between weight and activation overlap ( r=-0.293 , p=0.523 ), indicating that weight geometry and functional orthogonality are largely independent. Contributions. 1. We show that orthogonality regularization fails to reduce weight MSO—it actually increases it by up to 114%—and yields inconsistent effects on loss: marginal improvement on WikiText-103 ( -0.9\% ), slight degradation on TinyStories ( +0.9\% ), and high variance on PTB (std > 1.0). 2. We identify a weight-activation disconnect: activation overlap is \sim 1000 \times higher than weight overlap, with no significant correlation ( r=-0.293 , p=0.523 , n =7). 3. We demonstrate that weight-space regularization is an unreliable optimization target—it neither achieves its geometric goal nor reliably improves performance. 2 Method Orthogonality Loss. For expert weight matrices \{W_{i}\}_{i=1}^{N} , we define: \mathcal{L}_{\text{orth}}=\sum_{i<j}|\langle\tilde{W}_{i},\tilde{W}_{j}\rangle|^{2} (1) where \tilde{W}_{i}=\text{vec}(W_{i})/\|\text{vec}(W_{i})\| is the normalized flattened weight vector. This loss encourages orthogonality among expert representations and is added to the language modeling objective with weight \lambda . Mean Squared Overlap (MSO). We measure geometric diversity using: \text{MSO}=\frac{2}{N(N-1)}\sum_{i<j}|\langle\tilde{W}_{i},\tilde{W}_{j}\rangle|^{2} (2) Lower MSO indicates more orthogonal (diverse) experts. We compute MSO for both weights and activations . Activation MSO. For co-activated experts producing outputs \{h_{i}\} , we compute: \text{MSO}_{\text{act}}=\mathbb{E}_{x}\left[\frac{2}{k(k-1)}\sum_{i<j\in\mathcal{S}(x)}\left(\frac{\langle h_{i},h_{j}\rangle}{\|h_{i}\|\|h_{j}\|}\right)^{2}\right] (3) where \mathcal{S}(x) is the set of k selected experts for input x . This measures functional similarity between expert outputs on actual inputs. 3 Experiments Setup. We train NanoGPT-MoE ( \sim 130M parameters, 8 experts, 6 layers, top-2 routing) on TinyStories ( Eldan and Li 2023 ) for 10K iterations with AdamW ( Loshchilov and Hutter 2019 ) (lr= 5\times 10^{-4} , \beta_{1} =0.9, \beta_{2} =0.95, weight decay=0.1). Each MoE layer contains 8 experts with hidden dimension 512 and intermediate dimension 2048. TinyStories experiments use 5 random seeds (42, 123, 456, 789, 1337). Implementation Details. We regularize the up-projection weights ( W_{\text{up}}\in\mathbb{R}^{d_{\text{ffn}}\times d_{\text{model}}} ) of each expert. Each weight matrix is flattened and L2-normalized before computing pairwise inner products. The \lambda sweep uses 7 values: \{0,0.001,0.005,0.01,0.05,0.1,0.2\} . MSO is computed per layer and averaged across all 6 MoE layers. Activation MSO is computed on the post-gating expert outputs for the top-2 selected experts, unweighted by gating scores. We do not use auxiliary load balancing loss. 3.1 Weight MSO Under Regularization Table 1 reveals a surprising finding: orthogonality regularization does not reduce weight MSO—it increases it. Despite the loss explicitly penalizing expert overlap, the final weight MSO rises with \lambda , suggesting the regularization interferes with natural training dynamics rather than enforcing orthogonality. Method Weight MSO \Delta Baseline 5.43\times 10^{-4} — + Orth ( \lambda =0.001) 7.52\times 10^{-4} +39\% + Orth ( \lambda =0.01) 1.16\times 10^{-3} +114\% Table 1: Orthogonality regularization increases weight MSO, contrary to its intended effect. Baseline weights are already near-orthogonal. 3.2 Perplexity Does Not Improve Despite the explicit regularization objective, perplexity improvements are not statistically significant (Table 2 ). Method Val PPL \Delta % p-value Baseline 5.94\pm 0.08 — — + Orth ( \lambda =0.001) 6.00\pm 0.32 +0.9\% 0.727 Table 2: Orthogonality regularization does not improve perplexity ( p=0.727 , paired t-test, n =5 seeds). Variance increases from 0.08 to 0.32. Why Are p-values High? The high p-value ( p=0.727 ) reflects both minimal effect size and increased variance. The baseline shows low std (0.08), while \lambda =0.001 increases variance to 0.32—a 4 \times increase that destabilizes training. The slight PPL increase (+0.9%) is dwarfed by this variance, indicating the regularization adds noise without benefit. 3.3 The Weight-Activation Gap Table 3 reveals the core finding: weight and activation geometry are fundamentally decoupled. \lambda Weight MSO Act. MSO Ratio 0 (baseline) 5.43\times 10^{-4} 0.572 1053 \times 0.001 7.52\times 10^{-4} 0.581 773 \times 0.01 1.16\times 10^{-3} 0.577 496 \times 0.1 2.04\times 10^{-3} 0.593 290 \times 0.2 2.78\times 10^{-3} 0.564 203 \times Table 3: Activation MSO ( \sim 0.57) remains constant while weight MSO increases with \lambda . Pearson r=-0.293 , p=0.523 ( n =7), indicating no significant correlation. Correlation Analysis. Figure 1 visualizes the disconnect: as \lambda increases, weight MSO rises (regularization is applied), but activation MSO remains flat at \sim 0.57. Across 7 regularization strengths, we find Pearson r=-0.293 ( p=0.523 , 95% CI: [-0.857,0.590] )— not statistically significant . This confirms that weight and activation geometry are independent. Figure 1: Weight-Activation Gap. Weight MSO responds to regularization; activation MSO does not. No significant correlation ( r=-0.293 , p=0.523 ). 3.4 Cross-Dataset Validation To test whether our findings generalize beyond TinyStories, we evaluate orthogonality regularization on WikiText-103 ( Merity et al. 2016 ) and Penn Treebank ( Marcus et al. 1993 ) —two standard benchmarks with different characteristics (Table 4 ). Dataset Tokens Base Orth \Delta WikiText-103 118M 3.76_{\pm.02} 3.73_{\pm.05} -0.9\% TinyStories 2.1M 5.94_{\pm.08} 6.00_{\pm.32} +0.9\% PTB 1.2M 6.17_{\pm.95} 5.74_{\pm 1.11} -7.0\%^{*} Table 4: Cross-dataset validation ( n =3 seeds for WikiText-103/PTB, n =5 for TinyStories, \lambda =0.001). ∗ PTB shows high variance; effect direction unreliable. Dataset-Dependent Effects. Multi-seed experiments reveal mixed effects. On WikiText-103 (118M tokens), orthogonality regularization yields a small, consistent improvement ( -0.9\% ). However, on PTB (1.2M tokens), results are highly variable across seeds (std \sim 1.0), making conclusions unreliable. This high variance suggests that the effectiveness of geometric regularization may depend on dataset-seed interactions rather than dataset characteristics alone. Interpretation. The high variance on PTB (std \sim 1.0) compared to WikiText-103 (std \sim 0.05) may reflect dataset-scale effects. Smaller datasets may exhibit more seed-dependent expert specialization patterns, leading to unstable outcomes. Regardless of direction,

Read the complete text