NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

Nandan Kumar Jha & Brandon Reagen · 2026

Read NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks by Nandan Kumar Jha & Brandon Reagen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06922.txt.

Introduction

Jha and Reagen introduce NerVE, a framework for spectral study of feed-forward networks in language models. Feed-forward networks take most weights, yet their complex motion remains poorly known. NerVE tracks spectrum motion cheaply with four metrics. They are spectral entropy, participation for dimension count, early enrichment for top-heavy, and Jensen-Shannon gap for shift patterns. Their key insight is that feed-forward nonlinear parts recycle variance across modes, setting dimension use, while optimizer geometry guides that recycling.

Catalog record

CREATORNandan Kumar Jha & Brandon Reagen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54766
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06922

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks Nandan Kumar Jha & Brandon Reagen Abstract We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model’s generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-error. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language tasks, driven in part by advances in transformer-based architectures. While much emphasis has been devoted to understanding attention mechanisms and token-wise interactions, the role of feed-forward networks (FFNs), particularly their nonlinear components, remains underexplored, despite FFNs dominating both the parameter budget and computational footprint of transformer-based models ( 13 ; 9 ) . Despite their apparent simplicity, FFNs perform high-dimensional nonlinear transformations that regulate information flow by reorganizing, compressing, and propagating the information extracted by attention modules across layers. Understanding how these transformations evolve and interact with architectural design choices remains a fundamental open question. One challenge in interpreting FFNs is the absence of systematic and efficient tools for characterizing how latent representations are structured and transformed by nonlinear activations. FFN transformations unfold in a high-dimensional latent space which is far less accessible for direct visualization and probing compared to multi-head attention. Prior work, 25 used attention maps to study the input-contextualization effect of FFNs, and 3 characterized FFN geometry through piecewise-affine spline partitions. Neither lens reveals how nonlinearity redistributes variance, nor captures the rich spectral structure inherent in these transformations. Figure 1: NerVE quantifies nonlinear eigenspectrum dynamics in FFNs of GPT-2. FFN nonlinearity (GELU) regulates information flow by reinjecting variance, reactivating under-utilized directions (post-activation SE \uparrow and PR \uparrow ), and flattening the eigenspectrum, less top-heavy (post-activation EEE \downarrow ). The JS heatmap shows a depth-localized transition band where redistribution is strongest. To this end, we introduce NerVE, a unified, online, and memory-efficient framework for analyzing FFN latent geometry through the eigenspectrum analysis. NerVE summarizes pre- and post-activation spectra using four scale-invariant, distribution-aware metrics: spectral entropy (dispersion vs uniformity), participation ratio (effective latent dimensionality), eigenvalue early enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shift). From a methodological standpoint, these metrics span a broad theoretical range and expose the complementary facets of the eigenspectrum that any single scalar would obscure, thereby enabling continuous tracking of latent geometric dynamics. Spectral Entropy (SE) captures the uniformity of variance distribution ( 8 ) , and Participation Ratio (PR) reflects the geometric notion of effective dimensionality, indicating how many directions meaningfully contribute to total variance ( 11 ) . Unlike SE and PR, Eigenvalue Early Enrichment, which quantify the top-heaviness, can distinguish the eigenspectrum utilizing different fractions of the latent space ( 38 ) . Finally, the Jensen-Shannon (JS) divergence provides an information-theoretic distance measure between two eigenspectra ( 32 ) , quantifying distributional shifts of variance. We apply this framework across a diverse range of architectural settings, including LayerNorm placements: PreLN, PostLN, and MixLN ( 30 ) ; normalization-free variants ( 21 ) ; FFN weight-geometry constraints ( 39 ; 50 ) , and hyperspherical constraints ( 34 ) ; positional encoding scheme ( 54 ) ; and the optimizer choices, including Adam ( 24 ; 36 ) , Muon ( 22 ) , Dion ( 1 ) , Adafactor ( 52 ) , and SGD; see Table 1 . Across settings, a clear pattern emerges: FFN nonlinearities do not merely rescale the activations, they actively reinject the variance across eigenmodes and reawakens the inactive directions in high-dimensional latent space. As shown in Figure 1 , the post-activation spectra in GPT-2 consistently show increases in SE and PR, and decreases in EEE, while JS heatmaps reveal depth-localized transition bands where redistribution is strongest. This highlight the active role of FFN nonlinearities in regulating information flow and latent geometry that downstream layers further exploit. Contributions: Our contributions can be summarized as follows: 1. Conceptual. We demonstrate that FFN nonlinearities do not simply rescale activations but actively reorganize eigenspectra, reinjecting variance into under-utilized directions. Moreover, optimizer geometry modulates the extent of variance reinjection, altering the role of FFN nonlinearity from repair (recovering spectral collapse) to refinement (stabilizing a well-conditioned spectrum). 2. Framework. We introduce NerVE, a lightweight and memory-efficient methodology for online tracking of FFN eigenspectrum dynamics, using four distribution-aware, scale-invariant metrics. 3. Diagnostic. We show that architectural (normalization layers, activation functions, gating, weight geometry, positional encodings), and optimizer (AdamW, Muon, Dion, Adafactor, SGD) choices imprint distinct spectral signatures in FFNs, which can be used for diagnosing the model behaviors. 4. Empirical. We validate NerVE on GPT-2 and LLaMA models (71M to 1.3B) trained from scratch on CodeParrot, OpenWebText, FineWeb, and C4 datasets; extend to the non-transformer MLP-Mixer (B/16) on CIFAR-100, confirming cross-architecture generality; and perform extensive robustness studies across normalization variants, optimizer family, and token positions. Table 1: Summary of key NerVE findings per experimental axis. Experimental Axis Key Findings Activation (GELU vs ReLU) Similar trend, distinct dynamics; GELU explores broader subspace (§ 3.1 ) Norm-free models GELU exhibits spectral inertia; ReLU compensates LayerNorms (§ 3.2 ) FFN weight geometry Spectral dynamics matter; performance tracks sustained flattening (§ 3.3 ) Norm placement (Pre/Post/Mix) PreLN: best return-on-width; PostLN: diminishing spectral returns (§ 3.4 ) Positional encoding RoPE prevents mid-to-deep spectral collapse (depth utilization \uparrow ) (§ 3.6 ) Optimizer (AdamW vs Muon) Repair or refinement; performance follows mid-layer capacity trends (§ 3.7 ) Token positions Position-uniform early on; depth shift capacity to later-tokens (Appendix J ) Non-transformer (MLP-Mixer) Core findings generalize beyond transformer architecture (Appendix I ) LayerNorm vs RMSNorm Robust; placement and FFN activation choices matter more (Appendix K ) 2 NerVE: A Principled Framework for Eigenspectrum Analysis Notations. Let L be the number of layers, d the embedding dimension, D the FFN hidden dimension, B the batch size, S the context length, and \Sigma the FFN (pre/post-activation) covariance matrix. 2.1 Formulation of Eigenspectrum-Based Framework To understand how information is structured and propagated through FFN latent space, we analyze: (1) variance distribution and its impact on effective dimensionality; (2) how nonlinearity within a layer reshapes this distribution; (3) how these patterns evolve across layers and training. The NerVE framework (Figure 10 in Appendix A.1 ) consists of four main components: i) activation collection, (ii) covariance matrix computation, (iii) eigendecomposition, and (iv) spectral metrics calculation. Activation collection. For a FFN with (non-gating) architecture \text{FFN}(x)=W_{\text{down}}\sigma(W_{\text{up}}x+b_{1})+b_{2} ,

Read the complete text