Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf, Samuel Tovey, Christian Holm · 2026

Read Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail by Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf, Samuel Tovey, Christian Holm (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31244.txt.

Introduction

Nikolaou and colleagues explain neural scaling laws through a new measure called spectral position. It tracks which eigenvalues of the neural tangent kernel drive loss reduction at each point in training. Learning moves from dominant modes into the spectral tail as training proceeds, they find, and bigger models push deeper into the tail than small ones. That extra reach, dubbed spectral reach, may explain why larger models end with lower loss. Feature learning enables the reach by amplifying gradients as training advances.

Catalog record

CREATORKonstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf, Samuel Tovey, Christian Holm
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101785
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31244

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf, Samuel Tovey, Christian Holm Abstract Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce spectral position : a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call spectral reach . This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design. 1 Introduction Figure 1 : Scaling laws and spectral position for Llama 2 language models on SimpleStories (left) and next-pixel prediction on CIFAR-5M (right). Following 22 , the compute is approximated as \text{FLOPs}=6ND , where N is the number of model parameters and D is the number of processed tokens/images. (top) Scaling laws emerge as power-law relations between model size and train loss. The dashed lines indicate power-law fits with exponents and R^{2} values. We provide test losses in Appendix D.1 . (bottom) Spectral position \chi_{\text{pos}} decreases throughout training and with model size. Larger models achieve lower values. The success of pre-training foundation models is largely attributed to the scaling paradigm: model quality improves predictably as training data, parameter count, and compute increase ( 21 ; 25 ; 22 ) . This is quantitatively expressed in neural scaling laws which describe power-law relationships between scale variables (model size, dataset size, and compute) and performance. In practice, these laws are central to pre-training large models: scaling curves are fit to computationally inexpensive runs and extrapolated to forecast the returns of additional compute. This guides choices such as model size, data allocation, and training duration ( 25 ; 22 ) . Motivated by their practical importance, there has been significant interest in explaining the mechanisms behind scaling laws ( 30 ; 5 ; 3 ; 40 ; 48 ; 52 ; 7 ) . However, many existing accounts rely on idealized assumptions or require kernel computations that are infeasible at modern scales, leaving open how scaling laws emerge in practical deep learning. To address this gap, we introduce a scalable diagnostic that reveals underlying spectral dynamics in large-scale training, and apply it to study the mechanisms of scaling in modern vision and language models. We derive the loss-network-position (LNP) decomposition, which factors the instantaneous (linearized) loss change into three interpretable components: a network scale \chi_{\text{net}} , a loss scale \chi_{\text{loss}} , and a scale-free spectral position \chi_{\text{pos}}\in[0,1] . Spectral position indicates which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction—close to 1 when dominant eigenmodes drive learning, close to 0 when learning focuses on the spectral tail. The factorization enables \chi_{\text{pos}} to be evaluated from per-sample gradients without explicit kernel construction, permitting measurement alongside training with modest overhead. To validate the framework, we study controlled random-feature models, where \chi_{\text{pos}} matches theory-predicted scaling relations. Applying this diagnostic to scaling experiments reveals a consistent pattern (Figure 1 ): as training progresses, spectral position decreases, and larger, better-performing models reach lower values. A lower spectral position means learning from progressively smaller eigenvalues in the eNTK spectrum; larger models therefore have greater spectral reach . This suggests why they achieve lower losses: they sustain learning from weak spectral signals that smaller models cannot access. A natural question is what enables spectral reach. Through linear-probing experiments, we find that feature learning plays a key role in spectral reach, and provide a mechanistic explanation: as spectral position decreases, feature-learning models adaptively amplify gradient magnitudes to sustain learning progress—a compensation mechanism unavailable to models with frozen representations. This opens practical avenues for enhancing spectral reach through targeted interventions, which we discuss in the Outlook. 2 Related Work Theoretical explanations of scaling (and the validation gap). A major line of work proposes explanations by analyzing simplified but tractable surrogates and connecting scaling exponents to spectral structure in data and kernels ( 30 ; 48 ; 29 ) . Random feature models (RFMs) are a prominent example and have become a testbed for deriving scaling behavior from first principles ( 30 ; 5 ; 3 ; 40 ) . In RFMs, a feature map is drawn at random and kept fixed, and only a linear readout is trained. This reduction to linear regression makes training and generalization analytically tractable, with closed-form expressions for many relevant quantities ( 56 ) . RFMs reproduce power-law scaling curves, when the kernel is assumed to exhibit a power-law spectral structure ( 30 ; 5 ) . At the same time, because representations are random and frozen, RFMs cannot capture the feature learning and hierarchical abstraction formation central to deep networks ( 56 ; 7 ) . Empirically, scaling curves in deep networks resemble those predicted by kernel surrogates, however, including feature learning and the nature of the power-law into the picture remains an open question ( 3 ; 7 ) . 6 studied dataset size scaling in kernel models, demonstrating that increasing data enables learning from modes with smaller eigenvalues in the kernel spectrum. 3 extended this idea by arguing that finite model capacity similarly induces an effective cutoff in the small-eigenvalue tail of the empirical kernel spectrum: Modes beyond this cutoff are effectively inaccessible, and the remaining (unlearned) tail governs the residual error. Thus, scaling improves performance by pushing this cutoff deeper, enabling the model to leverage progressively weaker spectral modes. While this reasoning was proposed for RFMs, it is unclear how feature learning interacts with this picture and whether this mechanism even applies to practical deep learning scenarios. NTK and scalable measurements. The neural tangent kernel (NTK) offers a principled lens on learning dynamics: under suitable initialization, the infinite-width limit yields a linearized training dynamics around the initial parameters, enabling kernel-based analysis ( 24 ; 27 ) . In this (“lazy training”) regime, the NTK is effectively constant ( 11 ) ; by contrast, at finite width the NTK can evolve, reflecting representation learning and feature drift ( 17 ; 55 ) . To highlight the contrast between these regimes, we refer to the latter as the empirical NTK (eNTK). However, direct computations of the eNTK remain limited to relatively small datasets and models. Even with efficient estimators ( 37 ) and sketching techniques ( 20 ; 58 ) , the computational and memory costs are typically infeasible in the regimes where scaling laws are measured. Our work targets this measurement bottleneck. The loss-network-position (LNP) decomposition can be computed from per-sample gradients ( 57 ) and enables tracking eNTK quantities at scales where explicit kernel computations are impractical. Empirical diagnostics of learning

Read the complete text