Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Nandan Kumar Jha, Brandon Reagen · 2026

Read Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws by Nandan Kumar Jha, Brandon Reagen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21803.txt.

Introduction

Nandan Kumar Jha and Brandon Reagen show the optimizer is a key axis of representation scaling in language models, not a fixed training detail. From eigenspectra of feed-forward representations, they find the same transformer architecture follows different spectral scaling laws under different optimizers. AdamW shows weak hard-rank scaling on rare-token representations, while Muon reaches linear scaling there, more than doubling the scaling exponent. Matched validation loss does not imply matched representation structure, and optimizer shifts often exceed architectural effects, spurring optimizer-architecture co-design.

Catalog record

CREATORNandan Kumar Jha, Brandon Reagen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101970
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws Nandan Kumar Jha, Brandon Reagen Abstract Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers . Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling ( β \beta =0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling ( β \beta = 1.02 ) in the same regimes, a 2.3 × 2.3\times increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure . Hard–soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer–architecture co-design. : https://optimizer-scaling-laws.github.io 1 Introduction Classical scaling-law studies showed that language-model loss follows predictable power-law trends with model size, training data, and compute [ 1 , 2 ] . This resource-centric view has made scaling actionable: given a compute budget, one can estimate how to allocate resources across parameters and data. Yet a central component of training remains largely outside this framework—the optimizer. Growing evidence suggests that optimizers do more than affect convergence speed; they also shape the representations learned by a model through implicit inductive biases [ 3 , 4 ] . Recent work has begun to incorporate optimizer effects into loss-level scaling. [ 5 ] show that loss-scaling exponents can remain shared across optimizers while multiplicative efficiency factors differ. This provides a useful abstraction for predicting validation loss, but it leaves open a representation-level question: if two optimizers exhibit similar loss scaling, do they also learn similar internal representations? If matched loss can arise from different representation geometries, then optimizer choice is not merely an efficiency knob. It is a design axis that determines how model capacity is structured across eigenmodes, allocated across token regimes, and realized during training. (a) (b) Figure 1 : Spectral scaling exponents depend on optimizer choice: Soft (left) and hard spectral rank (right) as a function of FFN width in GPT-2 160M. AdamW exhibits the largest hard-soft asymmetry ( \Delta_{1,2}=0.37 ), indicating concentrated eigenspectra. Muon and Dion (1/2) reduce this asymmetry to \Delta_{1,2}\approx 0.14 . Moreover, hard-rank scaling exhibits stronger dependence on optimizer choices. We study this question through the eigenspectra of feed-forward network (FFN) representations [ 6 ] . FFNs provide a natural setting for this analysis: in standard Transformer architectures, they account for roughly two-thirds of model parameters [ 7 ] , and their expansion–nonlinearity–compression structure exposes a spectrally measurable latent space. This structure allows us to ask how efficiently added FFN width is converted into utilized spectral capacity. Building on prior work [ 8 ] , which established spectral scaling laws for FFN utilization under a fixed optimizer, we ask whether these laws are invariant to optimizer choice or instead depend on the architecture–optimizer pair. We compare AdamW [ 9 ] , Muon [ 10 , 11 ] , NorMuon [ 12 ] , and rank-constrained Dion variants [ 13 ] , measuring how spectral capacity is realized during training and how it scales with FFN width. Holding the architecture and width schedule fixed, changing the optimizer alone yields markedly different spectral scaling laws. Figure 1 illustrates this phenomenon. We measure FFN capacity using soft spectral rank, which captures entropy-weighted spectral spread, and hard spectral rank, which is more sensitive to concentration in dominant eigenmodes [ 8 ] . The two metrics respond differently to optimizer choice: soft rank grows substantially with width across all optimizers, with scaling exponents clustered in a narrow range \beta_{\text{soft}}\in[0.66,1.01] , whereas hard rank is strongly optimizer-dependent, spanning \beta_{\text{hard}}\in[0.29,0.82] . AdamW exhibits weak hard-rank scaling ( \beta =0.29), while Muon achieves near-linear scaling ( \beta =0.82), under identical architecture and training data. This result changes how spectral scaling should be interpreted. In prior work, the gap between soft-rank and hard-rank scaling appeared to be a stable property of the Pre-LN Transformer architecture [ 8 ] . We show that this hard–soft asymmetry is itself optimizer-dependent. AdamW exhibits the largest asymmetry ( \Delta_{1,2}=0.37 ), while Muon-style optimizers substantially reduce this gap; in particular, Muon and Dion( 1/2 ) both reach \Delta_{1,2}\approx 0.14 . Hence, added FFN width is not automatically converted into usable capacity. The optimizer helps determine whether extra dimensions become dominant representational directions or remain diffuse spectral mass. This spectral divergence is not apparent from validation loss alone. AdamW configurations can match matrix-aware optimizer variants in perplexity under extended training, while their representation spectra remain structurally distinct. Thus, matched loss does not imply matched representation scaling , and neither learning-rate tuning nor extended training closes this gap. Therefore, optimizer shapes the geometry of the learned representation, not just the convergence speed. The optimizer effect is also structured across the token distribution. Language data follows a Zipfian distribution, and LLMs are known to struggle disproportionately with rare and long-tail knowledge [ 14 ] . We therefore stratify token representations by frequency and measure spectral scaling separately across frequency regimes. AdamW exhibits especially weak hard-capacity scaling on rare-token representations, while the largest AdamW-to-Muon scaling gain appears in the mid-frequency regime. This shows that optimizer geometry changes not only aggregate representation capacity, but also how capacity is allocated across the token-frequency distribution. Finally, we compare optimizer-induced effects against architectural interventions in attention rank [ 15 ] and positional encoding [ 16 , 17 ] . Optimizer-induced spectral shifts often dominate or reshape these architectural effects, increasing per-head attention rank produces smaller spectral changes than switching optimizers, RoPE removal yields optimizer-dependent redistribution, and orthonormal optimizers enable partial PostLN configurations to reach useful perplexity where AdamW fails. These results show that architectural capacity is not realized independently of optimization, architectural changes are expressed through optimizer geometry. Contributions. Our contributions are as follows: 1. Optimizer-induced spectral scaling laws. We show that the same Transformer architecture realizes substantially different FFN spectral-capacity scaling laws depending on the optimizer. Hard–soft rank asymmetry is optimizer-dependent, exposing spectral scaling as a property of the architecture–optimizer pair rather than architecture alone. Rényi-entropy spectral analysis further confirms that the optimizer-induced differences persist across concentration regimes. 2. Matched loss \neq matched geometry. We demonstrate that optimizer-induced spectral differences are not explained by learning-rate tuning, convergence speed, or final validation loss: configurations with matched perplexity can exhibit distinct spectral geometries. 3. Frequency- and update-rank-dependent capacity allocation. We show that optimizer-induced spectral scaling varies across token-frequency regimes, with mid- and low-frequency tokens

Read the complete text