Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Yizhou Liu, Jeff Gore · 2026
Read Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients by Yizhou Liu, Jeff Gore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25008.txt.
Introduction
Yizhou Liu and Jeff Gore argue, in a position paper, that the exponents in neural scaling laws are explained and the coefficients are where the action is. Scaling laws describe pretraining loss falling as a power law in time, model size, and compute. The authors claim the exponents come from generic mechanisms. A one third time scaling follows from the strong nonlinearity of softmax. Inverse width scaling follows from representational superposition. Inverse depth scaling follows from ensemble averaging across transformer layers.
Current large language models therefore sit in one universality class with fixed exponents. Coefficients are another matter: they are sensitive to data and architecture, and they fix practical quantities like optimal model shape and the compute optimal frontier. Understanding coefficients is thus the key to near term performance gains. Liu and Gore aim the field's theory effort at the part of the scaling law that engineers can change.
Catalog record
| CREATOR | Yizhou Liu, Jeff Gore |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients Yizhou Liu, Jeff Gore Abstract Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power laws are fixed by generic mechanisms: a one-third time scaling due to the strong nonlinearity of Softmax, an inverse width scaling due to representational superposition, and an inverse depth scaling due to ensemble averaging of Transformer layers. These mechanisms are robust to a wide range of data structures and architectural details, placing current large language models in a universality class with fixed exponents. The coefficients, however, are expected to be sensitive to data and architecture details, and directly determine practical quantities such as the optimal model shape and the compute-optimal frontier. We therefore argue that understanding the coefficients is the key to near-term performance improvements, and that a closer examination of the current universality class may reveal pathways to better universality classes. 1 Introduction Neural scaling laws, which describe pre-training loss decay with increasing model size, dataset size, and compute as power laws [ 25 , 28 , 44 , 26 ] , are among the key observations that led to the success of today’s large language models (LLMs). They describe and predict LLM performance [ 13 , 1 ] and have shaped modern training recipes [ 26 ] . However, the scaling laws remain largely empirical, leaving hope for further speedups and concerns about generalizability at larger scales. Extensive efforts have therefore been devoted to exploring the mechanistic origins of neural scaling laws. The mainstream view, summarized from a wide range of frameworks [ 48 , 27 , 38 , 47 , 40 , 4 , 10 , 11 ] , attributes the power-law loss scaling to some power-law structures in the data. The intuition is that there are various features, skills, or tasks to be learned. If these entities have power-law importance or frequency distributions, and the model learns the more important or frequent ones first, the loss will also decay as power laws whose exponents sensitively depend on the data power laws. Recently, a new view has arisen, arguing that the power-law loss can emerge without power laws in data [ 35 , 34 , 33 , 36 , 5 ] , whose exponents are determined by generic mechanisms robust to data details. A series of three papers [ 35 , 34 , 33 ] (see Figure 1 ) identified (i) a one-third time scaling due to Softmax nonlinearity in learning peaked distributions (e.g., next-token distributions and attention distributions) [ 35 ] , (ii) an inverse width scaling due to geometric interference of more features than available dimensions, i.e., superposition [ 34 ] , and (iii) an inverse depth scaling due to ensemble averaging of errors across Transformer layers [ 33 ] . The time scaling can be transferred to dataset size scaling because large-scale pre-training is close to online, one-epoch training [ 35 ] . Width and depth scaling laws lead to a one-third model size scaling at the optimal aspect ratio [ 36 ] , which is determined by the coefficients of the width and depth scaling laws. The optimal token-to-parameter ratio or compute-optimal frontier is also determined by the coefficients, at which the loss follows a one-sixth compute scaling [ 36 ] . The power-law exponents are fixed by mechanisms that depend only on high-level architectural and data properties, such as the usage of Softmax and the low entropy of target distributions, neither of which is likely to change. In this framework, a wide range of detailed data structures, e.g., feature frequency distributions, mainly affect the coefficients [ 34 ] rather than the exponents. The coefficients therefore warrant close attention, as they directly affect the optimal model shape and compute-optimal frontier and are likely to shift with modified architectures and training strategies. Figure 1: Loss can be decomposed into three leading terms, one due to imperfect training, one due to limited representation, and one due to restricted transformation power. For imperfect training, loss decays largely as a power law with exponent 1/3 due to the strong nonlinearity of Softmax in reaching peaked distributions, which is related to low-temperature expansion and universality [ 35 ] . The final value includes a term limited by representation, which is inversely proportional to width, due to geometric interference of more features than dimensions, i.e., superposition [ 34 ] . Another part is from limited transformation from input to output, which is inversely proportional to depth, due to averaging of errors across layers with similar functions [ 33 ] . Data structures, e.g., a wide range of feature frequency distributions, affecting the difficulty of learning, primarily change the coefficients [ 34 ] rather than the exponents. Based on the three papers [ 35 , 34 , 33 ] , we propose a hypothesis, neural scaling universality : a universality class of neural scaling laws is defined by the set of fixed exponents, where varying architecture or data details changes the coefficients of the power laws but not their exponents. Although more work needs to be done to further verify this hypothesis, evidence provided in [ 35 , 34 , 33 ] already makes it plausible. If this view is correct, it will have implications that differ significantly from the mainstream view. Given the potential implications, it is important to explore the consequences of neural scaling universality now, even before its full verification. We therefore argue that the current neural scaling laws have fixed exponents due to generic mechanisms, and the coefficients matter for LLM performance and design, and may be sensitive to architectural and data choices, requiring more attention and research. We hope that a deeper exploration in the current universality class, via a closer examination of the coefficients, can help to achieve better universality classes or to identify pathways beyond it. The rest of the paper is organized as follows. We introduce the mechanisms leading to fixed exponents with more details in Section 2 . We then discuss their implications, including the step-limited nature of pre-training and the importance of coefficients in determining optimal model shape and compute-optimal frontier, in Sections 3 , 4 , and 5 , respectively. We also discuss alternative views in Section 6 and conclude with discussions in Section 7 . 2 The neural scaling laws trilogy The pre-training loss first decays as a power law in training time due to the generic properties of Softmax [ 35 ] . Here \tau is the time of optimization dynamics, or dynamic time for short, related to the number of training steps via an integral over the learning rate schedule, \tau(t)\propto\int_{0}^{t}\eta_{t^{\prime}}\mathrm{d}t^{\prime}, (1) where \tau(t) is the dynamic time at step t , and \eta_{t^{\prime}} is the learning rate at step t^{\prime} . The proportionality constant may depend on the optimizer. When a model learns low-entropy or peaked distributions, Softmax becomes highly nonlinear, which produces power-law loss and gradients, and ultimately drives the loss to decay as L\sim\tau^{-1/3} . This is a form of universality, as the one-third exponent is insensitive to the details of the data distribution and arises from the loss