On the Smallness of the Large Language Models Scaling Exponents
Sauro Succi, Peter V. Coveney, Alex Hansen · 2026
Read On the Smallness of the Large Language Models Scaling Exponents by Sauro Succi, Peter V. Coveney, Alex Hansen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24504.txt.
Introduction
Succi, Coveney, and Hansen ask why large language models improve so slowly with scale, and whether the slowness condemns the current path. Scaling laws tie model loss to size through a small exponent. Enormous growth thus buys modest gains, an unsustainable regime in energy terms. One comforting story blames a numerical bias. Fits neglect the loss floor reachable only with infinite data, the pedestal effect, artificially shrinking the exponent.
That comfort fails, the authors argue in 2026. Correcting for a nonzero asymptotic loss does not remove the unsustainability. The measured exponents are genuinely small. The energy trajectory of ever larger models stays implausible, regardless of fitting choices. A second thread connects exponent size to data smoothness, through an analogy with models of fluid turbulence, where roughness controls scaling. Succi's trio writes as physicists auditing an industry mantra. Bigger is better only while the exponents, and the power grid, cooperate.
Catalog record
| CREATOR | Sauro Succi, Peter V. Coveney, Alex Hansen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Smallness of the Large Language Models Scaling Exponents Sauro Succi, Peter V. Coveney, Alex Hansen Abstract We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (“pedestal effect”) does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence. 1 Introduction AI and most notably transformers-based large Language Models (LLM) have taken science and society by storm in the recent past [ 1 , 2 , 3 ] . Much of this explosive growth is driven by the “no-wall” finding, namely the fact that the learning capability of LLM-based chatbots keeps increasing with size, whence the mantra “bigger is better” which has driven much of the current leading-edge LLMs research. Let us recapitulate the main ideas behind the “no-wall” finding. The scaling law of LLMs is usually expressed in the form L(N)=A/N^{\alpha}\>, (1) where L is the loss function, a measure of the departure of the LLM outcome from the desired target, and N stands for data ( D ), number of parameters ( P ) and computational cost ( C ), respectively, each characterized by its constant prefactor A and exponent \alpha [ 4 , 5 ] . A positive value of the exponent indicates that the loss function decreases for increasing data size, so that using larger data sets leads to a closer match to the desired target. Formally, the so-called “wall” corresponds to the opposite regime, in which more data lead to a larger discrepancy between model and target, as formally reflected by a negative exponent. The LLMs exponents so far have remained positive across several decades, whence the enthusiastic claim that “there is no wall,” support a strategy oriented towards ever larger (and energy consuming) LLM applications [ 6 ] . While the accomplishment is remarkable under all scientific counts, many researchers have observed that the LLMs scaling exponents are very small, typically in the range 0.05\div 0.10 , pointing to a regime of “diminishing returns” for pre-trained LLMs. In a recent paper, two of the present authors have pointed out what “diminishing returns” really means in actual practice: an exponent 0.1 means that cutting the loss function down by a factor 2 requires 2^{10}=1024 more resources [ 7 ] . It was then argued that, while formally wall-free, such regime is simply unsustainable, whence the need of new directions paying more attention to physical insight and compliance with world models than muscular leverage of the number of parameters [ 8 , 9 , 10 , 11 ] . Among the critical feedback spawned by this simple observation, a recurrent one is that the loss function cannot be paralleled to numerical discretization errors, the reason being that the ML procedures do not necessarily aim at sending the loss function to zero, but are typically stopped before that limit is approached, usually in order to forestall overfitting and ensuing problems in generalizing, i.e., the ability to reproduce unseen targets [ 3 ] . In the following, we argue that such criticism does not change the conclusion on the unsustainability of LLMs scaling exponents. 2 The loss function as a pseudo-metric of accuracy The approximation error associated with a numerical method employing N degrees of freedom, say the grid discretization of a PDE, usually follows an asymptotic scaling relation of the form E(N)=A/N^{a}\;, (2) where a>0 is the order of accuracy and A a prefactor measuring the “critical” size N_{c}=A^{1/a} above which the error starts to show a power-law decay. Most grid methods work around a=2 , while stochastic particle methods, such as Monte Carlo, feature a\sim 1/2 . Note that a=2 means that reducing the error by a factor 2 takes just \sqrt{2} more resources ( N ), while with a=1/2 this number is 4 , which is generally regarded as poor convergence. The above relation encodes a basic requirement on any well-posed numerical scheme, called ”Consistency”, namely that upon sending N\to\infty the error should vanish, so as to reproduce the original target, generally the analytical solution of a continuum differential equation. Any non zero value E_{\infty}\equiv E(N\to\infty) (the ”pedestal”), or, worst, increasing error at increasing resolution (the ”wall”) is regarded as an anomaly, usually caused by some form of ill-posedeness of the numerical discretization, typically the breaking of a basic continuum symmetry by the discrete scheme. The above relation bears a direct analogy to the LLM scaling law [ 4 ] , whence the simple conclusion that an exponent \alpha=0.1 is simply unsustainable. Claiming that this analogy is flawed because machine learning practices do not concern themselves with the limit L\to 0 as N\to\infty is tantamount to saying that consistency, a prime requirement for any well-posed numerical method, is not relevant to machine learning as a scientific discipline. While it is not hard to see reasons why one might be happy to settle with “small enough” values of the loss function (the so-called early-stop empirical practice), it remains undeniable that dismissing consistency in favor of ambiguous “small enough” criteria, leaves much to be desired in terms of reliability of the methodology as a systematic scientific method. This is especially true with regard to the ability to generalize to unseen data, which is the essence of true learning as opposed to mere memorization of the training data. Yet, it is true that matching a given set of data is not equivalent to reproducing the solution of a continuum PDE, so let us accept the loss function as a sort of empirical pseudo -metric in no need to comply with the consistency requirement. In the following we shall argue that even this “lenient perspective” does not affect the claim of unsustainability made in [ 7 ] . The point is no longer consistency, but efficiency, namely how fast does the error decrease at increasing resources, which is precisely dictated by the actual value of the scaling exponent. 3 The pedestal effect Recently, a group of authors [ 12 ] have built on the so called Chinchilla scaling [ 13 ] to argue that LLM scaling exponents published by Anthropic and subsequent works are “biased” by the fact of ignoring a “pedestal” in the scaling laws, namely the fact that the loss function does not vanish in the ”continuum limit” N\to\infty . To appreciate the point, let us begin by casting the loss function in the form L(x)=L_{0}+(L_{1}-L_{0})\;x^{\alpha}\;, (3) where we have set x\equiv 1/N for the sake of convenience, so that L_{0}=L(x=0) is the continuum limit and L_{1}=L(x=1) is the large-scale limit N=1 . Note that \alpha>0 denotes the “no-wall” regime in which L grows with x=1/N . Clearly, the pedestal L_{0} introduces its own exponent, \alpha=0 , so that the “effective” exponent associated with the above relation must necessarily lie between 0 and \alpha , the precise form of the transition depending on the ratio p=L_{0}/L_{1} .