Unified Neural Scaling Laws

Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish · 2026

Read Unified Neural Scaling Laws by Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26248.txt.

Introduction

Ethan Caballero and three colleagues propose a Unified Neural Scaling Law, a single form modeling how deep-network scores vary as parameters, data, training steps, inference steps, compute, and hyperparameters change together. It covers vision, language, math, and reinforcement learning tasks, upstream and downstream. Compared with other scaling formulas, it extrapolates considerably more accurately across architectures and tasks. The law thus offers one form for many scaling dimensions at once.

Catalog record

CREATOREthan Caballero, Priyank Jaini, David Krueger, Irina Rish
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE153889
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Neural Scaling Laws Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish Abstract We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL) ) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, number of inference steps, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. 1 Introduction Training today’s state-of-the-art neural networks requires significant amounts of computational resources and training data. Given a wide range of available methods and architectures to choose from, accurate forecasting of their performance is essential for selecting those that are likely to perform best at scale, especially since the top-performing methods at smaller scales often fail to maintain their performance at larger scales ( 59 ; 62 ) . Moreover, accurate forecasting of neural network behaviors at scale is critical not only for identifying the top-performing approaches but also for ensuring AI safety, as predicting the emergence of novel capabilities at scale is essential for responsible development and deployment of advanced AI systems. This realization motivated the study of neural scaling laws ( 16 ; 27 ; 48 ; 32 ; 68 ; 1 ; 11 ; 6 ; 4 ; 12 ) which aim to predict the behavior of large-scale models as the amount of compute, data, and model parameters increases. Clearly, the accuracy, as well as the confidence of predictions made by neural scaling laws can only increase (or remain the same) as a larger number of relevant predictors are included, due to the standard conditional entropy inequality, H(Y|{\bf X})\leq H(Y) , where {\bf X} is the vector of predictive variables and Y is the performance evaluation metric. Namely, as the number of predictive variables X_{i},i=1,...,m increases, the conditional entropy H(Y|(X_{1},...,X_{m})) can only decrease (or remain the same). Ultimately, to obtain the maximal achievable reduction in the entropy of Y , one would need to identify the set of all possible X_{i} that are causally related to Y , and develop a complete model P(Y|{\bf X}) that can serve as a “unified functional form” of neural network behavior(s) at scale. To address this need for a (more) unified functional form, we present Unified Neural Scaling Laws (UNSL) , a functional form that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Additionally, this functional form accurately models and extrapolates multivariate scaling behavior that other functional forms are incapable of expressing such as the nonmonotonic transitions present in the scaling behavior of overfitting and hyperparameters (such as learning rate and standard deviation of weights at initialization) that have a nonmonotonic relationship with the performance evaluation metric. 2 The Functional Form of Unified Neural Scaling Laws Figure 1: An illustration of a Unified Neural Scaling Law (UNSL) (dark solid lines) with two input dimensions, x_{1} and x_{2} ; the central and the right plots show the projections on each of the input dimensions, respectively. In this particular example, an UNSL contains 3 hyperbreaks highlighted by brighter dotted lines - orange, yellow, and green. The green hyperbreak is created by a non-bottleneck component. The orange hyperbreak is created by an x_{1} bottleneck component. The yellow hyperbreak is created by an x_{2} bottleneck component. See Section 2 for detailed explanation of hyperbreaks. Let y denote a performance evaluation metric of interest, e.g. prediction error or cross-entropy, "upstream" (i.e., measured on the validation dataset from the pretraining data distribution) or "downstream" (i.e., measured on new data and/or tasks that the model does not encounter during pretraining), and let (x_{i})_{i=1}^{m}\in\overline{\mathbb{R}}_{>0}^{\,m} denote a tuple of m quantities that can be viewed as predictors of y , e.g. number of model parameters, training dataset size, number of training steps, number of inference steps, and values of various hyperparameters. We present the following general functional form of a unified neural scaling law (UNSL): y=a_{0}+\left(\bigg(Q(3)+\underbrace{\left(Q(S+4)+{a_{1}}^{-1}\right)^{-1}}_{\text{oppositional force of overfitting}}\bigg)^{-1}+{a_{2}}^{-1}\right)^{-1}, (1) where Q is defined as follows: \begin{split}Q(&q)=\left(\left(R(q)\right)^{-1}+{a_{q}}^{-1}\right)^{-1}+\underbrace{\sum_{s=1}^{S}\left(R(q+s)+{a_{q+s}}^{-1}\right)^{-1}}_{\text{oppositional force of hyperparameters}},\end{split} (2) where R is defined as follows: \begin{split}R(&r)=\underbrace{K(\,U_{r},\,\,n_{r_{0}},\,\,r\!\cdot\!(m\!+\!1))}_{\text{non-bottleneck component}}\,+\,\underbrace{\sum_{\mathclap{\kern 2.56073ptt\in T_{r}}}K(\{t\},\,\,n_{r_{t}},\,\,r\!\cdot\!(m\!+\!1)\!+\!t)}_{\text{bottleneck components}},\\ &\hskip-10.81204pt\text{where }U_{r},T_{r}\subseteq\{1,\ldots,m\},\end{split} (3) and where K is a Multivariate Broken Neural Scaling Law (MBNSL) , defined as follows: K(M,n,k)=b_{k}\cdot\left(\prod_{i\in M}x_{i}^{-c_{i_{0_{k}}}}\right)\prod_{j=1}^{n}\left(1+\left(\frac{\prod_{\begin{subarray}{c}i\in M\end{subarray}}x_{i}^{c_{i_{j_{k}}}}}{d_{j_{k}}}\right)^{\left|\frac{1}{f_{j_{k}}}\right|}\right)^{-f_{j_{k}}}. (4) The parameters whose values are unknown constants that must be estimated by fitting the above functional form to the ( x_{1}...x_{m},y ) data points are all those whose base is one of these: a,b,c,d,f . The purpose of the variables i , j , k , q , r , s , t is indexation. n is a bound of a product operator; as a result, each of n_{r_{0}} and n_{r_{t}} implicitly is a bound of a product operator. S is a bound of a summation operator. M\subseteq\{1,\ldots,m\} . M is a product index set; as a result, U_{r} implicitly is a product index set. T_{r} is a summation index set. K,Q,R are functions and the contents of the parentheses in K(\cdot),Q(\cdot),R(\cdot) are arguments of those functions. Whenever an argument of K , Q , or R is obtained via addition(s) and/or multiplication(s), the sole reason that those additions and multiplications occur is to cause each instantiation of K to have a unique value for k . Equations 1 , 2 , 3 , and 4 are interpreted as follows. We use the term multi-log space to refer to the (m+1)-dimensional space obtained by applying the logarithmic transformation to each of every dimension ( x_{1}...x_{m},y ). Equation 4 is an extension of the univariate broken neural scaling law (BNSL) of 12 to multivariate settings. When |M|=1 , its expressivity is identical to the univariate broken neural scaling law functional form (with the performance limit term subtracted out) from 12 . When |M|>1 , Equation 4 defines a sequence of n+1 smoothly connected hyperplanes in multi-log space. Constant n corresponds to the number of (smooth) “hyperbreaks” (i.e. transitions) between n+1 consecutive hyperplanes in multi-log space; the dimensionality of each hyperplane is |M| , and the dimensionality of each hyperbreak is |M|-1 . When n=0 , Equation 4 becomes b_{k}\prod_{i\in M}x_{i}^{-c_{i_{0_{k}}}} . In multi-log space, the initial exponent for each input dimension (c_{i_{0_{k}}})_{i\in M} corresponds to the gradient of the first hyperplane with respect to the input dimensions (x_{i})_{i\in M} . In multi-log space, b_{k} corresponds to the offset of the output of Equation 4 . The j-th hyperplane smoothly transitions to the (j+1)th hyperplane at the values of (x_{i})_{i\in M} for which this equality is true: d_{j_{k}}=\prod_{i\in M}x_{i}^{c_{i_{j_{k}}}} . The j-th exponent for each input dimension (c_{i_{j_{k}}})_{i\in M} multiplied by \operatorname{sign}(f_{j_{k}}) corresponds to the change in gradient (with respect to the input dimensions (x_{i})_{i\in M} ) between the j-th hyperplane and the (j+1)th hyperplane in multi-log space. Constant f_{j_{k}} represents the

Read the complete text