Neural Scaling Laws for Deep Regression
Tilen Čadež, Kyoung-Min Kim · 2025
Read Neural Scaling Laws for Deep Regression by Tilen Čadež, Kyoung-Min Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10000.txt.
Introduction
Tilen Cadez and 1 co-author hunt for scaling laws in deep regression in a 2025 machine learning paper. Scaling laws tie error to the size of a model or its data. They steer the huge budgets behind language models. Far less is known for regression. There the answer is a number, not a word. The pair pick a physics task to test the idea. They gauge the traits of twisted layered magnets from data.
Three kinds of network were trained. The set held plain, residual, and vision transformer designs. Loss fell as a power of data size. It also fell as a power of model size. The steepness of that fall sat between one and two. It shifted with the trait being gauged. It also shifted with the design. Such steep laws promise large gains from more data.
Catalog record
| CREATOR | Tilen Čadež, Kyoung-Min Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10000
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Scaling Laws for Deep Regression Tilen Čadež, Kyoung-Min Kim Abstract Neural scaling laws—power-law relationships between generalization errors and characteristics of deep learning models—are vital tools for developing reliable models while managing limited resources. Although the success of large language models highlights the importance of these laws, their application to deep regression models remains largely unexplored. Here, we empirically investigate neural scaling laws in deep regression using a parameter estimation model for twisted van der Waals magnets. We observe power-law relationships between the loss and both training dataset size and model capacity across a wide range of values, employing various architectures—including fully connected networks, residual networks, and vision transformers. Furthermore, the scaling exponents governing these relationships range from 1 to 2, with specific values depending on the regressed parameters and model details. The consistent scaling behaviors and their large scaling exponents suggest that the performance of deep regression models can improve substantially with increasing data size. I Introduction Figure 1: Neural scaling laws in a parameter estimation model. Panels (a)–(c) show the geometric mean of mean squared error (MSE) test loss in the prediction of three magnetic parameters ( \theta , J , D ), respectively, as a function of training dataset size. Each marker corresponds to a different architecture: fully connected networks (FCNs), residual network (ResNet-18), and vision transformer (ViT). The two FCNs, indicated by red and yellow squares, have different numbers of neurons per hidden layer ( n_{n}=4 and n_{n}=512 , respectively) with the same number of hidden layers ( n_{l}=3 ). In each panel, dashed lines represent power-law fits of the form \epsilon\sim N_{D}^{-\alpha_{D}} , where \epsilon is the loss, N_{D} is the training dataset size, and \alpha_{D} is the scaling exponent. All data are plotted on a log-log scale, with dataset sizes ranging from 224 to 114688. Data points for the two largest model sizes in panel (a) are omitted. Deep learning has established a new data-driven programming paradigm, where data determines the algorithms used to solve specific problems. This approach often provides substantial advantages over traditional logic-driven programming, particularly in addressing intricate scientific and real-world problems, where the required logic of algorithms is unknown or too complex. Generally, the performance of deep learning models depends on the quantity and quality of the data used to train deep neural networks. Researchers have identified various techniques for enhancing model performance, either by utilizing various types of regularization and batch normalization from the model side, or by pre-processing a given dataset Lathuilière et al. 2020 or through data augmentation Wang and Perez 2017 ; Steiner et al. 2022 from the data side. The latter potential can be systematically examined using “generalization error scaling laws,” which describe the power-law relationship between generalization errors and dataset size, model size and the computing time used for the training. Early theoretical studies sought to derive generalization error scaling laws by solving statistical models ichi Amari 1993 ; Amari et al. 1992 , while recent research has focused on empirical investigations in the context of classification models Bailly et al. 2022 ; Hestness et al. 2017 ; Rosenfeld et al. 2020 ; Henighan et al. 2020 ; Mühlenstädt and Frtunikj 2024 ; Meir et al. 2020 ; Figueroa et al. 2012 ; Cho et al. 2016 ; Sun et al. 2017 ; Figueroa et al. 2012 ; Sharma and Kaplan 2022 ; Alabdulmohsin et al. 2022 ; Bahri et al. 2024 and language models Kaplan et al. 2020 ; Alabdulmohsin et al. 2022 ; Hoffmann et al. 2022 ; Alabdulmohsin et al. 2022 ; Sharma and Kaplan 2022 ; Kaplan et al. 2020 ; Hutter 2021 ; Alabdulmohsin et al. 2022 ; Bahri et al. 2024 . In particular, the exploration of the scaling laws in large language models found that the sizes of datasets and models are crucial for model performance, while other features such as network structure have secondary impacts Kaplan et al. 2020 ; Bahri et al. 2024 . These insights have significantly contributed to the recent success of large language models. In contrast to substantial advancements in classification and language models, the scaling laws for deep regression models remain largely understudied, despite recent emerging research Lin et al. 2024 . Gaining a deeper understanding of these laws could provide valuable insights for developing deep regression models with predictable performance. Such insights are particularly crucial in scientific disciplines, where computational and data resources are often limited and optimizing resource usage is important. Figure 2: Dataset and regression model used in the scaling analysis of generalization error. (a) Dataset: simulated magnetic domain images generated via atomistic spin simulations of twisted bilayer CrI 3 . Red (blue) indicates +1 (-1) of the local out-of-plane normalized magnetization. Only images of the top layer are shown. (b) Regression model: the parameter estimation model predicts the magnetic Hamiltonian parameters (\theta,J,D) for twisted bilayer CrI 3 from the input magnetic domain images. In this work, we empirically investigate the scaling laws of generalization error in deep regression, using a parameter estimation model that predicts magnetic Hamiltonian parameters from input magnetic domain images [Fig. 2 ]. By training networks on datasets of varying sizes or on multiple networks with a fixed dataset, we examine how the mean squared error (MSE) test loss in the model’s predictions scales with the training dataset size. We find that the test loss follows a power-law scaling behavior, expressed as \epsilon\sim N_{D}^{-\alpha_{D}} , where \epsilon is the MSE test loss, N_{D} is the dataset size, and \alpha_{D} is the corresponding scaling exponent [Fig. 1 ]. Notably, the scaling behavior often spans more than three orders of magnitude in dataset size. Furthermore, the scaling exponents \alpha_{D} range from approximately 1 to 2—much larger than those typically observed in classification or language models, which are often below 1 Bailly et al. 2022 ; Hestness et al. 2017 ; Rosenfeld et al. 2020 ; Henighan et al. 2020 ; Meir et al. 2020 ; Mühlenstädt and Frtunikj 2024 ; Figueroa et al. 2012 ; Cho et al. 2016 ; Sun et al. 2017 ; Figueroa et al. 2012 ; Sharma and Kaplan 2022 ; Alabdulmohsin et al. 2022 ; Kaplan et al. 2020 ; Hoffmann et al. 2022 ; Hutter 2021 , with a rare case slightly above 1 Bahri et al. 2024 . This suggests that the model’s performance improves more rapidly with increased data or model complexity than typically expected. Additionally, increasing model complexity significantly reduces the MSE test loss, showing a similar power-law scaling \epsilon\sim N_{M}^{-\alpha_{M}} with the model size N_{M} and the scaling exponent \alpha_{M} [Fig. 6 ]. These findings demonstrate that deep regression models can achieve substantial improvements in accuracy as data and model sizes grow. II Result II.1 Neural network realizations Figure 3: Statistics of MSE test loss across different realizations of FCNs. (a)–(d) Distributions of test loss across multiple network realizations for the regression of J , with bars indicating the number of networks N_{\mathrm{bin}} within each loss range. Each panel corresponds to a specific combination of the number of neurons per hidden layer n_{n} and dataset size N_{D} , with a fixed number of hidden layers n_{l}=3 : panels (a) and (c) show results for n_{n}=4 and