How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines

Ayan Sengupta *, Yash Goel *, Tanmoy Chakraborty · 2025

Read How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines by Ayan Sengupta *, Yash Goel *, Tanmoy Chakraborty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12051.txt.

Introduction

Ayan Sengupta, Yash Goel, and Tanmoy Chakraborty survey what fifty-plus studies say about growing neural networks wisely, in this 2025 paper. Scaling laws, the power-law links between model size, data, compute, and performance, reshaped how big models are designed, promising compute-optimal recipes. Reality has since complicated the picture.

Sparse and mixture-of-experts models, retrieval-augmented systems, and multimodal architectures all bend away from the classic curves. Vision, reinforcement learning, and fine-tuning each scale by their own habits. The survey lays out the theory, the empirical record, and the practical guidance, then dwells on open problems: data efficiency, the scaling of inference-time compute, and architecture-specific limits. Their verdict is measured: scaling laws are an excellent guide that must be adapted, not obeyed blindly, and adaptive strategies tuned to each setting are the way forward.

Catalog record

CREATORAyan Sengupta *, Yash Goel *, Tanmoy Chakraborty
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12051

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines Ayan Sengupta *, Yash Goel *, Tanmoy Chakraborty Abstract Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computational resources. Early research established power-law relationships in model performance, leading to compute-optimal scaling strategies. However, recent studies highlighted their limitations across architectures, modalities, and deployment contexts. Sparse models, mixture-of-experts, retrieval-augmented learning, and multimodal models often deviate from traditional scaling patterns. Moreover, scaling behaviors vary across domains such as vision, reinforcement learning, and fine-tuning, underscoring the need for more nuanced approaches. In this survey, we synthesize insights from over 50 studies, examining the theoretical foundations, empirical findings, and practical implications of scaling laws. We also explore key challenges, including data efficiency, inference scaling, and architecture-specific constraints, advocating for adaptive scaling strategies tailored to real-world applications. We suggest that while scaling laws provide a useful guide, they do not always generalize across all architectures and training strategies. * 0 0 footnotetext: Equal contribution 1 Introduction Figure 1: Papers surveyed under different categories. The detailed paper list is provided in Table 9 of Appendix B . Scaling laws have become a fundamental aspect of modern AI development, especially for large language models (LLMs). In recent years, researchers have identified consistent relationships between model size, dataset volume, and computational resources, demonstrating that increasing these factors leads to systematic improvements in performance. These empirical patterns have been formalized into mathematical principles, known as scaling laws , which provide a framework for understanding how the capabilities of neural networks evolve as they grow. Mastering these laws is crucial for building more powerful AI models, optimizing efficiency, reducing costs, and improving generalization. Figure 2: A taxonomy of neural scaling laws. The study of neural scaling laws gained prominence with the foundational work of Kaplan et al. (2020) , who demonstrated that model performance follows a power-law relationship with respect to size, data, and compute. Their findings suggested that larger language models (LMs) achieve lower loss when trained on sufficiently large datasets with increased computational resources. Later, Hoffmann et al. (2022) refined these ideas, introducing the notion of compute-optimal scaling, which revealed that training a moderate-sized model on a larger dataset is often more effective than scaling model size alone. However, recent studies Muennighoff et al. (2023) ; Caballero et al. (2023) ; Krajewski et al. (2024) have challenged the universality of these laws, highlighting cases where sparse models, mixture-of-experts architectures, and retrieval-augmented methods introduce deviations from traditional scaling patterns. These findings suggested that while scaling laws provide a useful guide, they do not always generalize across all architectures and training strategies. Despite the growing importance of scaling laws, existing research remains fragmented, with limited synthesis of theoretical foundations, empirical findings, and practical implications. Given the rapid evolution of this field, there is a need for a structured analysis that consolidates key insights, identifies limitations, and outlines future research directions. While theoretical studies have established the mathematical principles governing scaling, their real-world applications, such as efficient model training, optimized resource allocation, and improved inference strategies, are less explored. To address this gap, we reviewed over 50 research articles (Figure 1 highlights papers on scaling laws on different topics) to comprehensively analyze scaling laws, examining their validity across different domains and architectures. While prior surveys have made valuable contributions to understanding scaling laws, they have primarily focused on specific aspects of the scaling phenomenon (See Table 1 ). Choshen et al. (2024) emphasized statistical best practices for estimating and interpreting scaling laws using training data, while Li et al. (2024b) emphasized on methodological inconsistencies and reproduction crisis in existing scaling laws. Our survey distinguishes itself by offering comprehensive coverage of architectural considerations, data scaling implications, and inference scaling – areas that previous surveys either overlooked or addressed only partially. Category Choshen et al. (2024) Li et al. (2024b) Ours Covers neural scaling laws broadly Yes No Yes Discusses fitting methodologies Yes Yes Yes Analyzes architectural considerations No Limited Yes Includes data scaling and pruning No Limited Yes Explores inference scaling No Limited Yes Considers domain-specific scaling No No Yes Provides practical guidelines Yes Yes Yes Critiques limitations of scaling laws Limited Yes Yes Proposes future research directions Limited Yes Yes Table 1: Key differences between our survey and existing surveys on neural scaling laws ( Choshen et al., 2024 ; Li et al., 2024b ) . (a) Architecture-wise statistics (b) Variable-wise statistics (c) Task-wise statistics Figure 3: Number of paper studied in this survey paper for different model architectures (a), scaling variables (b) and scaling tasks (c). The detailed paper list is provided in Table 9 of Appendix B . 2 Taxonomy of neural scaling laws Understanding the scaling laws of neural models is crucial for optimizing performance across different domains. We predominantly explore the scaling principles for language models, extending to other modalities such as vision and multimodal learning. We also examine scaling behaviors in domain adaptation, inference, efficient model architectures, and data utilization. We highlight the taxonomy tree of scaling laws research in Figure 2 . As highlighted in Figure 1 , neural scaling laws have been proposed predominantly for pre-training and fine-tuning scaling of large neural models. Among the models studied, as highlighted in Figure 3a , decoder-only Transformers dominate the subject, followed by vision transformers (ViT) and Mixture-of-Experts (MoE). The most common neural scaling laws take the form of power laws (Equation 1 ), where the model’s loss ( L ) or performance metric assumes to follow a predictable relationship with different scaling variables, L(P_{i....n})=\sum_{i=1}^{n}\alpha_{i}\cdot P_{i}^{-\beta_{i}} (1) with appropriate scaling parameters \beta_{i} and fitting parameters \alpha_{i} for different scaling parameter P_{i} . Figure 3b highlights that the number of model parameters and data size are the most common used scaling factors. The exact forms of all the scaling laws are highlighted in Table 10 of Appendix B . Among all the tasks, Figure 3c suggests that language generation is the most common task used for developing these scaling laws, where the training cross-entropy loss is widely used to fit the laws. Based on the values obtained empirically, the scaling laws are fitted with non-linear optimization, most commonly by running algorithms like least square and BFGS (Broyden-Fletcher-Goldfarb-Shanno). Statistical methods like goodness-of-fit metrics are used to validate the correctness of the fitted curves. We elaborate on the evaluation of neural scaling laws in Appendix A . In the following sections, we review the existing literature on neural scaling across various domains. Model scaling includes both parameter and data scaling. Parameter scaling is often studied in decoder-only Transformers ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) , with newer works addressing small and efficient models ( Hu et al., 2024 ; Clark et al., 2022 ) . These studies establish power-law relationships between loss and model size or compute (Equation 1 ). In parallel, data scaling research has proposed laws for optimizing mixtures ( Ye et al., 2024 ) , repeated training exposures ( Muennighoff et al., 2023 ) , vocabulary size ( Tao et al., 2024 ) , and knowledge capacity ( Allen-Zhu and Li, 2024 ) .

Read the complete text