Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches

Yifang Chen, Xuyang Guo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song · 2025

Read Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches by Yifang Chen, Xuyang Guo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01314.txt.

Introduction

Yifang Chen and five colleagues ask whether the scaling laws behind large language models also hold in simpler regression settings. For Transformer models, test loss follows a power law in model size, dataset size, and training computation, across more than seven orders of magnitude. That behavior clashes with the traditional expectation that overparametrized models overfit and generalize poorly.

Scaling laws have already been identified in linear regression, but a full explanation for large practical models remains out of reach. The authors show that the phenomenon extends to multiple regression and kernel regression, which are significantly more expressive and powerful than linear methods. Their analysis gives deeper insight into why performance improves so predictably with scale. Because these simpler settings can be analyzed rigorously, the results may sharpen understanding of the scaling behavior that large language models display.

Catalog record

CREATORYifang Chen, Xuyang Guo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01314

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches Yifang Chen, Xuyang Guo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song Abstract Recently, Large Language Models (LLMs) have achieved remarkable success. A key factor behind this success is the scaling law observed by OpenAI. Specifically, for models with Transformer architecture, the test loss exhibits a power-law relationship with model size, dataset size, and the amount of computation used in training, demonstrating trends that span more than seven orders of magnitude. This scaling law challenges traditional machine learning wisdom, notably the Oscar Scissors principle, which suggests that an overparametrized algorithm will overfit the training datasets, resulting in poor test performance. Recent research has also identified the scaling law in simpler machine learning contexts, such as linear regression. However, fully explaining the scaling law in large practical models remains an elusive goal. In this work, we advance our understanding by demonstrating that the scaling law phenomenon extends to multiple regression and kernel regression settings, which are significantly more expressive and powerful than linear methods. Our analysis provides deeper insights into the scaling law, potentially enhancing our understanding of LLMs. 1 Introduction The rapid advancement of large language models (LLMs) has revolutionized natural language processing, enabling remarkable generalization across diverse applications [ 19 , 101 ] . Transformer-based architectures [ 123 ] , when scaled to billions of parameters and trained on vast datasets, have proven to be highly effective for numerous NLP tasks. These models, such as BERT [ 33 ] , PaLM [ 31 ] , Llama [ 120 ] , and GPT-4 [ 101 ] , have benefited from methods like multitask fine-tuning [ 105 ] , prompt tuning [ 69 ] , and reinforcement learning from human feedback (RLHF) [ 24 ] , among others. Alongside these empirical advances, scaling laws have been extensively studied [ 68 ] , revealing the relationship between generalization error and key factors such as compute, sample size, and model parameters [ 94 ] . These scaling insights have provided guidelines for optimally allocating resources in training deep learning models. Multiple regression, which predicts vector-valued outputs simultaneously, offers crucial insights for multi-task learning systems. In contrast, kernel regression extends these concepts through non-linear feature mappings that enable more expressive modeling. Recent research has explored both approaches across various regimes, addressing phenomena like double descent and overparameterization [ 17 ] . Notably, connections between kernel methods and neural networks have been drawn, with insights from the Neural Tangent Kernel (NTK) [ 63 ] highlighting similarities between kernel regression and training infinitely wide neural networks. Thus, it is natural to ask, Can we extend the error bound from linear regression to multiple and kernel regressions? While previous research has established scaling laws for simple linear models [ 94 ] , the extension to multiple regression settings—where we predict multiple output variables simultaneously—represents a significant advancement in understanding more complex learning paradigms. This work demonstrates that scaling law phenomena extend robustly to multiple regression settings, where we predict vector-valued outputs rather than scalar values. Our analysis reveals that under appropriate assumptions of Gaussianity, well-specified models, and power law decay of eigenvalues in the feature covariance matrix, multiple regression exhibits predictable scaling behaviors similar to those observed in simpler models. This extension to multiple regression settings offers crucial insights for multi-task learning systems. It serves as a bridge between simple linear models and the more expressive kernel methods that we subsequently analyze. Our primary contributions are as follows: • Deriving the scaling law for multiple regression and kernel regression (see Theorem 4.6 ). • Presenting novel generalization error bounds for sketched multiple regression (see Theorem 4.5 ). • Presenting novel generalization error bounds for sketched kernel regression (see Corollary 4.7 ). Our work leverages stochastic gradient descent (SGD) under specific assumptions. These bounds provide insight into the bias-variance trade-off in the sketched predictor setting and help understand the effectiveness of sketching in reducing computational complexity while maintaining predictive accuracy. Furthermore, we derived the scaling law for both the multiple regression and kernel regression based on our generalization error bounds. Roadmap. Section 2 discusses the work on scaling laws of Large Language Models and regression problems. In Section 3 , we present the notation, definitions, and foundational concepts necessary for understanding their theoretical analysis of multiple and kernel regression. Section 4 shows our key theoretical findings on scaling laws for multiple regression and kernel regression, including generalization error bounds. Section 5 provides detailed mathematical analysis of approximation errors in multiple regression with upper and lower bounds. In Section 6 , we derive the bias component of error in multiple regression scenarios with both upper and lower bounds. In Section 7 , we investigate excess errors in multiple regression by examining hypercontractivity, Gaussianity, and well-specified noise under sketched features. In Section 8 , we summarize our findings on scaling laws in multiple and kernel regression and discuss implications for understanding generalization in more complex machine learning models. 2 Related Work 2.1 Scaling Laws The scaling behavior of deep learning models concerning compute, sample size, and model size has been a central topic in machine learning research [ 55 , 104 , 19 , 52 , 51 , 134 , 98 ] . A pivotal contribution by [ 68 ] revealed that generalization error in transformers decays as a power law with respect to these three factors. Their work provided joint formulas predicting how model performance improves with increased compute, data, and parameters. This finding highlighted the importance of scaling models and led to a surge in interest in large-scale models for NLP and other tasks. Further research refined these scaling laws. For instance, [ 51 ] proposed the Chinchilla law, which suggested that under compute constraints, balancing data size with model size yields optimal performance, contrasting with earlier models that focused primarily on increasing parameter counts. Meanwhile, [ 98 ] introduced methods for data reuse across training passes, showing that efficient data handling can maintain performance while lowering the need for excessive compute resources. These empirical studies have provided clearer guidelines for allocating resources effectively when training large models. In parallel, theoretical work has advanced understanding of these scaling laws. [ 106 ] demonstrated that, in regression, the generalization error scales as n^{-4/d} , linking model performance to the intrinsic dimensionality of the data. Additionally, other researchers, such as [ 15 ] and [ 13 ] , applied statistical physics to derive scaling laws for linear models, particularly under infinite parameter regimes. Theoretical contributions like these complement empirical findings and offer deeper insights into how models behave as they scale, shaping future directions in machine learning research. 2.2 Regression Problems Regression problems are a very important research topic: they model the dependent variable Y using one or more independent variables X . Recent studies have explored various regression problems. One of the most prominent regression problems is linear regression [ 116 , 114 ] . Linear regression deals with a single independent variable, whereas multiple regression, \min_{X\in\mathbb{R}^{d\times N}}\|AX-B\|_{F} [ 116 ] given A\in\mathbb{R}^{n\times d} and B\in\mathbb{R}^{n\times N} , incorporates two or more predictors to improve prediction accuracy. They have been extensively studied across various domains, including statistics [ 127

Read the complete text