Generalisation and benign over-fitting for linear regression onto random functional covariates
Andrew Jones, Nick Whiteley · 2025
Read Generalisation and benign over-fitting for linear regression onto random functional covariates by Andrew Jones, Nick Whiteley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13895.txt.
Introduction
Andrew Jones and Nick Whiteley study a basic question in statistics: how well does linear regression predict new data when the inputs come from random curves? In their setup, each data point is made by sampling a set of random functions at hidden locations, with noise added. The data points are linked to each other rather than fully independent.
The authors analyze two forms of the method, ridge regression and its ridge-less cousin, which can fit training data exactly. They derive probability bounds on prediction error that fit this random-curve setting. Their rates show how the error falls as the number of functions grows fast relative to the number of points. The work also clarifies when exact fitting stays harmless, a pattern called benign over-fitting, and how noise in the inputs affects that behavior.
Catalog record
| CREATOR | Andrew Jones, Nick Whiteley |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 132 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13895
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalisation and benign over-fitting for linear regression onto random functional covariates Andrew Jones, Nick Whiteley Abstract We study theoretical predictive performance of ridge and ridge-less least-squares regression when covariate vectors arise from evaluating p p random, means-square continuous functions over a latent metric space at n n random and unobserved locations, subject to additive noise. This leads us away from the standard assumption of i.i.d. data to a setting in which the n n covariate vectors are exchangeable but not independent in general. Under an assumption of independence across dimensions, 4 4 -th order moment, and other regularity conditions, we obtain probabilistic bounds on a notion of predictive excess risk adapted to our random functional covariate setting, making use of recent results of Barzilai and Shamir 2023 . We derive convergence rates in regimes where p p grows suitably fast relative to n n , illustrating interplay between ingredients of the model in determining convergence behaviour and the role of additive covariate noise in benign-overfitting. 1 Introduction In studies of the theoretical predictive performance of supervised Machine Learning methods it is very commonly assumed that training data in the form of covariate/response pairs \{(\mathbf{x}_{i},y_{i});i=1,\ldots,n\} and test data (\mathbf{x}_{test},y_{test}) are i.i.d. For ridge regression and ridge-less least squares regression, the assumption of i.i.d. data is standard in studies of the classical regime where n\to\infty , with \mathbf{x}_{i}\in\mathbb{R}^{p} for p<n or with covariate vectors valued in a possibly infinite-dimensional Hilbert space [ 33 , 28 , 10 , 17 , 24 ] . The i.i.d. assumption is also prominent in the emerging literature on the ‘benign overfitting’ phenomenon [ 4 , 16 , 6 , 30 , 12 , 13 , 5 ] , where, under conditions on the eigenvalues of the covariance matrix of covariate vectors, least-norm solutions of the over-parameterised least squares problem which interpolate — i.e., fit training data exactly — when p\gg n , can have good predictive performance. These high-dimensional settings do not involve sparsity as in the very well known \ell_{1} -penalised settings of, e.g., [ 9 , 31 , 8 , 15 ] . Studies of non-sparse, high-dimensional linear regression when there is some form of dependence across samples have appeared recently: [ 25 ] consider benign over-fitting for linear regression in a time series model with stationary, centered, Gaussian covariates and noise; [ 23 ] consider ridge regression with linearly-dependent, non-Gaussian data in a setting where n and p grow proportionally; [ 2 ] assume covariates are generated from a Gaussian process with spatio-temporal covariance; [ 21 ] assume covariates follow a right-rotationally invariant distribution. Motivated by desire to understand the double-descent phenomena in neural networks, some recent studies [ 16 , 22 ] have considered a situation in which (\mathbf{x}_{i},y_{i})_{i\geq 1} are i.i.d., but least squares linear regression of y_{i} is performed on to mapped covariate vectors \sigma(\mathbf{W}\mathbf{x}_{i}) , where \mathbf{W} is a random matrix with i.i.d. elements, and \sigma(\cdot) is some element-wise nonlinearity. This is a simple model of a neural network, with a single nonlinear layer with random weights, and a linear output layer. The mapped vectors \sigma(\mathbf{W}\mathbf{x}_{i}) , i=1,2,\ldots, are exchangeable but not independent of each other, because of the presence of the random matrix \mathbf{W} . The overall aim of the present work is to study the generalisation performance of ridge and ridge-less least squares regression when simultaneously p/n\to\infty and n\to\infty , in the setting of the Latent Metric Model (LMM), a general form of model for high-dimensional data explored in the forthcoming JRSSB discussion paper of Whiteley et al. 2024 . The interest in the LMM is that it serves as a general alternative to the assumption of i.i.d. data with rich behaviour: Whiteley et al. 2024 illustrated how intrinsically low-dimensional nonlinear structure can emerge in high-dimensional data from the LMM when there is independence across dimensions, providing a statistical grounding for the so-called Manifold Hypothesis [ 11 , 7 , 14 ] . Moreover the Gaussian Process Latent Variable model of Lawrence 2003 , Lawrence 2005 is a special case of the LMM. Under the LMM, data vectors are exchangeable but not independent in general, and arise from evaluation of a collection of p random functions at n random, latent, metric space-valued locations, subject to additive noise. We introduce a novel regression setup tailored to the structure of the LMM: the domain of the unknown regression function is the latent metric space in the LMM; we assume that we do not have access to the latent variables associated with the training or test data; and there is dependence between training and test covariates arising from the random functions in the LMM. We also work under mild finite 4 th-moment conditions which are much weaker than sub-Gaussian assumptions which are common in studies of benign overfitting, e.g., [ 4 , 25 ] . Our study is largely inspired and stimulated by numerous recent contributions regarding benign overfitting, [ 4 , 16 , 6 , 30 , 12 , 13 , 5 ] , and in particular we rely heavily on modifications to some results of [ 5 ] , which are a key building block for us. The connection to [ 5 ] which we explore is that, when p is large and there is independence across dimensions of the random functions and additive noise in the LMM, the predictions obtained from ridge/ridge-less least squares regression can be viewed as perturbations of predictions from kernel ridge/ridge-less regression, where the kernel is what we shall call the implicit kernel , defined by the ingredients of the LMM. Moreover, we show how implicit regularisation of the regression problem can arise from the ingredients of the LMM as p\to\infty . The results of Barzilai and Shamir 2023 are also useful for us because they rely on realistic assumptions on the kernel in question. In particular we note that [ 5 , Sec 2.2] sets out in detail the ways in which their setup substantially loosens restrictive assumptions in prior work. Perturbations of kernel ridge regression have been considered in the literature on Random Fourier Features (RFF), e.g., [ 27 , 3 , 20 ] . RFF is an approach to ameliorating the cost of kernel methods using randomised functional approximations to kernel Gram matrices. Both the mathematical details and motivation of RFF are different to our setting: in RFF the Gram matrix approximations arise by the user sampling from the spectral measure (or some closely related measure over frequencies) in the Bochner’s theorem representation of the kernel (the kernel is chosen to be shift invariant), with the aim of controlling computational cost. In our setting we make no assumptions on the functional form of the kernel. Instead, we take the perspective that the LMM is nature’s data generating mechanism, rather than being an engineered sampling device which the user controls, and the kernel we consider is defined implicitly through the LMM and not something the user of the regression method is free to choose. The rest of this article is structured as follows. • In Section 2 we introduce the LMM and some special cases thereof. We explain key differences in our regression setup and definition of prediction error compared to