A Random Matrix Theory of Masked Self-Supervised Regression
Arie Wortsman, Federica Gerace, Bruno Loureiro, Yue M. Lu · 2026
Read A Random Matrix Theory of Masked Self-Supervised Regression by Arie Wortsman, Federica Gerace, Bruno Loureiro, Yue M. Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23208.txt.
Introduction
Arie Wortsman, Federica Gerace, Bruno Loureiro, and Yue M. Lu analyze masked self-supervised learning with random matrix theory. Masked training, foundational for transformers, hides parts of the input and predicts them. Training mixes many masking patterns, creating a joint, matrix-valued predictor rather than a single estimator.
That object encodes how coordinates condition on one another, posing new analytical challenges. Random matrix tools are well suited to such large, structured objects. Theory of this kind explains why masked pretraining works as well as it does. How coordinates condition on one another is encoded in that matrix-valued predictor, the new object the theory must handle.
Catalog record
| CREATOR | Arie Wortsman, Federica Gerace, Bruno Loureiro, Yue M. Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23208
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Random Matrix Theory of Masked Self-Supervised Regression Arie Wortsman, Federica Gerace, Bruno Loureiro, Yue M. Lu Abstract In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik–Ben Arous–Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods. 1 Introduction Self-supervised learning (SSL) — a training paradigm in which models learn useful representations from unlabeled data by exploiting the data itself as a source of supervision — has emerged as a foundational component of the recent success of transformer architectures. By avoiding the need for manual annotations, SSL retains many of the benefits traditionally associated with supervised learning while avoiding reliance on labeled data. Consequently, SSL is widely adopted as a pretraining paradigm for learning general-purpose representations that substantially accelerate the optimization of downstream tasks, especially in data-scarce settings. A canonical example of a self-supervised learning task is masked language modeling (MLM), in which a neural network is trained to predict masked tokens in text using the remaining tokens as contextual information ( 16 ; 27 ; 47 ; 9 ; 44 ) . For example, given the sentence “ The capital of France is Paris ”, a typical MLM task would be to teach the model to infer that we are speaking about the capital of a country from the context “France” and “Paris” from the masked sentence “ The [MASK] of France is Paris ”. Masked language modeling is the core training framework of LLMs in the BERT family ( 17 ; 34 ) , where it is used to train state-of-the-art NLP encoders for which bi-directional context is crucial, as well as in vision transformers ( 6 ; 26 ) . Recent works ( 50 ; 49 ) have numerically shown that, when trained with MLM, transformers learn the conditional distribution of a missing token given the observed ones by sequentially inferring positional and semantic relationships across the entire text. However, several fundamental questions remain open. For instance: How much data is required to achieve good generalization performance in MLM? How does this performance depend on the underlying structure of the data, and in particular on the correlations among tokens? In this work, we investigate masked self-supervised learning in its simplest declination: learning real-valued sequence data with a matrix-valued linear predictor. More precisely, given n real-valued sequences x_{1},\dots,x_{n}\in\mathbb{R}^{d} , where d corresponds to the length of the sequence, we train a family of ridge-regularized linear predictors X\mapsto X\hat{A} under the constraint that no coordinate of the sequence can be used to predict itself (this will be made precise in the next section). In this context, our main contributions are: • Asymptotic limit for training and generalization errors: We derive a sharp asymptotic characterization of the training and generalization performance of the aggregate predictor \hat{A} in the high-dimensional regime where n,d\to\infty with a fixed ratio \nicefrac{{n}}{{d}}\to\alpha>0 . The resulting asymptotic limit depends only on the population covariance of the data, yielding a precise and interpretable description of which structural properties of the data enable successful generalization in masked self-supervised regression (SSR). • Deterministic equivalent for the predictor: Beyond performance metrics, we establish a high-dimensional deterministic equivalent for (the resolvent of) the matrix-valued aggregate predictor \hat{A} itself, which allows us to characterize how self-supervised learning encodes and exploits the underlying data geometry. Technically, this result relies on the analysis of a random matrix defined as an aggregate of a diverging number of correlated predictors. This matrix does not belong to standard random-matrix ensembles, and our analysis may be of independent interest. • Spiked vs AR(1): We further provide an in-depth analysis of two prototypical statistical models. In a spiked covariance model , we show that principal component analysis (PCA) strictly outperforms masked self-supervised learning, and we establish a BBP-type transition in the asymptotic spectrum of \hat{A} occurring at the same threshold as the corresponding transition for the sample covariance matrix. In sharp contrast, for a first-order autoregressive model , we prove that masked self-supervised regression can strictly dominate PCA in performance if the number of directions in PCA is not close to the dimension. We further provide a detailed analysis of the effective inductive bias induced by strong temporal correlations, elucidating the mechanisms by which self-supervision leverages sequential structure. Together, our results elucidate the mechanisms by which masked self-supervised learning exploits correlations in the training data, and provide a principled comparison between SSR and classical spectral approaches, revealing their respective strengths and limitations in the high-dimensional regime. Further Related Works: Deterministic equivalents for sample covariance matrices have a long history in the random matrix theory literature, starting with the seminal work of ( 39 ) and followed by several generalizations to the anisotropic settings ( 4 ; 10 ; 31 ; 52 ; 35 ; 12 ) . In the context of statistics and machine learning theory, RMT techniques have been used to derive insight into a manifold of high-dimensional models, including linear regression ( 18 ; 58 ; 25 ; 11 ) , kernels ( 28 ; 59 ; 41 ; 38 ) , random features ( 45 ; 36 ; 33 ; 40 ; 20 ; 23 ; 7 ; 53 ; 54 ; 15 ; 1 ; 32 ) , one-step analysis of feature learning ( 2 ; 42 ; 14 ) and in-context learning ( 37 ) . In the unsupervised context, RMT has been used to study mainly PCA ( 5 ; 48 ) ( 48 ) and clustering methods ( 13 ) . More recently, RMT techniques were used to investigate sequential and time series data in ( 29 ) , and a matrix-valued least-squares problem in ( 22 ) . Notation: Given a matrix X\in\mathbb{R}^{n\times d} , we consistently index by i,j\in[n] the rows of X , and by k,l\in[d] its columns, e.g. X=(x_{ik})_{\begin{subarray}{c}1\leq i\leq n\\ 1\leq k\leq d\end{subarray}} . We denote by X_{i}\in\mathbb{R}^{d} the i th row of X and X_{k}\in\mathbb{R}^{n} the k th column of X . In the same way, for i\in[n] , we denote by X^{(-i)}\in\mathbb{R}^{(n-1)\times d} the matrix obtained by removing the i th row, and by X^{(-k)}\in\mathbb{R}^{n\times(d-1)} the matrix obtained by removing the k th column. 2 Setting Let x_{1},\dots,x_{n}\in\mathbb{R}^{d} denote n data points in \mathbb{R}^{d} , for n,d\in\mathbb{N} , and let X=[x_{1},\dots,x_{n}]^{\top}\in\mathbb{R}^{n\times d} denote the data matrix. Here, each x_{i} can be thought as a sequence of (real valued) tokens or pixels of length d , in the