From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations

Anthony Bisulco, Rahul Ramesh*, Randall Balestriero, Pratik Chaudhari · 2025

Read From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations by Anthony Bisulco, Rahul Ramesh*, Randall Balestriero, Pratik Chaudhari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15404.txt.

Introduction

Anthony Bisulco, Rahul Ramesh, Randall Balestriero, and Pratik Chaudhari work out how mask-learning networks read spatial ties in an image. Such networks hide parts of an image and learn to fill them, a top pre-training for vision. Their knobs, hide share and patch size, need fresh tuning on each new set, with dim rules. Their 2025 sums solve the straight case in full.

In the straight network, hide share and patch size pick which spatial ties the network learns: near ties or far ties. Bent networks keep the bend: their marks adapt to the own tie shape of the set, past pair counts. Rules for knob picks in daily work close the piece. Tune to the tie span you need, they say.

Catalog record

CREATORAnthony Bisulco, Rahul Ramesh*, Randall Balestriero, Pratik Chaudhari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations Anthony Bisulco, Rahul Ramesh*, Randall Balestriero, Pratik Chaudhari Abstract Masked Autoencoders (MAEs) have emerged as a powerful pretraining technique for vision foundation models. Despite their effectiveness, they require extensive hyperparameter tuning (masking ratio, patch size, encoder/decoder layers) when applied to novel datasets. While prior theoretical works have analyzed MAEs in terms of their attention patterns and hierarchical latent variable models, the connection between MAE hyperparameters and performance on downstream tasks is relatively unexplored. This work investigates how MAEs learn spatial correlations in the input image. We analytically derive the features learned by a linear MAE and show that masking ratio and patch size can be used to select for features that capture short- and long-range spatial correlations. We extend this analysis to non-linear MAEs to show that MAE representations adapt to spatial correlations in the dataset, beyond second-order statistics. Finally, we discuss some insights on how to select MAE hyper-parameters in practice. 0 0 footnotetext: AB supported by NSF FRR 2220868, NSF IIS-RI 2212433, ARO MURI W911NF-20-1-0080, and ONR N00014-22-1-2677, and AB, RR, and PC supported by NSF IIS-2145164, CCF-2212519, ONR N00014-22-1-2255, and DSO Laboratories, Singapore. Figure 1 : MAEs (Black Arrow) can adapt the spatial scale of features they encode through two key hyperparameters: patch size and masking ratio. Generally, increasing either of these hyperparameters yields features with a broader spatial extent. Selecting these hyperparameters allows MAEs to be used for datasets with different spatial scales. In contrast, standard autoencoders (Green Arrow) learn short-range spatial features. On the right side of the figure, we show an example of feature spatial scales ranging from local textures to objects to global scenes. 1 Introduction Masked autoencoders (MAEs) are a powerful pretraining technique for images [ 20 , 54 ] and video [ 17 , 46 , 48 ] playing a key role in pretraining for foundational models such as Segment Anything [ 28 ] , InternVideo [ 49 ] , EVA [ 16 ] , and Unified IO [ 35 ] . Despite their success, MAEs are not yet well understood—key design choices such as masking ratio, patch size, and model depth heavily rely on heuristics. In this work, we aim to better understand MAEs by analyzing their underlying mechanisms and their role in representation learning. A key challenge in applying MAEs to new datasets or modalities is the need for extensive cross-validation to tune hyperparameters such as patch size and masking ratio. Suboptimal choices can lead to degraded task performance and unnecessary computational cost. Exhaustive search over these parameters is often impractical, as MAEs’ long training times make full cross-validation prohibitively expensive for large datasets or high-dimensional inputs. As a step towards addressing this challenge, we aim to study how masking ratio and patch size affect the representations learned by MAEs ( Fig. 1 ). Our central hypothesis is that MAEs learn spatial correlations in the input image , with the masking ratio and patch size controlling the spatial scale of these correlations. We argue that MAEs introduce a spatial locality bias in ViT (Vision Transformer) architectures that inherently lack it, although some efforts have attempted to explicitly incorporate such biases [ 34 , 14 ] . We make the following key contributions: 1. We derive an analytical expression for the encoder and decoder of a linear MAE. We show how MAEs can learn features that capture short- or long- range correlations by controlling the masking ratio and patch size. 2. We show that MAEs can be viewed as learning an adaptive basis for denoising, progressively shifting from localized to global feature representations over the course of training. 3. We provide insights for practitioners on how to select hyperparameters such as masking ratio, patch size, and number of encoder/decoder layers in MAEs. 2 Linear Masked Autoencoders Reconstruction is a popular objective for self-supervised learning, but it often fails to learn useful features for downstream perception tasks [ 6 ] . Masked reconstruction [ 20 , 54 ] has emerged as a popular choice for pretraining deep networks. In this section, we study how reconstruction differs from masked reconstruction and why the latter is useful for downstream perception tasks. Figure 2 : A d=8 dimensional input can be divided up into 4 patches, each of size p=2 . We first consider a linear MAE and analyze the solutions obtained for different masking ratios and patch sizes. Consider input data X\in\mathbb{R}^{n\times d} where n is the number of samples, and each sample is a real-valued vector in d dimensions. Consider a linear encoder with weights A\in\mathbb{R}^{d\times k} and a linear decoder with weights B\in\mathbb{R}^{k\times d} —the network has no non-linearities. The encoder projects the input from d input dimensions to k latent dimensions, and the decoder projects the representation back to d dimensions. A linear MAE minimizes the objective \ell_{m}(A,B)=\mathbb{E}_{R}\left[\left\|X-(R\odot X)AB\right\|^{2}\right], (1) where R\in\{0,1\}^{n\times d} is a random variable denoting a mask and \odot is the element-wise product. We can arrange our d -dimensional input vector into patches, each of size p (see Fig. 2 ). This is similar to how images are divided into patches in ViTs [ 13 ] . Patches are either fully masked or fully visible; this means R_{ki}=R_{kj} if i and j belong to the same patch for any image with index k . Each patch is masked according to a Bernoulli random variable with probability m . 1 1 1 Typical implementations of MAEs exactly mask a fraction m of the patches instead of using a random variable to determine if a patch should be masked. Typically, MAEs also reconstruct only the unmasked patches at the output of the decoder, instead of employing an objective that reconstructs all patches, as we have done here. Lemma 1 computes the expectation in Eq. 1 in closed form to reduce the linear MAE objective to \hskip-8.53581pt\ell_{m}(A,B){=}\underbrace{\left\|X{-}(1{-}m)XAB\right\|^{2}}_{\text{reconstruction}}\hskip-2.84526pt{+}m(1{-}m)\underbrace{\left\|GAB\right\|^{2}}_{\text{regularizer}}\hskip-11.38109pt (2) where G\in\mathbb{R}^{d\times d} satisfies G^{\top}G=\blkdiag_{p}(X^{\top}X) . The block-diagonal matrix \blkdiag_{p}(X^{\top}X) consists of block diagonal entries of size p\times p , with the off-diagonal blocks set to zero. The MAE objective in Eq. 2 consists of two loss terms: the first term is like the reconstruction loss of an autoencoder that encourages X\approx(1-m)XAB , the second term \left\|GAB\right\|^{2} acts as a regularizer tuned to the data via G . The masking ratio m controls the strength of the regularizer while the patch size controls the block-diagonal structure of the regularizer. Note that m=0 recovers the objective for a non-masked autoencoder (AE). The regularizer term here, therefore, is the “bias” of an MAE. It forces the representation of an MAE to deviate from that of an AE. 2.1 Characterizing the minima of MAEs We next analyze the encoder obtained in a linear MAE using Eq. 2 . The classic work of Baldi and Hornik 1989 shows that AEs are closely related to principal component analysis (PCA) [ 4 ] . AE’s encoder at any global minimum extracts the top- k principal components of the data. Linear AEs, therefore, discover features that best explain the variance in the input data. The training objective of a linear AE does not have local minima. All global minima are of the form A=UC^{-1},\hskip 10.00002ptB=CU^{\top} , where the columns

Read the complete text