Self-supervised learning from structural invariance
Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee, Shahab Bakhtiari, [0.05em] Eilif B. Muller, Laurent Charlin, [0.05em] Mila - Québec AI Institute, [0.05em] HEC Montréal · 2026
Read Self-supervised learning from structural invariance by Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee, Shahab Bakhtiari, [0.05em] Eilif B. Muller, Laurent Charlin, [0.05em] Mila - Québec AI Institute, [0.05em] HEC Montréal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02381.txt.
Introduction
Yipeng Zhang and 5 co-authors tackle a mismatch in how computers learn from pictures without labels, in a 2026 arXiv paper. Self-supervised learning trains a model by matching related pairs, such as nearby video frames. Trouble arises when one item can match several valid partners, a one-to-many situation that standard methods handle poorly.
Zhang's team adds a hidden variable that stands for the uncertainty about which partner is right. From that idea they derive a simple extra penalty term that can be added to standard training goals. The resulting method, AdaSSL, works with both contrastive and distillation-style objectives, two common training styles. Experiments show gains in causal representation learning, fine-grained image understanding, and video world modeling. The fix is small but broadly usable.
Catalog record
| CREATOR | Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee, Shahab Bakhtiari, [0.05em] Eilif B. Muller, Laurent Charlin, [0.05em] Mila - Québec AI Institute, [0.05em] HEC Montréal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02381
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-supervised learning from structural invariance Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee, Shahab Bakhtiari, [0.05em] Eilif B. Muller, Laurent Charlin, [0.05em] Mila - Québec AI Institute, [0.05em] HEC Montréal Abstract Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos. 1 1 1 Code will be made available at https://github.com/SkrighYZ/AdaSSL . 1 Introduction Over the last decade, joint-embedding self-supervised learning (SSL) has become the dominant approach in representation learning from unlabeled visual data ( 22 ; 112 ; 46 ; 84 ; 3 ) . The intuition behind SSL is to obtain semantically-related data pairs, often called positive pairs , and encourage their representations to be similar, with proper regularization to prevent the encoder collapsing to a constant function ( 104 ; 41 ; 115 ) . The hope is that invariance to the differences between positive pairs lead to generalizable representations. However, positive pairs are typically built with handcrafted augmentations (e.g., cropping, color jittering), which perturb pixels while preserving semantics. Such augmentations cannot precisely mimic changes in natural factors of variation that drive real-world distribution shifts ( 61 ) . For example, brightness jitter across pixels does not reproduce how lighting varies across objects or environments (e.g., from indoor to outdoor). Consequently, augmentations may fail to induce the right invariances ( 61 ; 60 ; 15 ) , discard fine-grained information ( 22 ; 114 ) , incur additional computation burden ( 14 ) , and require modality-specific heuristics ( 7 ) , ultimately harming downstream performance. One alternative is to exploit naturally-paired data—e.g., nearby video frames ( 68 ; 9 ; 91 ) , image–caption pairs ( 84 ) , class labels ( 65 ) , or embeddings from other models ( 94 ; 38 ) —which (better) reflect real-world variations. From the lens of causal representation learning (CRL) ( 108 ; 85 ) , positive pairs ({\mathbf{x}},{\mathbf{x}}^{+}) are deterministically mapped from latent factors sampled according to ({\mathbf{z}},{\mathbf{z}}^{+})\sim p({\mathbf{z}})p({\mathbf{z}}^{+}\mid{\mathbf{z}}) , and we aim to recover {\mathbf{z}} from {\mathbf{x}} . Unlike augmentations that perturb the observations directly, natural positive pairs differ according to structured changes in latent factors of the data generating process (DGP). Modelling these latent changes often improves generalization ( 60 ; 28 ; 64 ) and visual understanding ( 5 ; 40 ; 74 ) . Despite the aforementioned benefits, modeling natural pairs for SSL remains challenging. When learning with augmented pairs, we typically assume the semantic latent factors are invariant, so the latent conditional distribution p({\mathbf{z}}^{+}\mid{\mathbf{z}}) is highly concentrated near {\mathbf{z}} . However, natural pairs induce complex p({\mathbf{z}}^{+}\mid{\mathbf{z}}) . For example, in world modeling ( 51 ; 50 ; 52 ; 4 ) , the present state {\mathbf{z}} may lead to multiple plausible futures \{{\mathbf{z}}^{+}\} (e.g., a car may turn left or right), making the conditional distribution inherently multimodal. For image–caption pairs, caption details vary with image complexity, producing heteroscedastic noise. SSL methods that fail to capture this uncertainty often discard information not shared between the pair, leading to degraded performance ( 22 ; 84 ; 63 ; 110 ; 100 ; 114 ) . We argue that leveraging the structure of p({\mathbf{z}}^{+}\mid{\mathbf{z}}) enables SSL to learn more diverse and generalizable features—a principle we call SSL from structural invariance . Recent works enable contrastive SSL to learn p({\mathbf{z}}^{+}\mid{\mathbf{z}}) that has constant, anisotropic noise ( 69 ; 116 ; 87 ) . In this work, we contribute a solution to model more complex conditional distributions with unknown shapes. We are inspired by joint-embedding predictive architectures (JEPAs) ( 72 ; 39 ; 4 ) , which use a latent variable to capture the prediction uncertainty. However, in contrast to prior work ( 27 ; 39 ; 44 ; 25 ) , we do not assume access to this variable and infer it purely from (the structure hidden in) positive pairs. For contrastive learning, this latent variable allows us to derive a new tractable bound on the mutual information (MI) between paired embeddings—a core motivation for SSL ( 73 ; 101 ; 82 ) —and we empirically show its compatibility with non-contrastive methods. We name our method Adaptive SSL (AdaSSL) to highlight that it can adapt to different conditional distributions. We evaluate AdaSSL in controlled settings with numerical data, natural images, and videos. On numerical data, we show that existing SSL methods lack the ability to model non-trivial conditionals, and AdaSSL achieves better performance both in- and out-of-distribution (OOD). On images, AdaSSL consistently recovers fine-grained features better than baselines and learns more disentangled representations. On videos, AdaSSL captures stochastic object accelerations that baselines discard without sacrificing class accuracy. In summary, our main contributions show that: • Naturally paired data that differ in their data-generating factors can help SSL methods to recover these factors better and lead to improved generalization performance. • Existing SSL methods cannot capture the non-trivial conditional distributions induced by natural pairs, and we propose two variants of AdaSSL to address this limitation. • AdaSSL consistently outperforms baselines on a variety of benchmarks, including weakly-supervised CRL, fine-grained image understanding, and world modeling. 2 Problem formulation In this section, we lay out our problem (§ 2.1 ) and discuss the limitations of existing SSL methods in addressing it (§ 2.2 , § 2.3 ). We then present a theoretical result showing the importance of modeling heteroscedastic noise (§ 2.4 ), which motivates our method. 2.1 Data generating process In CRL, representation learning is viewed as learning to invert the DGP such that the latent factors responsible for generating the data are recovered ( 116 ; 69 ; 87 ; 85 ) . We assume a data pair ({\mathbf{x}},{\mathbf{x}}^{+}) conforms to the following generative process: {\mathbf{z}}\sim p({\mathbf{z}})\,,\quad{\mathbf{r}}\mid{\mathbf{z}}\sim p({\mathbf{r}}\mid{\mathbf{z}})\,,\quad{\mathbf{z}}^{+}\mid{\mathbf{z}},{\mathbf{r}}\sim p({\mathbf{z}}^{+}\mid{\mathbf{z}},{\mathbf{r}})\,,\quad{\mathbf{x}}=g({\mathbf{z}})\,,\quad{\mathbf{x}}^{+}=g({\mathbf{z}}^{+})\,, (1) where g:{\mathcal{Z}}\to{\mathcal{X}} is an unknown mixing function that produces the observations {\mathbf{x}},{\mathbf{x}}^{+}\in{\mathcal{X}} based on the latent factors {\mathbf{z}},{\mathbf{z}}^{+}\in{\mathcal{Z}} . We introduce a latent variable {\mathbf{r}} that governs the transformation from {\mathbf{z}} to {\mathbf{z}}^{+} . This factorizes p({\mathbf{z}}^{+}\mid{\mathbf{z}}) into a two-step process: first, {\mathbf{r}} is drawn from a distribution of possible latent influences on {\mathbf{z}} ; then p({\mathbf{z}}^{+}\mid{\mathbf{z}},{\mathbf{r}}) uses {\mathbf{r}} to produce {\mathbf{z}}^{+} in latent space. For example, {\mathbf{r}} may represent camera movement, agent actions, or temporal gaps—factors that modify the underlying latent factors before they are mapped to observations through g . Under Eq. 1 , the variability in {\mathbf{x}} is entirely captured by the latent factors {\mathbf{z}} since g is deterministic. A representation that encodes the full latent variability is therefore important for supporting a broad range of downstream tasks, each relying