Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning
David Zollikofer * [email protected], Béni Egressy * [email protected], Frederik Benzing [email protected], Matthias Otth [email protected], Roger Wattenhofer [email protected] · 2024
Read Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning by David Zollikofer * [email protected], Béni Egressy * [email protected], Frederik Benzing [email protected], Matthias Otth [email protected], Roger Wattenhofer [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01926.txt.
Introduction
David Zollikofer and co-authors widen the idea of redundancy used to teach machines from unlabeled data, in a paper from 2024. A popular family of methods learns by making the features of a representation less redundant, but only counts simple pairwise links between features. Real embeddings can hide tangle that pairs never reveal.
The authors define redundancy more fully and add measures that catch higher order dependence. They relate the measures to each other in theory, then measure them inside well known trained models. Their new method sets an encoder against a predictor in a game: one hides patterns, the other hunts them. Driving that game down strips more redundancy from the features. Tests check whether the leaner features perform better on downstream tasks.
Catalog record
| CREATOR | David Zollikofer * [email protected], Béni Egressy * [email protected], Frederik Benzing [email protected], Matthias Otth [email protected], Roger Wattenhofer [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01926
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning David Zollikofer * [email protected], Béni Egressy * [email protected], Frederik Benzing [email protected], Matthias Otth [email protected], Roger Wattenhofer [email protected] Abstract Several self-supervised learning (SSL) approaches have shown that redundancy reduction in the feature embedding space is an effective tool for representation learning. However, these methods consider a narrow notion of redundancy, focusing on pairwise correlations between features. To address this limitation, we formalize the notion of embedding space redundancy and introduce redundancy measures that capture more complex, higher-order dependencies. We mathematically analyze the relationships between these metrics, and empirically measure these redundancies in the embedding spaces of common SSL methods. Based on our findings, we propose Self Supervised Learning with Predictability Minimization (SSLPM) as a method for reducing redundancy in the embedding space. SSLPM combines an encoder network with a predictor engaging in a competitive game of reducing and exploiting dependencies respectively. We demonstrate that SSLPM is competitive with state-of-the-art methods and find that the best performing SSL methods exhibit low embedding space redundancy, suggesting that even methods without explicit redundancy reduction mechanisms perform redundancy reduction implicitly. * * footnotetext: These authors contributed equally to this work. 1 Introduction Self-supervised learning has been shown to produce image embeddings of similar quality as its supervised learning counterparts ( Chen et al., 2020 ) . Early methods, such as SimCLR, achieved this by using positive and negative sample pairs within the loss function. A positive pair refers to two different (random) perturbations of the same input image, whereas a negative sample pair refers to perturbed versions of two different input images. The SimCLR loss function encourages similar representations for positive pairs and dissimilar ones for negative pairs. In contrast, subsequent work by Zbontar et al. (2021) replaced negative sample pairs in the loss function with covariance reduction in the representation space. Instead of using negative sample pairs, their loss function reduces the redundancy between representation dimensions by reducing pairwise correlations. While pairwise decorrelating the representation space produces impressive results ( Zbontar et al., 2021 ) , it ignores potential redundancies that might remain. More precisely, the loss function does not explicitly reduce higher-order (involving more than two representation features) and non-linear redundancies in the representation space. In light of this observation, several natural questions arise: • How can redundancy in SSL embedding spaces be quantified? • How do redundancies in the embedding space affect downstream performance? • Can performance be further improved by removing additional, more complex redundancies? To answer these research questions we: (1) Introduce formal definitions of embedding space redundancies, specifically pairwise, linear, and non-linear redundancies, and derive their theoretical relationships; (2) Propose Self-Supervised Learning with Predictability Minimization (SSLPM), a novel SSL method built on redundancy reduction, which we show to be competitive with current state of the art; (3) Empirically investigate the relationship between model performance and embedding space redundancy in a wide range of SSL methods including Barlow Twins ( Zbontar et al., 2021 ) , BYOL ( Grill et al., 2020 ) , NNCLR ( Dwibedi et al., 2021 ) , SimCLR ( Chen et al., 2020 ) , MocoV3 ( He et al., 2020 ; Chen et al., 2021 ) , VICReg ( Bardes et al., 2021 ) , and VIbCReg ( Lee & Aune, 2021 ) . In our experiments, we find that • Reducing additional redundancies in training does not result in higher downstream performance. • Models with explicit redundancy reduction, such as Barlow Twins or SSLPM, show a clear link between performance and linear redundancy. However, this link does not hold in general. • All methods outperforming our SSLPM model exhibit strictly less embedding space redundancy, suggesting that even methods without explicit redundancy reduction perform redundancy reduction implicitly. However, for SSL methods in general, performance is only weakly correlated with embedding space redundancy. • The projector 1 1 1 Typically SSL networks use a projector between the embedding space used for downstream tasks and the representation space where the loss is calculated. The projector network consists of a stack of Feed-forward layers. Please refer to Figure 1 . depth has a significant impact on redundancy reduction: more projector layers result in less linear and nonlinear redundancy in the embedding space. 2 Related Work Early methods for SSL, such as SimCLR ( Chen et al., 2020 ) or InfoNCE ( Oord et al., 2018 ) , use a contrastive loss function based on positive and negative input pairs. The loss function incentivizes augmentations from the same input (positive pair) to produce similar embeddings, while pushing embeddings from different inputs (negative pairs) further apart in the embedding space. Follow-up works on SSL, such as BYOL ( Grill et al., 2020 ) and SiamSiam ( Chen & He, 2020 ) , have shown that negative samples are not always necessary by using asymmetric designs in their Siamese networks. A key challenge in SSL is the prevention of model collapse, whereby all embeddings converge to the same point so as to reduce the positive pair loss. Model collapse leads to no useful representations being learned. Instead of using negative pairs explicitly, Barlow Twins ( Zbontar et al., 2021 ) , VicReg ( Bardes et al., 2021 ) , VibCReg ( Lee & Aune, 2021 ) and W-MSE ( Ermolov et al., 2021 ) address model collapse by decorrelating features in the representation space of the model. By decorrelating the features in the representation space, redundancy is reduced. The idea of representation space redundancy reduction was first introduced by Schmidhuber et al. (1996) and Schraudolph et al. (1999) through their concept of predictability minimization. In predictability minimization, an encoder transforms inputs to an n -dimensional representation space, on which n predictors try to predict, in a leave-one-out fashion, every single representation feature from the others. The encoder tries to minimize redundancy by ensuring predictors cannot achieve a low prediction loss. The idea of two networks engaging in a competitive game was later revisited in the context of Generative Adversarial Networks ( Goodfellow et al., 2014 ; Schmidhuber, 2020 ) . Nevertheless, predictors are not novel to SSL and have found applications in ( Assran et al., 2023 ) where they aim to reconstruct representations of surrounding image patches from a single patch. Moreover, recent work by Shwartz Ziv & LeCun (2024) has drawn connections between compression in SSL methods and information theory. Our work, empirical in nature offers new insights in the role of redundancy reduction in SSL. 3 Methods 3.1 Background Figure 1 : Siamese training set up with encoder in self supervised learning. Embeddings are used in downstream applications, whereas representations are used for loss calculation during pretraining. After pretraining, during evaluation, only the encoder is kept and the randomized augmentations \tau are not applied. Our work is inspired by and builds on Barlow Twins ( Zbontar et al., 2021 ) . Barlow Twins is a self-supervised learning approach based on the Siamese training setup depicted in Figure 1 . It is used to pretrain neural networks before these are applied to downstream tasks. In a Siamese network, an input batch \mathcal{I} is passed through two different transformations \mathbf{t} and \mathbf{t^{\prime}} drawn at random from a distribution of transformations