On the Importance of Embedding Norms in Self-Supervised Learning
Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, Erik Bekkers · 2025
Read On the Importance of Embedding Norms in Self-Supervised Learning by Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, Erik Bekkers (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09252.txt.
Introduction
Andrew Draganov and six co-authors settle what the length of a learned embedding does in self-supervised learning. Most such methods grade embeddings by the cosine of their angle, which puts them on a ball and seems to bar length from play. Their 2025 work shows length is at work in two ways. It sets how fast training converges, and it codes the confidence of the network, with short embeddings for odd samples.
Proof, simulations, and bench runs all back the pair of roles. Push length by hand and convergence speed swings wide. Length is thus no scrap of the ball trick: it is a dial. Read it for confidence, tune it for speed, they say. Past readings that set length aside miss half the story of such learning.
Catalog record
| CREATOR | Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, Erik Bekkers |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09252
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Importance of Embedding Norms in Self-Supervised Learning Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, Erik Bekkers Abstract Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in machine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on a hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, a few recent works have suggested that embedding norms have properties related to network convergence and confidence. In this paper, we resolve this apparent contradiction and systematically establish the embedding norm’s role in SSL training. Using theoretical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL convergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulating embedding norms can have large effects on convergence speed. Our findings demonstrate that SSL embedding norms are integral to understanding and optimizing network behavior. 1 Introduction Self-supervised learning (SSL) has emerged as a powerful tool for learning representations from unlabeled data. This is because, when trained on large unlabeled datasets with contrastive objectives, SSL models often achieve performance levels comparable to those of supervised methods and can even induce emergent properties ( Chen et al. 2020a ; Caron et al. 2021 ) . SSL has also significantly advanced multi-modal learning, particularly in vision-language models ( Girdhar et al. 2023 ; Bommasani et al. 2021 ) . Indeed, prominent methods such as CLIP ( Radford et al. 2021 ) , ALIGN ( Jia et al. 2021 ) and Florence ( Yuan et al. 2021 ) all rely on standard SSL objectives discussed in this paper. These algorithms work by representing similar inputs near each other and dissimilar inputs far apart in an embedding space normalized to a hypersphere. This is done by optimizing objective functions based on the cosine similarity between embeddings: popular SSL frameworks like SimCLR ( Chen et al. 2020a ) and MoCo ( He et al. 2020 ) optimize the InfoNCE objective, while SimSiam ( Chen & He 2021 ) and BYOL ( Grill et al. 2020 ) optimize the cosine similarity directly. Thus, theoretical studies of SSL representations only consider the distribution of points on this latent hypersphere ( Wang & Isola 2020 ; Zimmermann et al. 2021 ) . Figure 1 : Left : Network outputs (blue) lie in ambient \mathbb{R}^{d} , but only their projections (white) onto the hypersphere S^{d-1} enter the loss. However, the embedding norms (shades of blue) influence the norms of the SSL loss gradient. Middle : Gradient field of the cosine similarity with respect to the orange direction. Highlighted gradients (black) illustrate the inverse relationship between the gradient and embedding norm (Proposition 1 ) The blue points trace an embedding’s trajectory using gradient descent. Right : Because gradient updates are orthogonal to an embedding point, they must increase the point’s norm (Corollary 3.3 ). At the same time, there has been some empirical evidence that the embedding norms prior to normalization contain meaningful information. For example, Kirchhof et al. 2022 noted that embedding norms are related to network confidence. No theoretical explanation of this phenomenon exists. Separately, Zhang et al. 2020 and Wang et al. 2017 showed that training based on the cosine similarity loss affects the embedding norms and that the embedding norms affect the gradient magnitudes. However, the implications of this for SSL methods have not been fully explored. This implies a clear gap in the literature: several works suggest interactions between SSL models and the embedding norms but the extent of this relationship is unknown. Our work provides the first thorough analysis showing how the embedding norms interact with SSL training dynamics. First, we prove theoretical bounds showing that the embedding norms impose a quadratic slowdown on SSL convergence and verify that this occurs in simulation and in practice. At the same time, we show that optimizing the cosine similarity grows these embedding norms across SSL methods. This creates an inherent catch-22: small embedding norms are required to train SSL models but these norms grow during training. Put simply, effectively training SSL models requires managing the embedding norms . We offer and evaluate several mechanisms for doing this. Second, we argue that, because the embeddings grow with each gradient update, their norms naturally correspond to the frequency of observed latent features. Since models are more certain in frequently observed data ( Zhong et al. 2021 ) , we contend that SSL embedding norms encode a model’s confidence . We devise methods for studying this and show that it holds with remarkable consistency. Moreover, these two phenomena interact during SSL training. For instance, since norms both encode uncertainty and scale gradients, SSL models implicitly learn unevenly over the samples — an effect we show must be mitigated when training on imbalanced datasets. Throughout our paper, we draw attention to similar interactions between embedding norms and SSL dynamics by raising several open questions as directions for future work. Our code is available at https://github.com/Andrew-Draganov/SSLEmbeddingNorms . This paper supersedes our earlier workshop paper ( Draganov et al. 2024 ) . 2 Preliminaries and Related Work Although this paper concerns general SSL techniques, we will use the familiar language of image representation learning. The typical SSL pipeline consists of obtaining two augmented variants x_{i} and x_{j} from an input image x and ensuring that the corresponding embeddings z_{i} and z_{j} have high cosine similarity. We refer to (x_{i},x_{j}) (resp. (z_{i},z_{j}) ) as ‘positive’ pairs of points (resp. embeddings). Since attracting all the points together will collapse the representation, each SSL method has an additional mechanism to prevent this. The most common approach enforces low similarity between embeddings of dissimilar inputs: for unrelated inputs x_{i} and x_{k} , the embeddings z_{i} and z_{k} should be far apart. 2.1 Preliminaries & SSL methods Perhaps the prototypical method in this family is SimCLR ( Chen et al. 2020a ) , which appears among a broader line of contrastive methods for self-supervised learning on images. These all utilize repulsions from dissimilar (negative) samples to prevent representation collapse ( Oord et al. 2018 ; Henaff 2020 ; He et al. 2020 ; Chen et al. 2020b ; Misra & Maaten 2020 ) . In practice, this is performed by optimizing the InfoNCE loss, which we write with respect to embedding z_{i} as \displaystyle\mathcal{L}_{ij}(\mathbf{Z}) \displaystyle=-\log\frac{\text{ExpSim}(z_{i},z_{j})}{\sum_{k\not\sim i}\text{ExpSim}(z_{i},z_{k})} (1) \displaystyle=-\underbrace{\hat{z}_{i}^{\top}\hat{z}_{j}}_{\mathcal{L}^{\mathcal{A}}_{ij}(\mathbf{Z})}+\underbrace{\log\left(S_{i}\right)}_{\mathcal{L}^{\mathcal{R}}_{ij}(\mathbf{Z})}, where \hat{z} is the unit vector of z , \text{ExpSim}(a,b)=\exp\big(\hat{a}^{\top}\hat{b}\big) is the exponent of the cosine similarity, and S_{i}=\sum_{k\not\sim i}\text{ExpSim}(z_{i},z_{k}) represents the sum over all k\neq i in the batch. For ease of interpretation, we split the loss term into attractive and repulsive components, resp. \mathcal{L}^{\mathcal{A}}_{ij} and \mathcal{L}^{\mathcal{R}}_{ij}(\mathbf{Z}) . Note that \mathcal{L}^{\mathcal{A}}_{ij}(\mathbf{Z}) is simply the negative cosine similarity between positive pair z_{i} and z_{j} . Another common contrastive objective function is the triplet loss, wherein one normalizes the embeddings to the hypersphere and minimizes the mean squared error between positive samples while maximizing the mean squared error between negative ones ( Schroff et al. 2015 ) . Due to the normalization, this implicitly optimizes the cosine similarity between embeddings ( Grill et al. 2020 ) .