Temperature-Free Loss Function for Contrastive Learning

Bum Jun Kim, Sang Woo Kim · 2025

Read Temperature-Free Loss Function for Contrastive Learning by Bum Jun Kim, Sang Woo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17683.txt.

Introduction

Bum Jun Kim and Sang Woo Kim remove a fiddly setting from a leading self-supervised learning method, in a research paper from 2025. Contrastive learning teaches models by comparing pairs of data points, often with the InfoNCE loss. That loss needs a temperature value that scales similarity scores, and finding a good one takes many trial runs.

The authors swap the temperature scaling for an inverse hyperbolic tangent function, yielding a modified loss with no such setting. Their analysis shows the old scaling warps the gradients that drive learning, while the new form keeps them well behaved. Tests on five benchmarks gave sound results with no temperature search at all. The change even brought a small gain in performance.

Catalog record

CREATORBum Jun Kim, Sang Woo Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17683

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temperature-Free Loss Function for Contrastive Learning Bum Jun Kim, Sang Woo Kim Abstract As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to implementing contrastive learning is applying InfoNCE loss: By capturing the similarities between pairs, InfoNCE loss enables learning the representation of data. Albeit its success, adopting InfoNCE loss requires tuning a temperature, which is a core hyperparameter for calibrating similarity scores. Despite its significance and sensitivity to performance being emphasized by several studies, searching for a valid temperature requires extensive trial-and-error-based experiments, which increases the difficulty of adopting InfoNCE loss. To address this difficulty, we propose a novel method to deploy InfoNCE loss without temperature. Specifically, we replace temperature scaling with the inverse hyperbolic tangent function, resulting in a modified InfoNCE loss. In addition to hyperparameter-free deployment, we observed that the proposed method even yielded a performance gain in contrastive learning. Our detailed theoretical analysis discovers that the current practice of temperature scaling in InfoNCE loss causes serious problems in gradient descent, whereas our method provides desirable gradient properties. The proposed method was validated on five benchmarks on contrastive learning, yielding satisfactory results without temperature tuning. 1 Introduction Deep learning has shown remarkable results across numerous tasks using large datasets, often with labels. To cope with extensive manual labeling, research in self-supervised learning has sought to learn a representation of data without their labels ( Caron et al., 2020 ; Grill et al., 2020 ) . Among several directions in self-supervised learning, recent studies on contrastive learning have demonstrated promising results in various fields, including computer vision, graph representation learning, and anomaly detection ( Chen et al., 2020 ; Zhu et al., 2020 ; Reiss & Hoshen, 2023 ) . Table 1: Numerous studies with InfoNCE loss have adopted their own temperature values Study Best Temperature He et al. (2020) 0.07 Wu et al. (2018) 0.07 Caron et al. (2020) 0.1 Reiss & Hoshen (2023) 0.25 Grill et al. (2020) 0.3 Chen et al. (2020) 0.1 or 0.5 Contrastive learning aims to learn a representation space of data using the similarity scores of possible pairs that are obtained through different views ( Dosovitskiy et al., 2016 ) . A common approach in contrastive learning is to adopt InfoNCE loss ( van den Oord et al., 2018 ; Chen et al., 2020 ) , whose minimization enables discriminating similarities and dissimilarities within pairs, thereby acquiring representation of data. Figure 1: Overview of contrastive learning with InfoNCE loss with existing division by temperature scheme and proposed method. The cosine similarities [C,-C] follow the scenario in the main text. When approaching the optimal point C\rightarrow 1 , the division by temperature scheme raises several problems in gradient descent, whereas the proposed method ensures convergence. The graphs on the right represent the loss L_{1} . The similarity score in InfoNCE loss is accompanied by a division by temperature to calibrate its scale. However, several studies have emphasized the significance of temperature to the performance of contrastive learning ( Wang & Liu, 2021 ; Zhang et al., 2022b ) . Although the current consensus is to adopt a low temperature such as 0.07, 0.1, or 0.25, the exact value differs by each study (Table 1 ). Indeed, numerous studies using InfoNCE loss have searched for the best temperature that is obtained through their own hyperparameter tuning via trial-and-error-based approaches. For example, MoCo ( He et al., 2020 ) used a temperature of 0.07, whereas MSC ( Reiss & Hoshen, 2023 ) adopted 0.25. Grill et al. (2020) reported that the temperature of 0.1 from Chen et al. (2020) did not work suitably with their method and instead used 0.3 that is obtained from their hyperparameter search. However, in accordance with pre-existing hyperparameters, such as the learning rate and weight decay, finding the best combination of hyperparameters requires an exponentially large search space and experiments ( Pfisterer et al., 2021 ; Falkner et al., 2018 ) . The difficulty in finding the best combination of hyperparameters becomes a further serious problem when training a large model ( Yang et al., 2021 ) . To address the difficulty of temperature tuning, this study proposes a novel method to adopt InfoNCE loss without any temperature. Specifically, we replace the division by temperature with a scaled log-odds mapping, or equivalently the inverse hyperbolic tangent function. By injecting few lines of code into the existing design pattern of InfoNCE loss, we can enjoy temperature-free contrastive learning without extensive experiments to search for the best temperature value. To the best of our knowledge, this study is the first in a direction to completely remove the division by temperature scheme in InfoNCE loss to eliminate the need to search for its valid value. Apart from the temperature-free property, we observed that the proposed method also brought performance gains in contrastive learning. Through theoretical analysis, we discover that the existing division by temperature scheme is prone to vanishing gradient, which hinders optimization. We detail this behavior and analyze why the minimization of the InfoNCE loss is significantly sensitive to temperature value (Section 3 ). By contrast, we prove that the proposed method ensures alive gradients during training, which facilitates gradient descent optimization (Section 4 ). Our comprehensive experiments demonstrate that the proposed method—without temperature tuning—yields a promising performance that is comparable to, or even better than, the existing scheme of division by temperature (Section 5 ). 2 Contrastive Learning With InfoNCE Loss 2.1 Background Given a mini-batch of data \mathbf{x}_{i} with i\in\{1,2,\cdots,N\} , contrastive learning ( Dosovitskiy et al., 2016 ) obtains their different views \tilde{\mathbf{x}}_{i} via transformation such as data augmentation (Figure 1 ). Using a model f such as a deep neural network, their embedding vectors f(\mathbf{x}_{i}) and f(\tilde{\mathbf{x}}_{i}) are obtained. The InfoNCE loss computes cross-entropy with softmax using the scaled similarity scores of possible pairs within a mini-batch ( van den Oord et al., 2018 ; Chen et al., 2020 ) . The positive pair should exhibit higher similarity because the representations are obtained from merely different views of the same instance. By contrast, negative pairs should exhibit no similarity because they are arbitrary pairs within a mini-batch ( Chen et al., 2017 ) . In other words, the InfoNCE loss indicates cross-entropy -\sum_{i}y_{i}\log{\hat{y}_i} that applies one-hot label y_{i} describing the correctness of pairs and penalizes a case if \hat{y}_{i} , the softmax output from the scaled similarity score, is low for a positive pair. Learning this matching task enables the model to capture the nature of the dataset and preserved structure in data augmentation, thereby acquiring a generic representation of data ( Joulin et al., 2016 ) . Indeed, several studies have reported that the acquired representation from contrastive learning achieves outstanding performance, even when compared with counterparts from supervised learning; the learned representation has been reported to work with a simple classifier, such as linear or nearest neighborhood ( Chen et al., 2020 ; He et al., 2020 ) . The original InfoNCE loss, proposed by van den Oord et al. (2018) , compares the dot product of two vectors across a mini-batch. Additionally, Wu et al. (2018) incorporated a division

Read the complete text