Contrastive Predictive Coding Done Right for Mutual Information Estimation

J. Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, Gregory W. Wornell · 2025

Read Contrastive Predictive Coding Done Right for Mutual Information Estimation by J. Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, Gregory W. Wornell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25983.txt.

Introduction

J. Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, and Gregory W. Wornell argue that a popular objective for learning representations should not be used as it stands to estimate mutual information. Mutual information measures how much knowing one variable tells about another. The InfoNCE objective comes from contrastive learning, where a model learns to pick the matching pair among distractors, and its link to mutual information is indirect.

Ryu and the team add a simple fix, an auxiliary anchor class, which makes density ratio estimates consistent and yields a plug in estimator with much lower bias. They also place the method in a wider family based on proper scoring rules. Tests show the anchored version estimates mutual information best, though it does not improve downstream representation learning.

Catalog record

CREATORJ. Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, Gregory W. Wornell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrastive Predictive Coding Done Right for Mutual Information Estimation J. Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, Gregory W. Wornell Abstract The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce a simple modification, which we refer to as InfoNCE- anchor , for accurate MI estimation. Our modification introduces an auxiliary anchor class, enabling consistent density ratio estimation and yielding a plug-in MI estimator with significantly reduced bias. Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCE-anchor as a special case when the log score is employed. This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and f f -divergence variants, under a single principled framework. Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in self-supervised representation learning experiments, we find that the anchor does not improve the downstream task performance. These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios. 1 Introduction Contrastive learning has become a cornerstone of modern unsupervised representation learning, powering advances in computer vision ( Chen et al. 2020 ) , natural language processing ( Mikolov et al. 2013 ; Levy & Goldberg 2014 ) , and beyond ( Jaiswal et al. 2020 ; Hu et al. 2024 ) . One of the key ingredients in many contrastive methods is the InfoNCE objective ( van den Oord et al. 2018 ) . While originally proposed as a representation learning framework, van den Oord et al. 2018 noted that the InfoNCE objective can be used to evaluate mutual information (MI), interpreting the objective as a variational bound on MI; see ( van den Oord et al. 2018 , Appendix A) . This interpretation has led to its widespread use for MI estimation, e.g., ( Poole et al. 2019 ; Song & Ermon 2020a ; Gowri et al. 2024 ; Lee & Rhee 2024 ) . It is also widely known, however, that InfoNCE often yields a rather loose bound on MI ( Poole et al. 2019 ; Tschannen et al. 2020 ) . As a result, the InfoNCE estimator is generally considered a low-variance but high-bias MI estimator ( van den Oord et al. 2018 ) . Although several proposals have been made to address this issue since its inception, the effectiveness (i.e., the low-variance property) and the limitation (i.e., the high bias) of InfoNCE remain poorly understood. In this paper, we clarify the operational meaning of the InfoNCE objective, by showing that the objective should be understood as a variational lower bound of a statistical divergence different from the mutual information. Building on this clarification, we establish a sharp characterization of its relationship to the Kullback–Leibler (KL) divergence, revealing why the InfoNCE objective should not be regarded as a direct estimate of MI. We further argue that InfoNCE can be viewed as a density ratio estimation objective, while the critic (or its exponentiated form) estimates the density ratio \frac{p(x,y)}{p(x)p(y)} only up to an arbitrary function C(y) , rendering it unsuitable for use in a plug-in estimator. To address this limitation, we introduce a simple modification to the variational objective, which we call InfoNCE-anchor , corresponding to an alternative divergence. In the new framework, the inclusion of an anchor enables the (exponentiated) critic to estimate the density ratio \frac{p(x,y)}{p(x)p(y)} directly. This adjustment facilitates consistent density ratio estimation and yields a plug-in MI estimator that retains the low variance of InfoNCE while significantly reducing its bias. See Figure 1 for a quick comparison with existing estimators, where the new plug-in estimator based on InfoNCE-anchor demonstrates low-bias, low-variance performance. We generalize our framework using tools from statistical decision theory, showing that InfoNCE-anchor corresponds to the log score , a canonical example of a proper scoring rule. This insight reveals that many contrastive objectives, including NCE, InfoNCE, and certain f -divergence variants, can be unified under a single principled framework of density ratio estimation using proper scoring rules. Empirically, we show that estimators induced by the log score yields state-of-the-art MI estimates across a range of settings. In contrastive representation learning tasks, however, we find that multiple scoring rules yield similar performance, suggesting that MI estimation is not the primary driver of contrastive learning success. Instead, our results support that contrastive learning benefits from learning structured density ratios, regardless of whether the objectives are accurate MI estimators. While InfoNCE has played a canonical role in contrastive representation learning, its rather loose association to MI estimation has historically fostered the misconception that representation learning is essentially about maximizing MI; see, e.g., ( Bachman et al. 2019 ; Wu et al. 2020 ) . This paper clarifies why such an interpretation can be limiting and imprecise, and that contrastive representation learning should instead be framed as representation learned to factorize pointwise MI (PMI) \log\frac{p(x,y)}{p(x)p(y)} , or pointwise dependence (PD) \frac{p(x,y)}{p(x)p(y)} . All proofs can be found in Appendix F . Logarithms in this paper are in base 2 and thus KL divergence and MI are in bit. 2 Preliminaries In this paper, we first review different types of variational-bounds-based MI estimators in the literature, and provide a taxonomy. We then delve into the InfoNCE estimator, and show why the InfoNCE estimator should not be considered as a direct estimate for MI. 2.1 Types of Information Estimators Existing variational-bound-based MI estimators can be categorized into three principal categories as follows, based on the relationship of their optimization objectives to the final metrics to compute MI. Table 1 summarizes the representative estimators. \bullet Type 1: Training and evaluation with a single variational lower bound. These estimators optimize a tractable lower bound on the MI and use the same bound for evaluation. Examples include DV ( Donsker & Varadhan 1975 ) , NWJ ( Nguyen et al. 2010 ) , and InfoNCE ( van den Oord et al. 2018 ) . While conceptually simple and natural, McAllester & Stratos 2020 showed that any distribution-free high-probability lower bound of MI is upper bounded by \log N , where N is the sample size. This result implies that variational lower-bound–based sample estimates of MI suffer from an inherent limitation. \bullet Type 2: Training with a variational lower bound, evaluation by plugging-in to another variational lower bound. These estimators optimize a surrogate objective, often smoothed or stabilized for optimization, and then estimate MI via plug-in to a different bound such as DV or NWJ. Examples include MINE ( Belghazi et al. 2018 ) , JS ( Hjelm et al. 2019 ) , and SMILE ( Song & Ermon 2020a ) . These methods often improve stability during training, but introduce additional sources of mismatch between optimization and evaluation. Note that the critique of McAllester & Stratos 2020 still applies to this type of estimators. \bullet Type 3: Training with a variational lower bound, evaluation with a plug-in estimator. These estimators target to learn the density ratio \frac{p(x,y)}{p(x)p(y)} directly and compute MI by plugging the estimated score

Read the complete text