I-Con: A Unifying Framework for Representation Learning

Shaden Alshammari, John Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton, MIT Google Microsoft https://aka.ms/i-con · 2025

Read I-Con: A Unifying Framework for Representation Learning by Shaden Alshammari, John Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton, MIT Google Microsoft https://aka.ms/i-con (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16929.txt.

Introduction

Shaden Alshammari and co-authors present one equation that unifies many ways machines learn representations, in this 2025 paper. The field has spawned dozens of loss functions for clustering, contrastive learning, and more. Their framework, I-Con, shows these methods all shrink the same information gap.

Each method, in their view, pulls a learned conditional distribution toward a supervisory one. The lens exposes a shared geometry beneath more than twenty known approaches, with proofs for each link. Mixing tricks across families then yields new losses. Their derived image classifier beat the prior best on a major benchmark by eight points, and the frame also gives principled debiasing tools.

Catalog record

CREATORShaden Alshammari, John Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton, MIT Google Microsoft https://aka.ms/i-con
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

I-Con: A Unifying Framework for Representation Learning Shaden Alshammari, John Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton, MIT Google Microsoft https://aka.ms/i-con Abstract As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners. Figure 1: A “periodic” table of representation learning methods unified by the I-Con framework. By choosing different types of conditional probability distributions over neighbors, I-Con generalizes over 23 commonly used representation learning methods. 1 Introduction Over the past decade the field of representation learning has flourished, with new techniques, architectures, and loss functions emerging daily. These advances have powered state-of-the-art models in vision, language, and multimodal learning, often with minimal human supervision. Yet as the field expands, the diversity of loss functions makes it increasingly difficult to understand how different methods relate, and which objectives are best suited for a given task. In this work, we introduce a general mathematical framework that unifies a wide range of representation learning techniques spanning supervised, unsupervised, and self-supervised approaches under a single information-theoretic objective. Our framework, Information Contrastive Learning (I-Con) , reveals that many seemingly disparate methods including clustering, spectral graph theory, contrastive learning, dimensionality reduction, and supervised classification are all special cases of the same underlying loss function. While prior work has identified isolated connections between subsets of representation learning methods, typically linking only two or three techniques at a time ( Sobal et al., 2025 ; Hu et al., 2023 ; Yang et al., 2022 ; Böhm et al., 2023 ; Balestriero & LeCun, 2022 ) , I-Con is the first framework to unify over 23 distinct methods under a single objective. This unified perspective not only clarifies the structure of existing techniques but also provides a strong foundation for transferring ideas and improvements across traditionally separate domains. Using I-Con, we derive new unsupervised loss functions that significantly outperform previous methods on standard image classification benchmarks. Our key contributions are: • We introduce I-Con , a single information-theoretic loss that generalizes several major classes of representation learning. • We prove 15 theorems showing how diverse algorithms emerge as special cases of I-Con. • We use I-Con to design a debiasing strategy that improves unsupervised ImageNet-1K accuracy by +8% , with additional gains of +3% on CIFAR-100 and +2% on STL-10 in linear probing. 2 Related Work Representation learning spans a wide range of methods for extracting structure from complex data. We review approaches that I-Con builds upon and generalizes. For comprehensive surveys, see ( Le-Khac et al., 2020 ; Bengio et al., 2013 ; Weng, 2021 ) . Feature Learning aims to derive informative low-dimensional embeddings using supervisory signals such as pairwise similarities, nearest neighbors, augmentations, class labels, or reconstruction losses. Classical methods like PCA ( Pearson, 1901 ) and MDS ( Kruskal, 1964 ) preserve global structure, while UMAP ( McInnes et al., 2018 ) and t-SNE ( Hinton & Roweis, 2002 ; Van der Maaten & Hinton, 2008 ) focus on local topology by minimizing divergences between joint distributions. I-Con adopts a similar divergence-minimization view. Contrastive learning approaches such as SimCLR ( Chen et al., 2020a ) , CMC ( Tian et al., 2020 ) , CLIP ( Radford et al., 2021 ) , and MoCo v3 ( Chen* et al., 2021 ) use positive and negative pairs, often built via augmentations or aligned modalities. I-Con generalizes these losses within a unified KL-based framework, highlighting subtle distinctions between them. Supervised classifiers (e.g., ImageNet models ( Krizhevsky et al., 2017 ) ) also yield effective features, which I-Con recovers by treating class labels as discrete contrastive points, bridging supervised and unsupervised learning. Clustering methods uncover discrete structure through distance metrics, graph partitions, or contrastive supervision. Algorithms like k-Means ( Macqueen, 1967 ) , EM ( Dempster et al., 1977 ) , and spectral clustering ( Shi & Malik, 2000 ) are foundational. Recent methods, including IIC ( Ji et al., 2019 ) , Contrastive Clustering ( Li et al., 2021 ) , and SCAN ( Gansbeke et al., 2020 ) , leverage invariance and neighborhood structure. Teacher-student models such as TEMI ( Adaloglou et al., 2023 ) and EMA-based architectures ( Chen et al., 2020b ) enhance clustering further. I-Con encompasses these by aligning a clustering-induced joint distribution with a target distribution derived from similarity, structure, or contrastive signals. Unifying Representation Learning has been explored through connections between contrastive learning and t-SNE ( Hu et al., 2023 ; Böhm et al., 2023 ) , equivalences between contrastive and cross-entropy losses ( Yang et al., 2022 ) , and relations between spectral and contrastive methods ( Balestriero & LeCun, 2022 ; Sobal et al., 2025 ) . Other efforts, like Bayesian grammar models ( Grosse et al., 2012 ) , offer probabilistic perspectives. Tschannen et al. ( Tschannen et al., 2019 ) emphasized estimator and architecture design in mutual information frameworks but stopped short of broader unification. While prior work links subsets of these methods, I-Con, to our knowledge, is the first to unify supervised, contrastive, clustering, and dimensionality reduction objectives under a single loss. This perspective clarifies their shared structure and opens paths to new learning principles. (a) High-level I-Con architecture. (b) Illustrative examples of distribution families for p_{\theta} or q_{\phi} . Figure 2: Overview of the I-Con framework . (a) Alignment of learned and supervisory distributions. (b) Common distribution families in I-Con’s formulation. 3 Methods The I-Con framework unifies multiple representation learning methods under a single loss function: minimizing the average KL divergence between two conditional “neighborhood distributions” that define transition probabilities between data points. This information-theoretic objective generalizes techniques from clustering, contrastive learning, dimensionality reduction, spectral graph theory, and supervised learning. By varying the construction of the supervisory distribution and the learned distribution, I-Con encompasses a broad class of existing and novel methods. We introduce I-Con and demonstrate its ability to unify techniques from diverse areas and orchestrate the transfer of ideas across different domains, leading to a state-of-the-art unsupervised image classification method. 3.1 Information Contrastive Learning Let i,j\in\mathcal{X} be elements of a dataset \mathcal{X} , with a probabilistic neighborhood function p(j|i) defining a transition probability. To ensure valid probability distributions, p(j|i)\geq 0 and \int_{j\in\mathcal{X}}p(j|i)=1 . We parameterize this distribution by \theta\in\Theta , to create a learnable function p_{\theta}(j|i) . Similarly, we define another distribution q_{\phi}(j|i) parameterized by \phi\in\Phi

Read the complete text