Invariant Representations via Wasserstein Correlation Maximization
Keenan Eikenberry, Lizuo Liu, Yoonsang Lee · 2025
Read Invariant Representations via Wasserstein Correlation Maximization by Keenan Eikenberry, Lizuo Liu, Yoonsang Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11702.txt.
Introduction
Keenan Eikenberry, Lizuo Liu, and Yoonsang Lee show how to bolt invariance onto a trained neural network without retraining it, in a 2025 paper. The goal is a network that ignores rotations or noise yet answers unchanged on plain inputs. They formalize the task through augmented encoders, then propose two training losses built on optimal transport ideas.
A small one-layer adapter, trained with either loss, sits between the frozen network and its classifier. On a standard image set, rotation accuracy jumped from 71 to 94 percent, and noise accuracy from 58 to 86 percent. The base network's weights never moved, which keeps the method cheap.
Catalog record
| CREATOR | Keenan Eikenberry, Lizuo Liu, Yoonsang Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Invariant Representations via Wasserstein Correlation Maximization Keenan Eikenberry, Lizuo Liu, Yoonsang Lee Abstract This work investigates the use of Wasserstein correlation—a normalized measure of statistical dependence based on the Wasserstein distance between a joint distribution and the product of its marginals—for unsupervised representation learning. Unlike, for example, contrastive methods, which naturally cluster classes in the latent space, we find that an (auto)encoder trained to maximize Wasserstein correlation between the input and encoded distributions instead acts as a compressor, reducing dimensionality while approximately preserving the topological and geometric properties of the input distribution. More strikingly, we show that Wasserstein correlation maximization can be used to arrive at an (auto)encoder—either trained from scratch, or else one that extends a frozen, pretrained model—that is approximately invariant to a chosen augmentation, or collection of augmentations, and that still approximately preserves the structural properties of the non-augmented input distribution. To do this, we first define the notion of an augmented encoder using the machinery of Markov-Wasserstein kernels. When the maximization objective is then applied to the augmented encoder, as opposed to the underlying, deterministic encoder, the resulting model exhibits the desired invariance properties. Finally, besides our experimental results, which show that even simple feedforward networks can be imbued with invariants or can, alternatively, be used to impart invariants to pretrained models under this training process, we additionally establish various theoretical results for optimal transport-based dependence measures. Code is available at https://github.com/keenan-eikenberry/wasserstein_correlation_maximization . 1 Introduction Unsupervised representation learning aims to extract meaningful features from data without relying on labeled examples. One common approach is to train an encoder E:X\to Z to maximize the mutual information (MI) between an input distribution \mu on some space X and the latent distribution of learned representations E_{\sharp}\mu on some space Z , where E_{\sharp}\mu is the usual pushforward measure. This is the essence of the InfoMax principle first introduced by Linsker [ 31 ] , and there have been numerous modern incarnations of the idea, especially in the context of augmentation-based representation learning, where many approaches, including the popular InfoNCE contrastive loss [ 22 , 40 ] , amount to an MI-maximization objective between the data and its encoded, augmented views, or, alternatively, between different encoded, augmented views in the latent space alone; see, for example, [ 3 , 23 , 24 , 40 , 46 , 49 , 56 ] , and, in particular, see [ 52 ] for a unified perspective on many of these works. Intuitively, maximizing MI between a data distribution and the induced distribution of latent codes should produce representations that capture significant structure in the data. Despite this intuitive appeal, though, mutual information-based methods face several challenges. First, MI estimation is computationally difficult in high-dimensions, and, in practice, neural estimators, as defined in [ 5 ] or [ 44 ] , for example, are used to estimate lower bounds on the quantity. However, as shown in [ 35 ] , high-confidence, distribution-free lower bounds of any type have exponential sample complexity in the size of the bound. More strikingly, Tschannen et al. [ 52 ] show that maximizing MI does not necessarily lead to effective representations for various downstream tasks, such as, for example, classification with linear classifiers, since information preservation alone is not enough to ensure a useful latent geometry. More broadly, their findings show that the success of MI-based objectives involves a subtle interplay between encoder architectures and MI estimators. The above limitations naturally raise the question of whether there are alternative information-based objectives that might be better suited for unsupervised representation learning. Recent work in statistical inference and representation learning has established Wasserstein dependence as a theoretically appealing, optimal transport-based measure of statistical dependence. It offers a geometry-aware framework for various statistical and machine learning tasks, and it indeed shows promise as a general-purpose substitute for mutual information. Significant papers on the subject include [ 36 , 39 ] and [ 42 ] ; see Subsection 1.1 below for a more comprehensive list. In brief, Wasserstein dependence is defined as the Wasserstein distance between a joint distribution and the product of its marginals. That is, it is defined in the same way as mutual information, but the Kullback-Leibler (KL) divergence is replaced with the Wasserstein distance. This substitution introduces flexibility through the choice of an underlying cost function for the Wasserstein distance and yields a dependence measure that quantifies the difficulty of transforming samples from the product distribution into samples from the joint distribution. Many variants of Wasserstein dependence exist, including normalized forms, where the quantity is then typically called Wasserstein correlation, rather than dependence, as well as versions based on the adapted Wasserstein distance [ 54 ] . Additionally, as with all optimal transport-based quantities, entropic regularization and slicing techniques are commonly used to give computationally efficient implementations [ 29 , 32 ] . Our own contributions to this body of work are as follows: 1. First, we show that (sliced) Wasserstein correlation maximization can be used to train (deterministic) (auto)encoders that approximately preserve the topological and geometric properties of the input distribution (as measured by persistent homology, spectral distances, and heat kernel distances). 2. Second, we define the notion of (probabilistic) augmented encoders using the machinery of Markov-Wasserstein kernels, which are Markov kernels that take values in a Wasserstein space and that come equipped with a generalized L^{p} metric. 3. Third, we show that when the Wasserstein correlation maximization objective is applied to the augmented encoder, instead of the underlying deterministic encoder, we arrive at an (auto)encoder that is approximately invariant to a chosen augmentation, or collection of augmentations, and that still approximately preserves the structural properties of the non-augmented input distribution. In particular, we show that we can improve invariance properties in large-scale, pretrained models with the addition of only a few feedforward layers trained on their feature spaces. 4. Finally, we establish various theoretical results that contribute towards a better understanding of optimal transport-based dependence measures and their use in representation learning. The rest of the paper is organized as follows: In Subsection 1.1 below, we highlight work related to our own. Our main contributions are presented in Section 2 , and all necessary background material can be found in Appendix A . Finally, experimental results are given in Section 3 , and suggestions for future work are given in Section 4 . 1.1 Related Work This paper builds on a growing body of work on the theory and applications of Wasserstein dependence and its variants [ 29 , 32 , 36 , 37 , 39 , 42 , 54 ] . We were specifically influenced by [ 32 ] and [ 42 ] , which both use transport-based dependence measures for machine learning applications using an entropy-regularized approach and a Kantorovich duality-based approach, respectively. We were also influenced by [ 30 ] , which uses the Hilbert-Schmidt independence criterion as a substitute for mutual information maximization. More generally, our work relates to other research on the many uses of optimal transport-based quantities in representation learning, generative modeling, and adversarial robustness. This field is too large to survey exhaustively, but papers with methodologies or technical content similar to our own include [ 9 , 15 , 27