UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels

Hangke Sui, Yuqing Wang, Minh N. Do, Equal Contribution · 2026

Read UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels by Hangke Sui, Yuqing Wang, Minh N. Do, Equal Contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16678.txt.

Introduction

Sui, Wang and Do propose UniCon, a unified framework for contrastive alignment via kernels. Contrastive objectives power mixed models, yet training stays slow under long random tuning. UniCon spans linear and curved coders, one-to-one and many-to-many matches. A contrast score weight matrix enables closed-form solutions swapping batch backprop with exact updates. Through reproducing kernel Hilbert spaces, UniCon unifies matching and links it to spectral methods. Tests on fake, single-mode, multimodal and zero-shot tasks show efficiency gains with breadth and strong results.

Catalog record

CREATORHangke Sui, Yuqing Wang, Minh N. Do, Equal Contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98034
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels Hangke Sui, Yuqing Wang, Minh N. Do, Equal Contribution Abstract Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization. We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders as well as one-to-one and many-to-many alignments. At its core, UniCon introduces the contrastive similarity weight matrix S ⁡ ( γ ) S(\gamma) , which enables closed-form global solutions that provably replace minibatch back-propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods. To validate the theory, we conduct experiments on synthetic, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance. 1 Introduction Learning semantically aligned representations across different modalities, such as vision and language, has long been a central goal in machine learning ( 31 ; 42 ) . In particular, Multimodal Contrastive Learning (MMCL) ( 18 ) has recently achieved remarkable success in zero-shot classification ( 34 ; 20 ) , cross-modal retrieval ( 28 ; 7 ) , and general visual understanding ( 43 ; 27 ) . These models typically train modality-specific encoders, e.g., a vision encoder and a language encoder, such that paired inputs are mapped to nearby representations in a shared space, while unpaired inputs are mapped far apart. At the heart of MMCL lies contrastive representation learning ( 4 ; 9 ; 41 ; 32 ; 3 ; 34 ) . Its versatility has made it a core component across diverse domains, including NLP ( 6 ; 19 ) , bioimaging ( 36 ; 44 ; 12 ) , recommendation ( 54 ; 58 ; 22 ; 56 ) , and graph learning ( 24 ; 57 ) . The typical pipeline involves feature extraction via deep encoders and the optimization of contrastive loss. Despite the impressive empirical performance of contrastive learning across vision, language, and multimodal domains, the theoretical foundations underlying its success remain only partially understood. There are works on the analysis of loss and training dynamics ( 48 ; 46 ; 13 ) , provably guarantee of the model generalization ( 14 ; 15 ; 47 ; 33 ) , duality between contrastive and non-contrastive method ( 45 ; 2 ) . A growing body of theoretical work has sought to formalize contrastive learning ( 37 ; 45 ; 21 ; 52 ) , often by simplifying the problem to single-modality settings. Recent advancements in contrastive learning have introduced novel loss functions and analytical frameworks to enhance representation quality and training efficiency. ( 55 ; 51 ; 39 ) . Analytical studies have examined contrastive learning from different perspectives. For example, 40 interpret the CLIP loss through the lens of optimal transport; while 46 ; 30 analyze multimodal contrastive learning using SVD- and PCA-based formulations, showing that, under linear encoders , contrastive loss minimization reduces to calculating a weighted covariance matrix. Yet, this insight has not been translated into nonlinear encoder settings and practical implementations. We introduce UniCon , a Unified Framework for Efficient Contrastive Alignment via Kernels , which leverages a structured contrastive similarity weight matrix S(\gamma) to directly solve contrastive objectives. As illustrated in Figure 1 , UniCon departs from gradient-based training and instead: (i) in the linear setting ( 30 ) , solves a single spectral decomposition yielding optimal encoder matrices in closed form (ii) in a general nonlinear setting, provides a unified kernelized framework that enables fast alignment via implicit representation inference. Our key contributions are as follows: • Theoretically, we provide a kernel-based perspective that unifies linear and nonlinear encoders, showing that minimizing contrastive loss is equivalent to a spectral update. This leads to a provably optimal solution in closed form and connects contrastive learning to spectral methods. • Beyond one-to-one matching, our framework generalizes to many-to-many alignment, broadening the applicability of contrastive alignment. • Empirically, we demonstrate that UniCon converges fast and achieves competitive or superior performance across synthetic , unimodal (CIFAR-10), and multimodal (Flickr30k,MSCOCO) and zero-shot transfer (image-text retrieval), offering up to 461× speed-up over minimizing CLIP loss by stochastic gradient descent. Figure 1: Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon) . Starting from paired inputs, UniCon builds a contrastive similarity weight matrix S(\gamma) using hyper-spherical similarities, then computes either (i) a closed-form spectral update in the linear case (orange) or (ii) a kernelized solution in the nonlinear case (blue). 2 Background Contrastive Representation. Contrastive learning ( 4 ; 9 ; 41 ; 32 ; 3 ; 34 ) leverages paired inputs as a form of supervision. The central goal is to learn a representation space where positive (matching) pairs are mapped to nearby embeddings, while negative (non-matching) pairs are pushed apart. Learning representations on the hypersphere leads to better performance than in Euclidean space ( 49 ) , as it avoids conflicting forces between attractive and repulsive gradients. ( 50 ) further shows that the distribution of representations on the unit hypersphere is encouraged to be uniform. Contrastive Loss Contrastive loss ( 10 ) was first proposed in Siamese networks to pull together positive pairs and push apart negatives. The formulation was later unified under the InfoNCE loss ( 32 ) , which serves as the basis for many self-supervised methods, including SimCLR ( 3 ) . Supervised Contrastive (SupCon) loss ( 23 ) extended contrastive learning to the supervised setting, where each anchor can have multiple positive samples from the same class. Building on the general family of contrastive losses introduced by ( 46 ) , ( 30 ) showed that these multimodal contrastive objectives can be connected to singular value decomposition (SVD) under linear setting. Kernel Method Given a positive finite measure \mu over a parameter space \Theta , we define a kernel k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R} by k(x,\tilde{x})=\langle\phi(x;\theta),\phi(\tilde{x};\theta)\rangle_{\mu}:=\int_{\Theta}\phi(x;\theta)\phi(\tilde{x};\theta)\,d\mu(\theta) , which induces a Reproducing Kernel Hilbert Space (RKHS). Any function f in this space admits the representation f(x)=\sum_{j=1}^{m}w_{j}\,k(x,x_{j}),\ w_{j}\in\mathbb{R} . Kernels are commonly used to learn representations ( 26 ; 25 ) , as they capture the relative structure among samples, critical for many learning algorithms ( 1 ; 17 ; 29 ; 8 ) . 3 Methodology We observe N paired samples \{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{N} , where \mathbf{x}_{i}\in\mathbb{R}^{d_{1}} and \mathbf{y}_{i}\in\mathbb{R}^{d_{2}} . The objective of contrastive learning is to learn encoders \mathbf{f}_{\theta_{1}}:\mathbb{R}^{d_{1}}\to\mathbb{R}^{r} and \mathbf{f}_{\theta_{2}}:\mathbb{R}^{d_{2}}\to\mathbb{R}^{r} with modality-specific parameters \theta_{1} and \theta_{2} , such that paired inputs are mapped to similar representations in a shared r - dimensional embedding space, while non-paired inputs remain dissimilar. In the sections that follow, we first formalize the general contrastive learning framework, then analyze it under a linear representation setting, and finally unify our analysis spanning nonlinear encoders in RKHS. The proof can be found in the Appendix B . Definition 1 (Hyper-spherical similarity) . Define the similarity between \mathbf{x}_{i} and \mathbf{y}_{i} as the inner product on the hyper-sphere: s_{ij}=\langle\mathbf{f}_{\theta_{1}}(\mathbf{x}_{i}),\mathbf{f}_{\theta_{2}}(\mathbf{y}_{j})\rangle_{\mathbb{S}^{r-1}\subset\mathbb{R}^{r}}=\left\langle\frac{\mathbf{f}_{\theta_{1}}(\mathbf{x}_{i})}{\left\|\mathbf{f}_{\theta_{1}}(\mathbf{x}_{i})\right\|_{2}},\frac{\mathbf{f}_{\theta_{2}}(\mathbf{y}_{j})}{\left\|\mathbf{f}_{\theta_{2}}(\mathbf{y}_{j})\right\|_{2}}\right\rangle_{\mathbb{R}^{r}}=\frac{\mathbf{f}_{\theta_{1}}^{\top}(\mathbf{x}_{i})\mathbf{f}_{\theta_{2}}(\mathbf{y}_{j})}{\left\|\mathbf{f}_{\theta_{1}}(\mathbf{x}_{i})\right\|_{2}\left\|\mathbf{f}_{\theta_{2}}(\mathbf{y}_{j})\right\|_{2}}. (1) Definition 2 (Generalized contrastive loss) . Given N paired samples \{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{N} , we write the similarity matrix [s_{ij}] . With \phi,\psi:\mathbb{R}\!\to\!\mathbb{R}_{+} monotonically increasing, scaling factor \nu\!\geq\!1 , and weights \epsilon_{ij}\!\in\![0,1] , the bidirectional general contrastive loss

Read the complete text