Your contrastive learning problem is secretly a distribution alignment problem Zihao Chen, Chi-Heng Lin, Ran Liu, Jingyun Xiao, Eva L. Dyer * Abstract Despite the success of contrastive learning (CL) in vision and language, its theoretical foundations and mechanisms for building representations remain poorly understood. In this work, we build connections between noise contrastive estimation losses widely used in CL and distribution alignment with entropic optimal transport (OT). This connection allows us to develop a family of different losses and multistep iterative variants for existing CL methods. Intuitively, by using more information from the distribution of latents, our approach allows a more distribution-aware manipulation of the relationships within augmented sample sets. We provide theoretical insights and experimental evidence demonstrating the benefits of our approach for generalized contrastive alignment . Through this framework, it is possible to leverage tools in OT to build unbalanced losses to handle noisy views and customize the representation space by changing the constraints on alignment. By reframing contrastive learning as an alignment problem and leveraging existing optimization tools for OT, our work provides new insights and connections between different self-supervised learning models in addition to new tools that can be more easily adapted to incorporate domain knowledge into learning. \DeclareCaptionLabelFormat andtable#1 #2 & Table 0 1 Introduction In machine learning, the availability of vast amounts of unlabeled data has created an opportunity to learn meaningful representations without relying on costly labeled datasets [ 26 , 52 , 27 ] . Self-supervised learning has emerged as a powerful solution to this problem, allowing models to leverage the inherent structure in data to build useful representations. Among self-supervised methods, contrastive learning (CL) is widely adopted for its ability to create robust representations by distinguishing between similar (positive) and dissimilar (negative) data pairs. With success in fields like image and language processing [ 8 , 46 ] , contrastive learning now also shows promise in domains where cross-modal, noisy, or structurally complex data make labeling especially challenging [ 34 , 56 , 10 ] . Traditional contrastive learning methods primarily aim to bring positive pairs—often augmentations of the same sample—closer together in representation space. While effective, this approach often struggles with real-world challenges such as noise in views, variations in data quality, or shifts introduced by complex transformations, where positive pairs may not perfectly align. Additionally, in tasks requiring domain generalization, aligning representations across diverse domains (e.g., variations in style or sensor type) is critical but difficult to achieve with standard contrastive learning, which typically lacks mechanisms for incorporating domain-specific relationships. These limitations highlight the need for a more flexible approach that can adapt alignment strategies based on the data structure, allowing for finer control over similarity and dissimilarity among samples. To address this challenge, we introduce a novel generalized contrastive alignment (GCA) framework, which reinterprets contrastive learning as a distributional alignment problem. Our method allows flexible control over the alignment of samples by defining a target transport plan, \mathbf{P}_{tgt} , that serves as a customizable alignment guide. For example, setting \mathbf{P}_{tgt} to resemble a diagonal matrix encourages each positive to align primarily with itself or its augmentations, thereby reducing the effect of noise between views. Alternatively, we can incorporate more complex constraints, such as weighting alignments based on view quality or enforcing partial alignment structures where noise or data heterogeneity is prevalent. This flexibility enables GCA to adapt effectively to a wide range of tasks, from simple twin view alignments to scenarios with noisy or variably aligned data. Our approach also bridges connections between GCA and established methods, such as InfoNCE (INCE) [ 38 ] , Robust InfoNCE (RINCE) [ 12 ] , and BYOL [ 22 ] , demonstrating that these can be viewed as iterative alignment objectives with Bregman projections [ 6 , 21 ] . This perspective allows us to systematically analyze and improve uniformity within the latent space, a property that enhances representation quality and ultimately boosts downstream classification performance. We validate our method through extensive experiments on both image classification and noisy data tasks, demonstrating that GCA’s unbalanced OT (UOT) formulations improve classification performance by relaxing our constraints on alignment. Our results show that GCA offers a robust and versatile framework for contrastive learning, providing flexibility and performance gains over existing methods and presenting a promising approach to addressing different sources of variability in self-supervised learning. The contributions of this work include: • A new framework called generalized contrastive alignment (GCA), which reinterprets standard contrastive learning as a distributional alignment problem, using optimal transport to provide flexible control over alignment objectives. This approach allows us to derive a novel class of contrastive losses and algorithms that adapt effectively to varied data structures and build customizable transport plans. • We present GCA-UOT, a contrastive learning method that achieves strong performance on standard augmentation regimes and excels in scenarios with more extreme augmentations or data corrupted by transformations. GCA-UOT leverages unbalanced transport to adaptively weight positive alignments, enhancing robustness against view noise and cross-domain variations. • We provide theoretical guarantees for the convergence of our GCA-based methods and show that our alignment objectives improve representation quality by enhancing the uniformity of negatives and strengthening alignment within positive pairs. This leads to more discriminative and resilient representations, even in challenging data conditions. • Empirically, we demonstrate the effectiveness of GCA in both image classification and domain generalization tasks. Through flexible, unbalanced OT-based losses, GCA achieves superior classification performance and adapts alignment to include domain-specific information where relevant, without compromising classification accuracy in domain generalization. 2 Background 2.1 Contrastive learning Contrastive learning (CL) is a representation learning methodology that uses positive and negative pairs to define similarity in the latent space. Let \mathcal{D}=\{{\bf x}_{i}\}_{i=1}^{N} denote our dataset. For each sample {\bf x}_{i} in a batch of training data with size B , we create two augmented copies {\bf x}^{\prime}_{i} and {\bf x}^{\prime\prime}_{i} independently, i.e., {\bf x}_{i}^{\prime}=\psi({\bf x}_{i}) where \psi is a randomly drawn augmentation function from some augmentation class \mathcal{A} and likewise for {\bf x}^{\prime\prime}_{i} . The ({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{i}) is called a positive pair of {\bf x}_{i} while ({\bf x}_{i}^{\prime},{\bf x}^{\prime\prime}_{j}) is treated as a negative pair for any j\neq i . One of the most widely used formulations of the CL problem, InfoNCE (INCE) [ 8 ] , seeks to maximize the negative log probability that a sample is correctly classified as \mathcal{L}_{\text{INCE}}=-\log\bigg(\frac{e^{s_{ii}}}{e^{s_{ii}}+\sum_{i\neq j}e^{s_{ij}}}\bigg), (1) where s_{ij}=\varepsilon^{-1}f_{\theta}({{\bf x}_{i}^{\prime}})^{\top}f_{\theta}({\bf x}_{j}^{\prime\prime})/\|f_{\theta}({{\bf x}_{i}^{\prime}})\|\|f_{\theta}({{\bf x}_{j}^{\prime\prime}})\| is the score between augmented samples. Building upon the principles of INCE, SimCLR [ 8 ] and MoCo [ 24 ] are two representative works that form the foundation of contrastive learning methods for visual representation tasks. Alternatively, BYOL [ 22 ] and SimSiam [ 9 ] discard the use of negative samples to avoid large batch size and instead use exponential moving average-based updates to avoid representational collapse. Recent contrastive methods have focused on improving the tolerance to noise in samples to enhance robustness in diverse scenarios [ 13 ] . Among them, Robust INCE (RINCE) is a robust contrastive loss function characterized by its symmetric properties and theoretical resistance to noisy labels [ 47 , 12 ] . Specifically, RINCE provides robustness to noisy views by introducing adjustable parameters \lambda and q [ 12 ] which rebalance the cost of positive and negative views, resulting in the following loss: \mathcal{L}^{\lambda,q}_{\text{RINCE}}=\frac{1}{q}\big(-e^{qs_{ii}}+\lambda^{q}(e^{s_{ii}}+\sideset{}{{}_{i\neq j}}{\sum}e^{s_{ij}})^{q}\big) (2) By optimizing the above loss functions, the encoder f is trained to construct a semantically coherent representation space where positive pairs of samples are positioned nearby, while those negative pairs with divergent semantic attributes are separated [ 57 ] . 2.2 Proximal Operators and Projections To make the connections between different CL losses clearer later, we use the notion of proximal operators. In words, the proximal operator will provide a way to find the closest point in some closed convex set. Formally, we can define the proximal operator as follows. Definition 1 (Proximal Operator) . Let d_{\Gamma}({\bf x},{\bf v})=\Gamma({\bf x})-\Gamma({\bf v})-\langle\nabla\Gamma({\bf v}),{\bf x}-{\bf v}\rangle be a Bregman divergence with a convex function \Gamma . The proximal operator of h:\mathcal{X}\to\mathbb{R}\cup\{+\infty\} is defined for a point {\bf v}\in\mathcal{X} with a closed convex set \mathcal{B}\subseteq\mathcal{X} : ~\text{Prox}_{h,\mathcal{B}}^{d_{\Gamma}}({\bf v})=\arg\min_{{\bf x}\in\mathcal{B}}\left\{h({\bf x})+d_{\Gamma}({\bf x},{\bf v})\right\}. Moreover, we can define the concept of a projection as a special case of the proximal operator when we let h({\bf x}) be an indicator function h_{\mathcal{B}}(x)=\{0,\text{if }x\in\mathcal{B};\infty,\text{if }x\notin\mathcal{B}\} on constraint set \mathcal{B} . See Appendix A.2 for more details. 2.3 Solving Optimal Transport Through Proximal Point Methods Optimal transport (OT) is widely used in characterizing the distance between two collections of samples \{{\bf x}_{i}\}_{i=1}^{B} and \{{\bf y}_{j}\}_{j=1}^{B} with associated measures \mu=\sum_{i=1}^{B}\delta_{{\bf x}_{i}}p_{i}~\text{and}~\nu=\sum_{j=1}^{B}\delta_{{\bf y}_{j}}q_{j} with Dirac delta function \delta_{{\bf x}} and \delta_{\mathbf{y}} on finite support [ 43 ] . Here, p and q are vertices of the \mathbb{R}^{B} simplex defined as \Delta_{B}:=\{v\in\mathbb{R}^{B}:v_{i}\geq 0,\sum_{i=1}^{B}v_{i}=1\} . OT aims to learn a joint coupling matrix, or transport plan {\bf P}\in\mathbbm{R}^{B\times B}_{+} that minimizes the cost of transporting mass encoded by cost matrix {\bf C}\in\mathbbm{R}^{B\times B}_{+} , from one distribution to another. In practice, entropy regularization is used to solve the OT objective, resulting in the following entropy-regularized OT (EOT) objective: \min_{\textbf{P}\in\mathcal{B}}~\langle\textbf{P},\textbf{C}\rangle-\varepsilon H(\textbf{P}),\quad\text{where}~H({\bf P})=-\sum_{ij}{\bf P}_{ij}\log({{\bf P}}_{ij}), (3) where \varepsilon is a user specified parameter that controls the amount of smoothing in the transport plan, and {\bf C}({\bf x},{\bf y})=1-\langle{\bf x},{\bf y}\rangle/\|{\bf x}\|\|{\bf y}\| is often set to encode the cosine similarity between pairs of samples. The Sinkhorn Algorithm and its Interpretation as a Bregman Projection. Solving Equation ( 3 ) could be interpreted as iterative alignment problem on a Hilbert space generated from the kernel {\bf K}_{ij}=\exp(-{\bf C}_{i,j}/\varepsilon) . This alignment problem can be solved through iterative Bregman projections onto the two constraints sets that encode the marginals along the rows and columns [ 3 , 5 , 43 ] : C_{1}^{\mu}\coloneqq\{{\bf P}:{\bf P}\mathbbm{1}_{B}=\mu\},C_{2}^{\nu}\coloneqq\{{\bf P}:{\bf P}^{\top}\mathbbm{1}_{B}=\nu\} (4) The first step of Bregman projection is to find the minimizer {\bf P}^{(1)}=\arg\min\{\varepsilon\text{KL}({\bf P}\|{\bf K}):{\bf P}\mathbbm{1}_{B}=\mu\} by the proximal operator \text{Prox}_{C_{1}^{\mu}}^{\text{KL}}({\bf K}) with Lagrange multiplier f on the row constraint set \mathcal{C}_{1}^{\mu} , and compute its derivatives with respect to {\bf P} with {\bf u}=e^{f/\varepsilon}>0 : \varepsilon\log({\bf P}^{(1)}/{\bf K})-f\mathbbm{1}=0\Rightarrow{\bf P}^{(1)}={\bf u}{\bf K},\quad\langle{\bf P}^{(1)},\mathbbm{1}\rangle=\mu\Rightarrow\langle{\bf u}{\bf K},\mathbbm{1}\rangle=\mu,{\bf u}=\frac{\mu}{{\bf K}\mathbbm{1}} (5) Next, we project {\bf P}^{(1)} onto the column constraint set C_{2}^{\nu} , resulting in {\bf P}^{(2)}\coloneqq\text{Prox}^{\text{KL}}_{C_{2}^{\nu}}({\bf P}^{(1)})={\bf P}^{(1)}\text{diag}(\frac{\nu}{{\bf P}^{(1)\top}\mathbbm{1}_{B}}). The iterative updates can be succinctly expressed as the Sinkhorn iterations: {\bf P}^{(2t+1)}=\operatorname{diag}({\bf u}^{(t+1)})\mathbf{K}\operatorname{diag}({\bf v}^{(t)}),\quad{\bf P}^{(2t+2)}=\operatorname{diag}({\bf u}^{(t+1)})\mathbf{K}\operatorname{diag}({\bf v}^{(t+1)}), (6) with the scaling vectors {\bf u}^{(t)} and {\bf v}^{(t)} updated according to: {\bf u}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\mu}}{\mathbf{K}{\bf v}^{(t)}},\quad{\bf v}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\nu}}{\mathbf{K}^{\mathrm{T}}{\bf u}^{(t)}}. (7) Here, iterations converge to a stable transport plan {\bf P}^{(\infty)} as the optimal solution of Equation ( 3 ), which provides the minimum cost matching between two distributions. The convergence and dynamics of OT and its dual formulation have been studied extensively in [ 4 , 43 , 19 , 1 ] . Thus, these results guarantee that the iterates will converge to the optimal solution of the EOT objective, or that {\bf P}^{(t)}\rightarrow{\bf P}^{(\infty)} with t\rightarrow\infty . See Appendix A.3 for more details on both the continuous and discrete formulations of OT. 2.4 Wasserstein Dependency Measure The Wasserstein Dependency Measure (WDM) is a measure of deviation between two probability measures. We will use this later and thus provide the formal definition here [ 39 ] . Definition 2 (Wasserstein Dependency Measure) . Define the WDM as the Wasserstein distance ( W_{1} ) between the joint distribution \pi(x,y) and the product of marginal distributions \mu\otimes\nu(x,y) of two random variables x and y . W_{1}(\pi,\mu\otimes\nu)=\sup_{f\in\mathcal{C}(\mathcal{X}\times\mathcal{Y})}\left(\mathbb{E}_{\pi(x,y)}[f(x,y)]-\mathbb{E}_{\mu\otimes\nu(x,y)}[f(x,y)]\right) , where \mathcal{C}(\mathcal{X}\times\mathcal{Y}) denotes the set of all 1-Lipschitz functions from \mathcal{X}\times\mathcal{Y} to \mathbb{R} . 2.5 Optimal Transport and Alignment in Representation Learning Distribution alignment and OT have been widely used for domain adaptation [ 33 , 14 , 30 , 59 ] , and in generative modeling [ 2 , 55 , 49 , 58 ] . The connections between distribution alignment and contrastive learning, however, are still nascent. In [ 51 ] , the authors explore the connection between inverse OT (IOT) [ 32 , 53 , 18 ] and INCE. Our work builds on this connection to OT to build robust divergences (RINCE) and to build a novel unbalanced optimal transport (UOT) method (Section 3.3 ). Additionally, we show how our framework can be used to build flexible methods for encouraging contrast at multiple levels. We use this concept of hierarchical contrast and show that it can be used in domain generalization settings (Section 6.2 ). It is of note that GCA-UOT focuses on relaxing the hard constraints on the row and columns into the soft penalties, which is different with the idea of “unbalanced matching” in [ 51 ] which considers the case where the encoders may not have the same weights. 3 Generalized Contrastive Alignment (GCA) In this section, we will introduce a new framework for generalized contrastive alignment and demonstrate the connections between contrastive learning and optimal transport. 3.1 Problem Formulation Traditional contrastive learning methods focus on bringing positive examples, such as augmentations of the same sample, closer together in representation space. In contrast, our approach reframes contrastive learning as a distributional alignment problem, allowing flexible control over how pairs are matched by imposing specific constraints on the target transport plan, \mathbf{P}_{tgt} . Our objective is to learn an encoder f_{\theta} that minimizes the transport cost between positive samples. By defining \mathbf{P}_{tgt} with specific alignment rules, such as domain-specific or hierarchical constraints, we can influence how samples are organized in the latent space. For instance, setting \mathbf{P}_{tgt} to resemble a diagonal matrix encourages each positive to align primarily with itself or its augmentations, minimizing \text{div}({\bf I}||{\bf P})\approx 0 , where div measures the deviation from an identity matrix (e.g., KL-divergence). This flexibility allows us to encode more nuanced forms of similarity, adapting to tasks where alignment structure varies based on domain, class, or other high-level constraints. By expanding contrastive learning in this way, our method enhances separation of negatives while addressing complex relational patterns, making it suitable for a wider range of learning tasks. Defining the Kernel Space. Before formally stating our objective, we first need to define the concept of an augmentation kernel for our positive and negative examples. Definition 3 (Augmentation Kernel) . Let f_{\theta} denote an encoder with parameters \theta and let ({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{j})\sim\mathcal{A} be two views drawn from the family of augmentations \mathcal{A} . The augmentation kernel for the encoder \theta is defined as {\bf K}_{\theta}({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{j})=\exp(-\text{dist}(\widetilde{f}_{\theta}({\bf x}^{\prime}_{i}),\widetilde{f}_{\theta}({\bf x}^{\prime\prime}_{j}))/\varepsilon) , where \text{dist}(\cdot) can be an arbitrary distance metric, and \widetilde{f}_{\theta}({\bf x}^{\prime}_{i}) is the normalized output of f_{\theta} , and \varepsilon is the regularization parameter. Main Objective. With this definition in hand, we can now formalize our objective as follows: \min_{\theta}~~d_{M}\big({{\bf P}}_{\text{tgt}}||{\bf P}_{\theta}),~\text{with}\quad{\bf P}_{\theta}=\arg\min_{{\bf P}\in\mathcal{B}}\{h({\bf P})+d_{\Gamma}({\bf P}||{\bf K}_{\theta})\}, (8) where {\bf K}_{\theta} is the augmentation kernel defined in Definition ( 3 ), h(x) is a convex function (typically an indicator function), \mathcal{B} is a closed convex constraint set (i.e. Birkhoff polytope) that defines the constraints of proximal operators, d_{\Gamma} is a Bregman divergence that is used to find the nearest points {\bf P}_{\theta} on the constraint set \mathcal{B} of {\bf K}_{\theta} , d_{M} is a convex function (e.g., KL-divergence) that measures divergence between {\bf P}_{\theta} and the target coupling plan {\bf P}_{\text{tgt}} . Our objective is a bi-level optimization problem which aims to learn a representation that minimizes the divergence between the transport plan {\bf P}_{\theta} with the target alignment plan {{\bf P}}_{\text{tgt}} that encodes the matching constraints. When we consider a standard contrastive learning setup where we have pairs of positive examples the source and target distribution, then the target {{\bf P}}_{\text{tgt}} is the identity matrix \mathbf{I} . However, we will show later that other alignment constraints can be considered. Moreover, when \mathcal{B} is the intersection of more constraint sets like C_{1}^{\mu}\cap C_{2}^{\nu} in Equation ( 4 ), a nature way to get the approximation of the nearest points {\bf P}_{\theta} of {\bf K}_{\theta} is to run iterative projections algorithm [ 3 ] , which could be extended into the intersection of several constraint sets like \{\cap_{i=1}^{n}C_{i}\} , resulting in a multi-marginal problem [ 41 ] . 3.2 A Proximal Point Algorithm for GCA In practice, we can solve the alignment problem above by iteratively updating the two main components in our bi-level objective. First, for a fixed encoder parameters \theta , we obtain the transport coupling {\bf P}_{\theta} through our corresponding proximal operator. Second, we measure the deviation between the transport plan {\bf P}_{\theta} with the target {{\bf P}}_{\text{tgt}} that encodes our matching constraints, which denotes the ideal alignment plan on the intersection of the constraint sets. We provide pseudocode for this iterative approach in Algorithm 1 , which we refer to as generalized contrastive alignment or GCA. The implementation of our methods is in https://github.com/nerdslab/gca . Algorithm 1 Proximal-Point Algorithm for Generalized Contrastive Alignment (GCA) 1: Initialization: Initial encoder parameters \theta , target transport plan {{\bf P}}_{\text{tgt}} , kernel function {\bf K}_{\theta} , the function h(x) , divergences d_{\Gamma} and d_{M} (KL or W_{1} ). Initialize transport plan {{\bf P}}_{\theta} based on \theta . 2: Compute the transport coupling {\bf P}_{\theta} : Update {{\bf P}}_{\theta} using the proximal operator scaling for fixed \theta as described in Eq. ( 8 ): {\bf P}_{\theta}=\arg\min_{{\bf P}\in\mathcal{B}}\{h({\bf P})+d_{\Gamma}({\bf P}||{\bf K}_{\theta})\}. 3: Calculate the loss: Calculate deviation between the target and current transport plans \mathcal{L}_{GCA}=d_{M}({{\bf P}}_{\theta},{{\bf P}}_{\text{tgt}}). Update networks f_{\theta} (encoder) and g_{\theta} (projector) to minimize \mathcal{L}_{GCA} . 4: Repeat until convergence: Repeat steps 2 and 3 until convergence. Computing the transport coupling {\bf P}_{\theta} 1 1 1 With a single constraint set like C_{1}^{\mu} in Equation ( 4 ), computing the proximal point only involves a single projection. However, if there are intersecting constraint sets like C_{1}^{\mu}\cap C_{2}^{\nu} , solving for the proximal point requires multiple projections before we approach the nearest point on their intersection. (forward-pass) in GCA algorithms could be treated as a specific type of Dykstra’s projection algorithms [ 5 ] , which computes the iterative projection on the intersection of affine convex sets [ 3 , 42 ] . The proofs of convergence are provided in Appendix B.1 . 3.3 GCA-UOT Method We can also benefit from the rich literature on optimal transport to build different relaxations of our objective [ 43 , 7 , 54 , 33 , 35 ] . In particular, we choose to leverage a formulation of unbalanced optimal transport (UOT) to further relax the marginal constraints [ 11 ] in our objective. In this case, we can add the dual form of d_{\Gamma} to the Equation ( 8 ) and reformulate our objective as: \min_{\theta}~d_{M}(\bf P_{\text{tgt}}\|\bf P_{\theta})+\lambda_{1}h_{\mathcal{F}}(\bf P_{\theta}\mathbbm{1}||\mu)+\lambda_{2}h_{\mathcal{G}}(\bf P_{\theta}^{\top}\mathbbm{1}||\nu)+\varepsilon H(\bf P_{\theta}). (9) Here h_{\mathcal{F}} and h_{\mathcal{G}} can be different divergence measures (e.g., KL divergence) that penalize deviations from the desired marginals \mu and \nu , and \lambda_{1} and \lambda_{2} are regularization parameters that control the trade-off between the transport cost and the divergence penalties. This relaxation leads to different types of proximal operators which we outline in Appendix B.2 . The impact of the entropy regularization parameter \varepsilon on the coupling matrix is studied in Figure A5 , along with the number of iterations and corresponding sensitivity is provided in Figure A6 . 3.4 Modifying the Target Transport Plan to Encode Matching Constraints Contrastive learning objectives can be cast as a minimization of the deviations between the transport plan {\bf P}_{\theta} and the identity matrix, i.e., {\bf P}_{tgt}={\bf I} . However, our GCA formulation enables learning representations that extend beyond this one-to-one matching constraint. This flexibility allows us to incorporate additional matching constraints informed by domain-specific knowledge. For example, in domain generalization scenarios [ 23 , 28 ] , where each batch contains samples from multiple domains, the target alignment plan can be structured as: {\bf P}_{\text{tgt}}[i,j]={\bf I}[i,j]+\alpha\cdot\mathbb{I}(D_{i}=D_{j},\,i\neq j)+\beta\cdot\mathbb{I}(D_{i}\neq D_{j},\,i\neq j), Where \mathbb{I}(\cdot) is the indicator function, which equals 1 if the condition inside is true and 0 otherwise. D_{i} represents the domain of sample i , where \alpha\geq 0 and \beta\geq 0 . In this case, we can improve the representation by building the block constraints which encode either class information (in supervised setting) or domain information (in across domain generalization, visualized in Figure 1 ). 3.5 Computational Complexity The forward-pass only involves the scaling operations in Equation ( 7 ) and doesn’t affect the complexity of the backward-pass. Therefore, GCA methods can be thought of as a form of batch normalization operations with adaptive scaling. An analysis of the complexity is provided along with experiments in Appendix B.1 . Our results show that GCA iterations only slightly increase the computational complexity when compared with their single step equivalent (GCA-INCE vs. INCE). However, we found that GCA-UOT is faster than INCE due to the improved symmetry and smoothness of the loss. Moreover, we record the floating point operations per second (Flops) of running GCA methods. We find that GCA-INCE (6.65 MFlops) has 5\% more Flops than INCE (6.31 MFlops), while GCA-UOT saves 30\% Flops (4.54 MFlops). These results show that our GCA-UOT method is not only superior in terms of accuracy but also in speed. 4 Building Connections to Different CL Objectives Table 1: Comparison of different contrastive alignment objectives. Here we have C_{1}^{\mu} and C_{2}^{\nu} as constraint sets (denoted as \mathcal{B} ) defined in Equation ( 4 ) with their corresponding indicator function. "Iter" refers to iterative methods. Methods d_{M} d_{\Gamma} \mathcal{B} Iter INCE KL KL C_{1}^{\mu} GCA-INCE KL KL C_{1}^{\mu}\cap C_{2}^{\nu} \checkmark RINCE (q=1) W1 KL C_{1}^{\mu} GCA-RINCE (q=1) W1 KL C_{1}^{\mu}\cap C_{2}^{\nu} \checkmark BYOL KL L2 R^{B\times B} In this section, we show how the modification of the different parts of our main objective ( d_{\Gamma},d_{M},\mathcal{B},{\bf K}_{\theta} ) in Equation ( 8 ) can be connected to different contrastive losses. See Table 1 for a summary of how different losses can be mapped back to our formulation. 4.1 Connection to INCE An interesting connection that we can make between GCA main objective and contrastive learning is that we can interpret INCE as a single step in a iterative GCA objective [ 51 ] . This connection can be further summarized through the following theorem. Theorem 1 (INCE Equivalence) . Let {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ) with cosine similarity, d_{\Gamma} and d_{M} equal to KL-divergence, and constraint set as C_{1}^{\mu} in Equation ( 4 ). The INCE objective in Equation ( 1 ) can be re-expressed as a GCA problem in Equation ( 8 ) as follows: \min_{\theta}\text{KL}\big({\bf I}||\text{Prox}_{C_{1}^{\mu}}^{KL}({\bf K}_{\theta})). (10) The proof is contained in Appendix B.3 . Theorem ( 10 ) shows that the INCE loss can be viewed as solving the matching problems in Equation ( 3 ) with row normalization constraints C_{1}^{\mu} . This connection between GCA and INCE allows us to derive the iterative algorithm for GCA-INCE by running Bregman projection iteratively on both row and column normalization sets. 4.2 Connection to RINCE We introduce the following result to build the connection between our framework and RINCE [ 12 ] . Theorem 2 (RINCE Equivalence) . Let {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ). Set target plan {\bf P}_{\text{t}gt}=\bf I , d_{\Gamma} equal to the KL-divergence, d_{M}(\mathbf{I}\|{\bf P})=-\frac{1}{q}(\frac{\operatorname{diag}({\bf P}_{\theta})}{{\bf u}})^{q}+\left(\frac{\lambda\mathbf{I}}{{\bf u}}\right)^{q} with \lambda , q , and {\bf u}=\operatorname{diag}\left(\frac{\mu}{{\bf P}^{(0)}\mathbbm{1}}\right) , and constraint set C_{1}^{\mu} defined in Equation ( 4 ). The RINCE objective in Equation ( 2 ) can be re-expressed as a GCA problem as follows: \min_{\theta}d_{M}(\mathbf{I}\|{\bf P}_{\theta}),~~\text{with}~~{\bf P}_{\theta}=\text{Prox}_{C_{1}^{\mu}}^{\text{KL}}({\bf K}_{\theta}), (11) The proof is provided in Appendix B.4.1 . As we can see, RINCE introduces adjustable parameters q and \lambda , with \lambda controlling the weight of negative samples, while q\in(0,1] serves to switch between KL divergence and Wasserstein discrepancy. When q=1 , we have the following theorem: Theorem 3 (W1 Equivalence) . Let {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ) with cosine similarity. Set target plan {\bf P}_{\text{t}gt}=\bf I , d_{\Gamma} equal to the KL-divergence, d_{M} equal to the 1-Wasserstein distance (W_{1}) in Definition ( 2 ), and the constraint set as C_{1}^{\mu} defined in Equation ( 4 ). The RINCE object in Equation ( 2 ) with q=1 can be re-expressed as a GCA problem as follows: \min_{\theta}W_{1}\big({{\bf P}_{tgt}}||\text{Prox}_{C_{1}^{\mu}}^{KL}({\bf K}_{\theta})). (12) See Appendix B.5 for the proof. This connection to RINCE suggests an extended iterative formulation to calculate the coupling plan as the projection point {\bf P}^{(\infty)}=\text{Prox}_{C_{1}^{\mu}\cap C_{2}^{\nu}}^{\text{KL}}({\bf K}_{\theta}) of {\bf K}_{\theta} on the constraint set C_{1}^{\mu}\cap C_{2}^{\nu} . In this case, we can write an iterative algorithm for robust alignment called GCA-RINCE as follows: L_{\text{GCA-RINCE}}^{\lambda,q}=\min_{\theta}-q^{-1}(\operatorname{diag}({\bf P}_{\theta}^{(2t-1)})/{{\bf u}^{(t)}})^{q}+q^{-1}(\lambda{\bf P}_{tgt}/{\bf u}^{(t)})^{q}, (13) where \lambda and q are hyperparameters, {\bf P}^{(1)}\coloneqq\operatorname{diag}({\bf u}^{(1)}){\bf K}_{\theta}\operatorname{diag}({\bf v}^{(0)}) , and t is the number of iterations. 4.3 Connection to BYOL Our framework also allows us to make connections to BYOL [ 22 ] . BYOL learns by encouraging similarity between positive image pairs, without explicitly conditioning on negative examples. To build this connection, recall that BYOL has the online network parameterized by \theta and target network parameterized by \xi , where {\bf z}^{\prime}_{\theta}=\widetilde{f}_{\theta}({\bf x}^{\prime}) and {\bf z}^{\prime\prime}_{\xi}=\widetilde{f}_{\xi}({\bf x}^{\prime\prime}) are the normalized outputs of the online and target networks, respectively. A simplified version of the BYOL loss can be written as: L_{\text{BYOL}}=\|\widetilde{q}_{\theta}({\bf z}^{\prime}_{\theta})-{\bf z}^{\prime\prime}_{\xi}\|_{2}^{2}, where \widetilde{q}_{\theta}({\bf z}^{\prime}_{\theta}) is the normalized output after online network and q_{\theta} is the predictor. 2 2 2 In practice, BYOL also switches the order of views to symmetrize the loss. For ease of discussion, we consider just one pair of views but the same could be argued for the full symmetric version. In this case, we can provide the following connection between GCAand BYOL as follows. Theorem 4 (BYOL Equivalence) . Let \mathbf{S}_{\theta}({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{j})=\exp(-\|\widetilde{q}_{\theta}({\bf z}_{i}^{\prime})-{\bf z}_{j}^{\prime\prime}\|) denote the augmentation kernel. Set the target plan {\bf P}_{\text{t}gt}=\bf I , d_{\Gamma} equal to the L2-distance, d_{M} equal to the KL-divergence, and constraint set as R^{B\times B} . The BYOL objective can be re-expressed as a GCA problem as follows: \min_{\theta}\text{KL}\big({\bf I}||\mathbf{S}_{\theta}),~~\text{with}~~\mathbf{S}_{\theta}=\text{Prox}_{R^{B\times B}}^{\|\cdot\|}(\mathbf{S}_{\theta}). (14) See the proof in Appendix B.6 . 5 Theoretical Analysis In this section, we aim to show how the GCA-methods can improve alignment and uniformity in the latent space [ 57 ] . Here, alignment means that the features of the positive samples are as close as possible, while uniformity means that the features of negative samples are uniformly distributed on latent space (see Appendix C.1 for formal definitions). These quantities have been studied in a number of related works [ 57 , 45 ] , where one can show that improved alignment and uniformity can lead to different benefits in representation learning. 5.1 Improved alignment with GCA Contrastive learning minimizes the deviation between the target alignment plan with the transport plan in Definition 3 through empirical risk minimization (ERM). Therefore, a tighter bound on the empirical risk corresponds to a smaller difference between the ideal alignment with the coupling matrix. We show that this in turn leads to better alignment of the positive views. Analysis of INCE vs GCA-INCE. GCA-INCE ensures that the final transport plan {\bf P}^{(\infty)} is closer to the ideal identity matrix compared to the INCE, as we show in the following theorem. Theorem 5 (Improved Alignment with INCE) . Let {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ). Set d_{M} and d_{\Gamma} to the KL-divergence, and {{\bf P}}_{\text{tgt}}=\bf I . The GCA-INCE loss with converged plan {\bf P}_{\theta}^{(\infty)} is lower than the GCA-INCE loss with {\bf P}_{\theta}^{(t)} in Equation ( 6 ) for all t . The full proof is provided in Appendix C.1.1 . The above theorem tells us that solving Equation ( 8 ) with iterative projection will converge to a transport plans {\bf P}_{\theta}^{(\infty)} with lower KL divergence than the one-step solution provided by INCE. We can establish the convergence of the {\bf P}^{(t)}\rightarrow{\bf P}^{(\infty)} , based on the convergence of Bregman projection. Analysis of RINCE vs GCA-RINCE. GCA also benefits from other Bregman divergences, like the WDM in RINCE, which provides robustness against distribution shift compared to the KL-divergence in INCE. GCA-RINCE provides a lower bound on the RINCE loss in Equation ( 2 ), which allows us to develop a tighter bound with {\bf P}^{(\infty)} obtained by several proximal steps with GCA. Theorem 6 (Improved Alignment with RINCE) . GCA-RINCE loss with {\bf P}_{\theta}^{(t)} in Equation ( 13 ) is lower than the loss in the Theorem ( 2 ) as L_{\text{GCA-RINCE}}^{\lambda,q=1}({\bf P}_{\theta}^{(t)})\leq L_{\text{RINCE}}^{\lambda,q=1}({\bf P}_{\theta}^{(1)}) . See Appendix C.1.1 for the full proof and an analysis of GCA methods for different choices of d_{M} . 5.2 Improved Uniformity of Representations Through GCA The improved alignment of GCA-methods comes from maximization of the uniformity under the constraint of intersection C_{1}^{\mu}\cap C_{2}^{\nu} in Equation ( 4 ), rather than the constraint set C_{1}^{\mu} in INCE (see Table 1 ). Finding the projection of {\bf K}_{\theta} on set of C_{1}^{\mu}\cap C_{2}^{\nu} through proximal steps is equivalent to solving the dual problem of EOT, which can be summarized through the following theorem. Theorem 7 (Improved Uniformity) . Given the constraint sets in Equation ( 4 ), the optimal transport coupling upon convergence of Equation ( 6 ), denoted as {\bf P}^{(\infty)} , achieves a higher uniformity loss compared to the single-step transport plan {\bf P}^{(1)} obtained by INCE. The proof is provided in the Appendix C.2 . Through loss propagation, we show that the alignment plan offered by {\bf P}^{(\infty)} will guide the subsequent iterations towards more uniform representations. 5.3 Impacts of GCA on a downstream classification task We take this one step further and examine the impact of GCA on a downstream classification task. For a classification task, using a labeled dataset \mathcal{D}=\{(\bar{{\bf x}}_{i},\mathbf{y}_{i})\}\in\bar{\mathcal{X}}\times\mathcal{Y} where \mathcal{Y}=[1,\dots,M] with M classes, we consider a fixed, pre-trained encoder f_{\theta}\in\mathcal{F}:\mathcal{X}\to\mathcal{S} . Assume that positive and negative views of n original samples (\bar{{\bf x}}_{i})_{i\in[1..n]}\subset\bar{\mathcal{X}} are sampled from the data distribution p(\bar{{\bf x}}) . In this case, the uniformity loss is equivalent to optimizing the downstream supervised classification tasks with cross-entropy (CE) loss when the following two assumptions are satisfied [ 16 ] . Assumption 1 (Expressivity of the Encoder) . Let us define \mathcal{H}_{\bar{\mathcal{X}}} is the RKHS associated with the kernel {\bf K}_{\bar{\mathcal{X}}} defined on \bar{\mathcal{X}} , and (\mathcal{H}_{f_{\theta}},{\bf K}_{\theta}) defined on \mathcal{X} with augmentation kernel {\bf K}_{\theta}=\langle f_{\theta}(\cdot),f_{\theta}(\cdot)\rangle_{\mathbb{R}^{d}} in Definition 3 . And we assume that \forall g\in\mathcal{H}_{f_{\theta}},\ \mathbb{E}_{\mathcal{A}(x|\cdot)}g(x)\in\mathcal{H}_{\bar{\mathcal{X}}} . Assumption 2 (Small Intra-Class Variance) . For y\neq y^{\prime} , the intra-class variance \delta_{i},\delta_{j} are negligible compared to the distance among different class centroids, \mu_{y},\mu_{y^{\prime}} as \|\mu_{y}-\mu_{y^{\prime}}\|\gg\|\delta_{i}-\delta_{j}\| . Claim 1 . If Assumption 1 and Assumption 2 hold, then maximizing the uniformity is equivalent to minimizing the downstream CE loss. The proof is provided in Appendix C.2 . Optimizing the self-supervised loss under ideal conditions improves downstream CE tasks and helps to explain why maximizing uniformity aids classification. Remark. . Maximizing uniformity can enhance downstream classification but risks “feature suppression” by encouraging shortcut features that harm generalization [ 48 ] . In GCA-UOT, adding penalties modifies the transport plan from that of a pure uniformity loss, helping to avoid feature suppression. We find empirical evidence that UOT provides a more robust transport plan which appears to circumvent some of these shortcut features from being learned (Figure A4 in Appendix C.3 ). 6 Experiments In this section, we conduct empirical evaluations to study the performance of our approach in both handling noisy and corrupted views and in domain generalization tasks. 6.1 Comparison with CL Baselines Experiment setup. To examine the robustness of our framework, we trained INCE and RINCE as baselines, and developed their GCA-based alternatives (+GCA). In addition, we also compared with our novel GCA-UOT method, two variants of IOT established in [ 51 ] , and other CL baselines, including BYOL and SimCLR. For experiments with SVHN [ 36 ] and ImageNet100 [ 15 ] we use the ResNet-50 encoder as the backbone and use a ResNet-18 encoder as the backbone for CIFAR-10, CIFAR-100 [ 29 ] and a corrupted version of CIFAR called CIFAR-10C [ 25 ] . Standard Setting Noisy Setting Method CIFAR-10 CIFAR-100 SVHN ImageNet100 CIFAR-10 (Ex) CIFAR-100 (Ex) CIFAR-10C CIFAR-10C (Ex) INCE 92.01 ± 0.40 71.09 ± 0.31 92.42 ± 0.24 73.01 ± 0.61 82.03 ± 0.32 62.54 ± 0.20 81.52 ± 1.04 83.28 ± 0.25 GCA-INCE 93.02 ± 0.19 71.55 ± 0.12 92.64 ± 0.26 73.04 ± 0.76 82.18 ± 0.69 62.65 ± 0.17 82.63 ± 0.28 82.72 ± 0.27 \Delta +1.01 +0.46 +0.22 +0.03 +0.15 +0.11 +1.11 -0.56 RINCE 93.27 ± 0.20 71.63 ± 0.36 93.26 ± 0.15 71.91 ± 0.43 82.60 ± 0.63 63.55 ± 0.14 82.86 ± 0.21 83.64 ± 0.26 GCA-RINCE 93.47 ± 0.30 71.95 ± 0.48 93.57 ± 0.26 73.44 ± 0.55 82.76 ± 0.49 63.14 ± 0.41 82.87 ± 0.11 83.69 ± 0.16 \Delta +0.20 +0.32 +0.31 +1.53 +0.16 -0.47 +0.01 +0.05 SimCLR 92.07 ± 0.90 70.85 ± 0.50 92.13 ± 0.34 72.20 ± 0.78 81.87 ± 0.53 62.94 ± 0.13 81.74 ± 1.54 83.25 ± 0.18 BYOL 90.56 ± 0.59 69.75 ± 0.37 89.50 ± 0.46 69.75 ± 0.83 81.55 ± 0.50 62.11 ± 0.25 82.43 ± 0.06 70.09 ± 0.34 IOT [ 51 ] 92.09 ± 0.22 68.37 ± 0.42 92.25 ± 0.19 72.27 ± 0.53 80.59 ± 0.64 62.69 ± 0.34 82.01 ± 0.80 81.57 ± 0.83 IOT-uni [ 51 ] 91.49 ± 0.11 68.62 ± 0.35 92.33 ± 0.11 72.88 ± 0.71 80.79 ± 0.24 62.56 ± 0.22 81.19 ± 1.12 81.82 ± 0.51 GCA-UOT 93.50 ± 0.31 72.16 ± 0.38 93.82 ± 0.17 74.09 ± 0.40 83.18 ± 0.44 63.62 ± 0.27 82.90 ± 0.50 83.64 ± 0.19 Table 2: Test accuracy (%) on a downstream classification task after pretraining. Results are shown for CIFAR-10 (ResNet18), CIFAR-100 (ResNet18), SVHN (ResNet18), and ImageNet100 (ResNet50) under standard and extreme (Ex) augmentation conditions (averaged over 5 seeds). The CIFAR-100 column is now updated with the latest values from Table A3 . The top model is in bold and the second-place model is underlined. For INCE and RINCE, we also provide the improvement \Delta by adding GCA to each method. In all of these cases, we follow the standard self-supervised learning evaluation protocol [ 8 ] , where we train the encoder on the training set in an unsupervised manner and then train a linear layer on top of the frozen representations to obtain the final accuracy on the test set. In addition to standard data augmentation policies commonly used [ 12 ] , we also apply three different extreme augmentation policies to examine the robustness of GCA towards noisy views (details in Appendix D.2 ). Learning rates and other training details for CIFAR-10, CIFAR-100, SVHN, and ImageNet100 are provided in Appendix D.1 , while specific training details for CIFAR-10C are included in Appendix D.2 . Results on Standard Augmentations. First, we performed experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet100 using standard sets of augmentations that are applied to achieve state-of-the-art performance (Table 2 , Standard Setting). We found the +GCA versions of INCE and RINCE exhibit performance gains in almost all settings except for SVHN, with bigger gains observed when adding GCA to RINCE. Additionally, we find that our unbalanced OT method, GCA-UOT, achieves the top performance across the board, on all four datasets tested. The transport plans obtained by each methods are provided in Figure A4 along with a study of the sensitivity of the methods to hyperparameters (Appendix A7 ). Results on Corrupted Data and Extreme Augmentations. Next, we tested the methods in two noisy settings. In the first set of experiments, we apply extreme augmentations to CIFAR-10 (Ex) and CIFAR-100 (Ex) (see Appendix D.2 ) to introduce noisy views during training. In the second set of experiments, we used the CIFAR-10C to further test the ability of our method to work in noisy settings. Our experimental results demonstrate that the GCA-based strategy effectively enhances the model’s generalization ability and adaptability to aggressive data augmentations. In addition to improving classification accuracy, the GCA-based methods also improve the representational alignment and uniformity, as shown in Appendix E.2 . This observation is in line with our theoretical analysis in Section 5.2 , where we show that the obtained representations provide better overall alignment of positive views and better spread in terms of uniformity [ 57 ] . 6.2 Block Diagonal Transport in Domain Generalization Figure 1: Incorporating different priors into learning across multiple domains. (A) Example target alignment plan {\bf P}_{\text{t}gt} , where the target over all samples from the same domain are set to \alpha , the diagonal values are set to 1, and across-domain samples are set to \beta . (B) The domain classification accuracy (red) and overall class accuracy (blue) with ( \alpha-\beta ) increases. In a final experiment, we aimed to demonstrate the flexibility and robustness of our framework by applying it to a domain generalization task, where samples originate from different domains (e.g., Photo, Cartoon, Sketch, Art). We explored the effects of introducing domain-specific alignment constraints in our transport plan, hypothesizing that this could enhance the latent space organization to capture more nuanced domain similarities. Our approach enables additional contextual information to be seamlessly integrated into the transport process. In this case, domain information was incorporated to distinguish the alignment of samples from the same versus different domains. To achieve this, we adjusted the target transport plan {\bf P}_{tgt} , selectively modifying parameters (\alpha,\beta) to vary the influence of domain-based alignment constraints as shown in Figure 1 (A). Specifically, we set \{\alpha=0,\beta>0\} to prioritize cross-domain alignment and \{\alpha>0,\beta=0\} to focus on intra-domain alignment. The training was conducted on the PACS dataset [ 31 ] using a ResNet-18 encoder with the GCA-INCE objective. After training the encoder in an unsupervised manner, we freeze the encoder and then train a linear readout layer to predict either the sample’s class or the domain it belonged to. This setup allowed us to isolate the effect of our transport adjustments on the latent space’s capacity to encode both class and domain information. The results, displayed in Figure 1 (B), revealed that increasing the domain alignment weight enhances the accuracy of domain classification (from 72.11% to 95.16%) without diminishing classification performance. This outcome suggests that GCA can effectively encode both domain and class information in a single latent representation. The ability to adjust alignment constraints provides a powerful tool for domain generalization tasks, enabling multiple types of similarity to be jointly encoded. This flexibility can potentially alleviate issues related to information loss from data augmentation, especially in fine-grained classification settings, by retaining essential domain-specific characteristics across transformations. 7 Conclusion In this work, we introduced generalized contrastive alignment (GCA), a flexible framework that redefines contrastive learning as a distributional alignment problem using optimal transport to control alignment. By allowing targeted control over alignment objectives, GCA demonstrates strong performance across both standard and challenging settings, such as noisy views and domain generalization tasks. This work opens up broader possibilities for learning robust representations in real-world scenarios, where data is often diverse, noisy, or comes from multiple domains. Future work includes applications of GCA to graphs and time series data, as well as multi-modal settings where our approach can integrate various forms of similarity. As alignment strategies become integral to contrastive learning, GCA offers a promising foundation for more adaptive and expressive self-supervised models. Appendix Appendix A Background and Notation A.1 Notation Datasets and contrastive pairs: Let \mathbf{x} denotes a vector and \mathbf{X} denotes a matrix, with right subscript {\bf X}_{b} denote the batch of the input samples, {\bf X}_{b}:=[{\bf x}_{1},{\bf x}_{2},\dots,{\bf x}_{B}] , here B is equal to batch size. For each sample {\bf x}_{i} in the input batch matrix \mathbf{X}_{b} , {\bf x}^{\prime}_{i} means augmented view 1 of {\bf x}^{\prime}_{i} , {\bf x}^{\prime\prime}_{i} means augmented view 2 of {\bf x}_{i} , the positive pairs in input data denoted as ( {\bf x}_{i} , {\bf x}^{\prime\prime}_{i} ), negative pairs in input data denoted as ( {\bf x}^{\prime}_{i} , {\bf x}^{\prime\prime}_{j} ), i\neq j . Give a weights ( \theta ) parametrized representation function (artificial neural network) f_{\theta} with adjustable adjustable temperature \varepsilon , which project the the positive pairs in latent space denoted as s^{+}=\langle\varepsilon^{-1}\widetilde{f}_{\theta}({\bf x}^{\prime}_{i}),\widetilde{f}_{\theta}({\bf x}^{\prime\prime}_{i})\rangle , and negative pairs in latent space denoted as s^{-}=\langle\varepsilon^{-1}\widetilde{f}_{\theta}({\bf x}^{\prime}_{i}),\widetilde{f}_{\theta}({\bf x}^{\prime\prime}_{j})\rangle,i\neq j . Here, \langle\cdot,\cdot\rangle is the inner product, which means \langle\widetilde{f}_{\theta}({\bf x}^{\prime}_{i}),\widetilde{f}_{\theta}({\bf x}^{\prime\prime}_{i})\rangle=f_{\theta}({{\bf x}_{i}^{\prime}})^{\top}f_{\theta}({\bf x}_{j}^{\prime\prime})/\|f_{\theta}({{\bf x}_{i}^{\prime}})\|\|f_{\theta}({{\bf x}_{j}^{\prime\prime}})\| is the normalized form. Continuous settings for optimal transport \mathcal{X} and \mathcal{Y} are topological spaces, \mathcal{X}\times\mathcal{Y} is the product space, or Torus. C(\mathcal{X}) is the compact topological space which contains all of continuous functions on \mathcal{X} endowed with the sup-norm. On Torus \mathcal{X} and \mathcal{Y} we define M as a compact n-dimensional manifold in product space \mathcal{X}\times\mathcal{Y} ( \mathcal{X}=\mathcal{Y}:=\mathbb{R}^{n}/\mathbb{Z}^{n} ) endowed with a cost function c(x,y):=d_{M}(x,y)^{2}/2 (Euclidean dsitance function) on \mathbb{R}^{n} . Transport plan \pi(x,y):\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R} is an element in \mathcal{P}(\mathcal{X}\times\mathcal{Y}) . \mathcal{P}(\mathcal{X}\times\mathcal{Y}) means the collections of the joint distributions of the two marginal distributions \mu\in\mathcal{P}(\mathcal{X}) and \nu\in\mathcal{P}(\mathcal{Y}) . \mathcal{P}(\mathcal{X}) the space of all (Borel) probability measures on \mathcal{X} , \mathcal{P}(\mathcal{Y}) means the same to \mathcal{Y} . To find a joint distribution (or plan) \pi(x,y) in collections U(\mu,\nu) with marginals \mu and \nu in the product space \mathcal{X}\times\mathcal{Y} , we can formulate as: \min_{\pi\in U(\mu,\nu)}\sum_{x\in\mathcal{X},y\in\mathcal{Y}}\pi(x,y)c(x,y)\quad\text{s.t.}~\sum_{y\in\mathcal{Y}}\pi(x,y)=\mu(x),~\sum_{x\in\mathcal{X}}\pi(x,y)=\nu(y) Discrete settings of optimal transport : \mu and \nu can be discrete probability measures whose supports are finite sets X:=\{f_{\theta}({\bf x}^{\prime}_{i})\}_{i=1}^{N},Y:=\{f_{\theta}({\bf x}^{\prime\prime}_{i})\}_{i=1}^{B} . And we define \mu=\sum_{i=1}^{B}\delta_{f_{\theta}({\bf x}^{\prime}_{i})}p_{i},\nu=\sum_{i=1}^{B}\delta_{f_{\theta}({\bf x}^{\prime\prime}_{i})}q_{i}, with vectors p and q in a simplex \Delta_{B} in \mathbb{R}^{B} defined by \Delta_{B}:=\left\{v\in\mathbb{R}^{B}:v_{i}\geq 0,\sum_{i=1}^{B}v_{i}=1\right\} , which we identify with \mathcal{P}(\{1,\ldots,B\}) . {\bf C} is a B\times B cost matrix calculated by c(x,y) , whose sampled from finite sets X and Y defined previously, and N is the batch size. {\bf u} and {\bf v} are B\times 1 scale factors matrix, {\bf u}^{(t)} and {\bf v}^{(t)} mean the scale factor matrix after t iterations of sinkhorn algorithms. {\bf P} is a B\times B joint distribution matrix of \mu and \nu , which represents the transport plan \pi(x,y) that corresponds to minimize the cost. {\bf P}^{(2t-1)} means we use t iterations {\bf u}^{(t)} and {\bf v}^{(t-1)} to calculate {\bf P}^{(t)} . When t=1 means we use {\bf u}^{(1)} and {\bf v}^{(0)} to calculate {\bf P}^{(1)} , which is called half-step OT or one step Bregman projection. When t=2 means we use {\bf u}^{(1)} and {\bf v}^{(1)} to calculate {\bf P}^{(2)} , which is called half-step OT or one step Bregman projection. A.2 Proximal operator setup In this section, we are going to provide the detailed illustration about the proximal operators. How the proximal operator would convert to the projection. And how to solve the Bregman projection with KL divergence. A.2.1 Explanation of Definition 1 Let h:\mathcal{X}\rightarrow[-\infty,+\infty] be a proper, lower semi-continuous convex function on Hilbert space \mathcal{X} . The proximal operator of h at point {\bf v}\in\mathcal{X} is defined as a unique minimizer of the function {\bf x}\mapsto h({\bf x})+\frac{1}{2}\|{\bf x}-{\bf v}\|^{2}_{2} [ 40 ] . For an instance, h is convex function relative to the constraint set \mathcal{B} , like the indicator function h_{\mathcal{B}} . Given the Euclidean norm on \mathcal{X} , we can write the proximal operator \text{Prox}_{h,\mathcal{B}}^{\|\cdot\|^{2}}(\mathbf{v}) as the most common way: \text{Prox}_{h,\mathcal{B}}^{\|\cdot\|^{2}}(\mathbf{v})=\arg\min_{\mathbf{x}\in\mathcal{B}}\left\{h(\mathbf{x})+\frac{1}{2}\|\mathbf{x}-\mathbf{v}\|^{2}_{2}\right\} The solution to the proximal operator exists and is unique due to the strong convexity of the above function. A.2.2 Connection to the projection Here, if we treat h(x) as the indicator functions of the constraint set \mathcal{B} as: h(x)=\begin{cases}0,&\text{if }x\in\mathcal{B}\\ \infty,&\text{if }x\notin\mathcal{B}.\end{cases} Then the proximal operators problem could be understood intuitively as finding the "shortest distance" between the point v with the constraint set \mathcal{B} , which means the projection. And the following lemma holds: Lemma 1 . If the constraint set \mathcal{B} is closed and convex, then the projection of point v is unique on \mathcal{B} . Proof of the Lemma 1 : This lemma can be proved by the strict convexity of the proximal operator. Figure A1: Illustration of the proximal operators A. Visualization of proximal operators in \mathbb{R}^{3} . On the surface defined by h(x,y)=x^{2}+y^{2} within the domain constraints -1.20, (20) and we can use these relationship into the constraints sets with {\bf P}^{(0)}=\operatorname{diag}(\mathbbm{1}){\bf K}\operatorname{diag}(\mathbbm{1}) . \langle{\bf P}^{(1)},\mathbbm{1}\rangle=\mu\Rightarrow\langle{\bf u}^{(1)}{\bf K},\mathbbm{1}\rangle=\mu,{\bf u}^{(1)}=\frac{\mu}{\sum_{i}{\bf K}_{ij}},{\bf P}^{(1)}=\text{diag}\left(\frac{\mu}{{\bf P}^{(0)}\mathbbm{1}_{B}}\right){\bf P}^{(0)}, (21) If we repeat this progress for the set C_{2}^{\nu}\coloneqq\{{\bf P}:{\bf P}^{T}\mathbbm{1}_{n}=\nu\} , we will get the \text{Prox}_{C_{2}^{\nu}}^{\text{KL}}({\bf P}^{(t+1)}) . And in the second step, we project onto the second constraint set C_{2}^{\nu} with indicator function h_{\mathcal{G}}(x) defined on C_{2}^{\nu} and get: {\bf P}^{(2)}\coloneqq\text{Prox}^{\text{KL}}_{C_{2}^{\nu}}({\bf P}^{(1)})={\bf P}^{(1)}\text{diag}\left(\frac{\nu}{{\bf P}^{(1)\top}\mathbbm{1}_{B}}\right). (22) Iterating over these two sets of projections {\bf P}^{(t+1)}\coloneqq\text{Prox}^{\text{KL}}_{C_{1}^{\mu}}({\bf P}^{(t)}) and {\bf P}^{(t+2)}\coloneqq\text{Prox}^{\text{KL}}_{C_{2}^{\nu}}({\bf P}^{(t+1)}) until convergence could be summarized as Sinkhorn algorithm with t via recursive form: {\bf u}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\mu}}{\mathbf{K}{\bf v}^{(t)}},\quad{\bf v}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\nu}}{\mathbf{K}^{\mathrm{T}}{\bf u}^{(t+1)}},\quad{\bf P}^{(2t+2)}=\operatorname{diag}({\bf u}^{(t+1)})\mathbf{K}\operatorname{diag}({\bf v}^{(t+1)}). (23) The Sinkhorn algorithm is composed with two steps Bregman projection, Similarly, we can write out this recursive relationship as: {\bf P}^{t+1} can be updated with dual variables f , g and {\bf u}^{(t)}=e^{f^{(t)}/\varepsilon},{\bf v}^{(t)}=e^{g^{(t)}/\varepsilon} . The set U(\mu,\nu)=C_{1}^{\mu}\cap C_{2}^{\nu} , representing the feasible transport plans with given marginals. It could be any random sets, i. e. \mathcal{B}=C_{1}^{\mathbbm{1}}\cap C_{2}^{\mathbbm{1}} denote the Birkhoff polytope of doubly stochastic matrices where \mu=\mathbbm{1} and \nu=\mathbbm{1} are the uniform distributions with all one element. A.3 Background on OT This section defines discrete and continuous optimal transport. Since the section 2.3 lacks a discrete OT definition, we discuss it here and show the equivalence between solving Bregman projection and the entropy-regularized OT (EOT) problem. To support convergence proofs later, we introduce definitions of continuous measures. Symbols \mu and \nu may represent both discrete and continuous measures for intuitive consistency, with precise definitions at the start of each subsection. A.3.1 Background on discrete OT In section 2.3 , we provide a general definition of the discrete optimal transport. Here, we specifically define the optimal transport on the representations space after an encoder f_{\theta} with two augmented views ({\bf x}^{\prime},{\bf x}^{\prime\prime}) . As we mainly discussed the distribution on the representation space, so here we suppose there is an encoder f_{\theta} will project the augmented views into the latent. Here we define \mu=\sum_{i=1}^{N}\delta_{f_{\theta}({\bf x}^{\prime}_{i})}p_{i},\nu=\sum_{i=1}^{N}\delta_{f_{\theta}({\bf x}^{\prime\prime}_{i})}q_{i}, with vectors p and q in a simplex \Delta_{B} in \mathbb{R}^{B} defined by \Delta_{B}:=\left\{v\in\mathbb{R}^{B}:v_{i}\geq 0,\sum_{i=1}^{B}v_{i}=1\right\} . Here, {\bf P} is a B\times B joint coupling matrix of the marginal distributions \mu and \nu , which describes how much mass is needed to convert one distribution to match another. {\bf C} is a B\times B cost matrix calculated by the cost function c(x,y) i.e. cosine dissimilarity, and we can write the OT problem as the constrained linear programming problem: \displaystyle\min_{\bf{P}}\langle\bf{P},{\bf C}\rangle~~\text{s.t. }\bf{P}\mathbbm{1}=\mu,~\bf{P}^{\top}\mathbbm{1}=\nu. (24) Even though directly solving Equation ( 24 ) is high computational complexity O(n^{3}) , we introduce a common relaxation called entropic regularization to smooth the transport plan. A.3.2 Entropy regularized OT and the Sinkhorn algorithm. Solving the exact OT problem above can be very computationally intensive. In this case, we can add the Shannon entropy H({\bf P})=-({\bf P}_{ij}(\log({\bf P}_{ij})) to our objective in Equation ( 24 ) and obtain an approximation of entropy-regularized optimal transport (EOT) plan as: \min_{{\bf P}\in\mathcal{B}}~\langle{\bf P},{\bf C}\rangle-\varepsilon H({\bf P}),\quad\text{where}~H({\bf P})=-\sum{\bf P}_{ij}\log({\bf P}_{ij}), (25) where \varepsilon is a user specified parameter that controls the amount of smoothing in the transport plan. The cost matrix {\bf C} could be transformed into the Gibbs kernel matrix {\bf K} on a Hilbert space with the given formula, {\bf K}_{ij}=\exp\left(-\varepsilon^{-1}{\bf C}_{ij}\right) (26) To solve ( 25 ) under the kernel space induced by {\bf K} , we can use the iterative Sinkhorn algorithm with the initialization of {\bf u}^{(0)} and {\bf v}^{(0)} as all one vector divided by the batch size, and the update rules: \displaystyle{\bf u}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\mu}}{\mathbf{K}{\bf v}^{(t)}}\text{ and }{\bf v}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{{\nu}}{\mathbf{K}^{\mathrm{T}}{\bf u}^{(t+1)}}, (27) Then, the output of plan after t iterations is {\bf P}^{(t)}=\operatorname{diag}({\bf u}^{(t)})\mathbf{K}\operatorname{diag}({\bf v}^{(t)}). (28) It also could be interpreted with dual variables f and g : {\bf P}^{(t)}_{i,j}=e^{f^{(t)}_{i}/\varepsilon}e^{-{\bf C}_{i,j}/\varepsilon}e^{g^{(t)}_{j}/\varepsilon},\quad{\bf u}^{(t)}=e^{f^{(t)}/\varepsilon},{\bf v}^{(t)}=e^{g^{(t)}/\varepsilon} (29) After convergence, the resulting {\bf P} will be the optimal solution to Equation ( 25 ). The convergence and dynamics of OT and the dual formulation have been studied extensively in [ 4 , 43 , 19 , 1 ] . Here, iterations converge to a stable transport plan {\bf P}^{(\infty)} as the optimal solution of Equation ( 3 ), which provides the minimum cost matching between two distributions. The convergence and dynamics of OT and its dual formulation have been studied extensively in [ 4 , 43 , 19 , 1 ] . Thus, these results guarantee that the iterates will converge to the optimal solution of the EOT objective, or that {\bf P}^{(t)}\rightarrow{\bf P}^{(\infty)} with t\rightarrow\infty . This allows us to state the following lemma: Lemma 2 . Solving the entropy optimal transport in Equation ( 3 ) is consistent with iterative solving the Bregman projection. Proof of the Lemma 2 : Giving that some points {\bf K} and {\bf P} , their distance could be measured by KL divergence: \text{KL}({\bf P}\|{\bf K})=\sum_{ij}{\bf P}_{ij}\log\left(\frac{{\bf P}_{ij}}{{\bf K}_{ij}}\right)-{\bf P}_{ij}+{\bf K}_{ij} As {\bf C}_{ij}=-\varepsilon\log{\bf K}_{ij} in Sinkhorn, we can see find {\bf P} to minimize the Equation ( 3 ) can be transformed into some formula about {\bf K}_{ij} : \displaystyle\min_{\bf P}\langle{\bf P},{\bf C}\rangle-\varepsilon H({\bf P}) \displaystyle=\min_{\bf P}\sum_{i,j}{\bf C}_{ij}{\bf P}_{ij}+\varepsilon\sum_{i,j}{\bf P}_{ij}(\log({\bf P}_{ij})) (30) \displaystyle=\min_{\bf P}\varepsilon\sum_{i,j}(-{\bf P}_{ij}\log{\bf K}_{ij}+{\bf P}_{ij}\log({\bf P}_{ij})) (31) \displaystyle=\min_{\bf P}\varepsilon\text{KL}({\bf P}\|{\bf K})~~\text{s.t. }{\bf P}\mathbbm{1}=\mu,~{\bf P}^{\top}\mathbbm{1}=\nu, (32) Consider the {\bf K} is a point in Hilbert kernel space, and \varepsilon is the constant, we set the \mu and \nu form the \mathcal{B} , so here can have: ~{\bf P}=\text{Prox}^{\text{KL}}_{\mathcal{B}}({\bf K})=\arg\min_{{\bf P}\in\mathcal{B}}\text{KL}({\bf P}||{\bf K})=\arg\min_{\bf P}\{\langle{\bf P},{\bf C}\rangle+\varepsilon H({\bf P}):{\bf P}\mathbbm{1}=\mu,~{\bf P}^{\top}\mathbbm{1}=\nu\} (33) A.3.3 Background on continuous optimal transport To show the convergence of the Bregman projection, here we define the optimal transport problem with the continuous measure. Inherit the definition of \mathcal{X} and \mathcal{Y} in the Appendix A.1 , finding the optimal transport between two continuous measure \mu and \nu could be transformed into some problems with the minimization of Kantorovich functional. Definition 4 ( Continuous optimal transport ) . We redefine \mu and \nu be two probability measures on latent manifold \mathcal{M} with Hölder continuous and strictly positive densities e^{f} and e^{g} , respectively: \mu=e^{f}dM,\nu=e^{g}dM , where dM is the Riemannian normalized volume form on \mathcal{X} . For each x\in\mathcal{X} and y\in\mathcal{Y} : W(\mu,\nu)=\inf_{\pi\in\Pi(\mu,\nu)}\int_{\mathcal{X}\times\mathcal{Y}}c(x,y)\,d\pi(x,y). (34) Definition 5 ( Dual of continuous OT ) . The dual of standard OT reads: W(\mu,\nu)=\sup_{f,g\in\mathcal{U}(c)}\int_{\mathcal{X}}fd\mu(x)+\int_{\mathcal{Y}}d\nu(y) (35) where the constraint set \mathcal{U}(c) is defined by \mathcal{U}(c):=\{(\mu,\nu)\in\mathcal{C}(\mathcal{X})\times\mathcal{C}(\mathcal{Y})\}|f(x)+g(y)\leq c(x,y)\} . Here, C(\mathcal{X}) is the space of all continuous functions on \mathcal{X} , the functions which measured using the supreme norm ||f||_{\infty} , with the Legendre transform: Definition 6 ( Legendre c-transforms ) . For the dual variables, or so called potentials, there exists the Legendre c-transforms: f^{c}(y):=\sup_{x\in\mathcal{X}}(-c(x,y)+f(x)),\quad g^{c}(x):=\sup_{y\in\mathcal{Y}}(-c(x,y)+g(y)). (36) In which g^{c}(x) and f^{c}(y) are Legendre c-transforms of g(y)\in\mathcal{C}(\mathcal{Y}) and f(x)\in\mathcal{C}(\mathcal{X}) with cost function c(x,y) . Definition 7 ( Pushforward measure ) . The pushforward measure of \mu under the map T , denoted as T_{\mu} , is a measure on \mathcal{X} defined by T_{\mu}(B)=\mu(T^{-1}(B)) for any Borel set B in \mathcal{X} . T_{\mu}=\nu when T is an optimal transport map. Following the similar way we can define the push-forward measure T_{\mu} and T_{\nu} as: T_{\mu}:C(\mathcal{X})\rightarrow C(\mathcal{Y}):=\log\int{e^{-c(x,\cdot)+f(x)}}{\mu(x)},~~T_{\nu}:C(\mathcal{Y})\rightarrow C(\mathcal{X}):=\log\int{e^{-c(\cdot,y)+g(y)}}{\nu(y)} (37) Definition 8 ( \varphi -divergence regularized OT in continuous ) . Given two dual variables (also called potentials) f\in\mathbb{R}^{n} and g\in\mathbb{R}^{m} for each marginal constraint, the entropy regularized optimal transport in Equation ( 3 ) could be transformed into some problems with the Kantorovich functional: \displaystyle W^{\varphi}_{\varepsilon,c}(\mu,\nu)=\inf_{\pi in\Pi(\mu,\nu)}(\int_{\mathcal{X}\times\mathcal{Y}}c(x,y)d\pi(x,y)+\varepsilon\int_{\mathcal{X}\times\mathcal{Y}}\varphi\left(\frac{d\pi(x,y)}{d\mu(x)d\nu(y)}\right)d\mu(x)d\nu(y)\Bigg) (38) Proposition 1 (Dual of EOT) . Consider OT between two probability measures \mu and \nu with a convex regularizer \phi on \mathbbm{R}^{+} in Equation ( 38 ) W^{\varphi}_{c,\varepsilon}(\mu,\nu)=\sup_{f,g\in\mathcal{C}(\mathcal{X})\times\mathcal{C}(\mathcal{Y})}\int_{\mathcal{X}}fd\mu(x)+\int_{\mathcal{Y}}d\nu(y)-\varepsilon\int_{\mathcal{X}\times\mathcal{Y}}\varphi^{*}(\frac{f(x)+g(y)-c(x,y)}{\varepsilon})d\mu(x)d\nu(y) (39) where \varphi^{*} is the Legendre transform of \varphi defined by \varphi^{*}({\bf v}):=\sup_{\bf x}{\bf x}{\bf v}-\phi({\bf x}) A good choice for \varphi^{*} is that the \varphi^{*}({\bf v})=e^{\bf v} . The entropy regularization term ensures the problem is solvable, especially for computational schemes. If the \varepsilon\rightarrow\infty , the optimal primal plan \pi^{*} can be retrieved using, which corresponds to the mutual information formula: \frac{d\pi^{*}}{d\mu d\nu}(x,y)=\exp\left(\frac{f^{*}(x)+g^{*}(y)-c(x,y)}{\varepsilon}\right) In the discrete version in Equation ( 3 ), the optimal transport plan {\bf P} can often be expressed in terms of the optimal transport map T^{*} when it exists, one can define the so-called barycentric projection map T^{*}:{\bf x}_{i}\in\mathcal{X}\mapsto\frac{1}{\mu_{i}}\sum_{j}{\bf P}_{i,j}\mathbf{y}_{j}\in\mathbb{R}^{d}, This link provides the connection between the mutual information with the optimal mapping: T^{*}:x\in X\mapsto\int_{Y}y\frac{d\pi(x,y)}{d\mu(x)d\nu(y)}d\nu(y). Note that the joint distribution \pi always has a density \frac{d\pi(x,y)}{d\mu(x)d\nu(y)} with respect to \mu\otimes\nu , and the mutual information method will lead us to the optimal solution. Appendix B Analysis of GCA B.1 Convergence of GCA In this section, we provide a proof of convergence in the forward pass for our GCA algorithm. To do this, we show the general form in Algorithms 1 for all Bregman divergence ( d_{\Gamma} ) in forward pass in GCA algorithms could be converged through Djkstra’s projection algorithms. Finally, we show the uniformly convergence of the transport plan {\bf P} , and the convergence of its dual variables f^{(t)} in each iteration. Algorithm A1 Generalized Contrastive Alignment (GCA) Input: Encoder f_{\theta} , projector g_{\theta} , data \{{\bf x}_{k}\}_{k=1}^{N} , batch size B , cost function c(x,y) , entropy parameter \varepsilon , constant \tau , total iterations T , marginal constraints \mu and \nu , relax items d_{1},d_{2} and constant \delta_{\text{eps}} , some divergence d_{M} and d_{\Gamma} (could be KL or WDM), for sampled minibatch \{{\bf x}_{k}\}_{k=1}^{B} do Generate two views ({\bf z}^{\prime}_{k},{\bf z}^{\prime\prime}_{k}) using f_{\theta} , g_{\theta} with randomly sampled augmentations. end for \mathbf{u}^{(0)}=\mathbbm{1} , \mathbf{v}^{(0)}=\mathbbm{1} , f=0 , g=0 , {\bf C}_{ij}=c({\bf z}^{\prime}_{i},{\bf z}_{j}^{\prime\prime}) d_{1}\leftarrow d_{1}/(d_{1}+\varepsilon) , d_{2}\leftarrow d_{2}/(d_{2}+\varepsilon) , {\bf K}=\exp(C_{ij}/\varepsilon^{-1}) for i=1 to T do \delta f\leftarrow\exp{-f/(\varepsilon+d_{1})} , \delta g\leftarrow\exp{-g/(\varepsilon+d_{2})} \mathbf{u}\leftarrow\delta f\cdot\text{Prox}_{\mathcal{F}}(K\mathbf{v}+\delta_{\text{eps}})^{fi} \mathbf{v}\leftarrow\delta g\cdot\text{Prox}_{\mathcal{G}}(K^{T}\mathbf{u}+\delta_{\text{eps}})^{fi} if \mathbf{u}>\tau or \mathbf{v}>\tau then f\leftarrow f+\varepsilon\cdot\log(\max(\mathbf{u})) , g\leftarrow g+\varepsilon\cdot\log(\max(\mathbf{v})) K\leftarrow\exp{(f+g-{\bf C})}/\varepsilon , \mathbf{v}=\mathbbm{1} end if end for \log\mathbf{u}\leftarrow f/(\varepsilon+\mathbf{u}) , \log\mathbf{v}\leftarrow g/(\varepsilon+\mathbf{v}) Compute transport plan as: {\bf P}\leftarrow\exp(\log\mathbf{u}+\log\mathbf{v}-C/\varepsilon) Normalize {\bf P}_{u}^{(T)} by its column sums. Loss: \mathcal{L}_{GCA}=d_{M}({\bf\mu\otimes\nu},{\bf P}_{u}^{(T)}) Update networks f_{\theta} and g_{\theta} to minimize \mathcal{L}_{GCA} B.1.1 Convergence of GCA-INCE Corollary 1 . (Convergence of GCA-INCE [ 43 ] ) Given {\bf u}^{*}=e^{f^{\infty}} , {\bf v}^{*}=e^{g^{(\infty)}} and a kernel space H with the Hilbert-Birkhoff metric d_{H}({\bf u},{\bf u}^{*}):=\log\max_{i,j}\frac{{\bf u}_{i}{\bf u}^{*}_{j}}{{\bf u}_{j}{\bf u}^{*}_{i}}, for all positive pairs ({\bf u},{\bf u}^{*}) , with {\bf u}^{(t)}\rightarrow{\bf u}^{*} and {\bf v}^{(t)}\rightarrow{\bf v}^{*} , we can prove that: \|\log{\bf P}^{(t)}-\log{\bf P}^{*}\|_{\infty}\leq d_{H}({\bf u}^{(t)},{\bf u}^{*})+d_{H}({\bf v}^{(t)},{\bf v}^{*}), (40) Proof for Corollary 1 : First, let’s define the following Hilbert space: \forall({\bf u},{\bf u}^{\prime})\in(\mathbb{R}^{n}_{+,*})^{2},\quad d_{H}({\bf u},{\bf u}^{\prime}):=\log\max_{i,j}\frac{{\bf u}_{i}{\bf u}^{\prime}_{j}}{{\bf u}_{j}{\bf u}^{\prime}_{i}}. For any pairs of vectors that ({\bf v},{\bf v}^{\prime})\in(\mathbb{R}^{m}_{+,*})^{2} holds: d_{H}({\bf v},{\bf v}^{\prime})=d_{H}\left(\frac{{\bf v}}{{\bf v}^{\prime}},\mathbbm{1}_{m}\right)=d_{H}\left(\mathbbm{1}_{m}/\mathbf{v},\mathbbm{1}_{m}/\mathbf{v}^{\prime}\right). (41) Let {\bf K}\in\mathbb{R}^{n\times m}_{+,*} , then for ({\bf v},{\bf v}^{\prime})\in(\mathbb{R}^{m}_{+,*})^{2} we have d_{H}({\bf u}^{(t+1)},{\bf u}^{*})=d_{H}\left(\frac{\mathbbm{1}_{n}}{{\bf K}\mathbf{{\bf v}}^{(t)}},\frac{\mathbbm{1}_{n}}{{\bf K}\mathbf{{\bf v}}^{*}}\right)=d_{H}({\bf K}\mathbf{{\bf v}}^{(t)},{\bf K}\mathbf{{\bf v}}^{*})\leq\lambda({\bf K})d_{H}(\mathbf{{\bf v}}^{(t)},\mathbf{{\bf v}}^{*}), where \lambda({\bf K}):=\frac{\sqrt{\eta({\bf K})-1}}{\sqrt{\eta({\bf K})+1}}<1,\quad\eta({\bf K}):=\max_{i,j,k,\ell}\frac{{\bf K}_{i,k}{\bf K}_{j,\ell}}{{\bf K}_{j,k}{\bf K}_{i,\ell}}. (42) Based on the contraction mapping theory, one has ({\bf u}^{(\ell)},{\bf v}^{(\ell)})\rightarrow({\bf u}^{*},{\bf v}^{*}) and \displaystyle d_{H}({\bf u}^{(t)},{\bf u}^{*}) \displaystyle\leq d_{H}({\bf u}^{(t+1)},{\bf u}^{(t)})+d_{H}({\bf u}^{(t+1)},{\bf u}^{*}) (43) \displaystyle\leq d_{H}\left(\frac{\mu}{{\bf K}\mathbf{{\bf v}}^{(t)}},{\bf u}^{(t)}\right)+\lambda({\bf K})^{2}d_{H}({\bf u}^{(t)},{\bf u}^{*}) (44) \displaystyle=d_{H}\left(\mu,{\bf u}^{(t)}\odot({\bf K}\mathbf{{\bf v}}^{(t)})\right)+\lambda({\bf K})^{2}d_{H}({\bf u}^{(t)},{\bf u}^{*}), (45) d_{H}({\bf u}^{(t)},{\bf u}^{*})\leq\frac{d_{H}({\bf P}^{(t)}\mathbbm{1}_{m},\mu)}{1-\lambda({\bf K})^{2}},\quad d_{H}({\bf v}^{(t)},{\bf v}^{*})\leq\frac{d_{H}({\bf P}^{(t)\top}\mathbbm{1}_{n},\nu)}{1-\lambda({\bf K})^{2}}, (46) where we denoted {\bf P}^{(t)}:=\operatorname{diag}({\bf u}^{(t)}){\bf K}\operatorname{diag}({\bf v}^{(t)}) . Last, one has \|\log({\bf P}^{(t)})-\log({\bf P}^{*})\|_{\infty}\leq d_{H}({\bf u}^{(t)},{\bf u}^{*})+d_{H}({\bf v}^{(t)},{\bf v}^{*}), (47) where {\bf P}^{*} is the unique solution of Equation ( 3 ). The above formula also shows that the t-step solution gives a better lower bound than the 1-step solution. ∎ B.1.2 Convergence of the Djkstra’s projection algorithms The previous subsection proved the convergence of GCA-INCE. Here, we extend this to show the convergence of all generalized proximal operators in Algorithm 1 . Additionally, we demonstrate that these operators can iteratively solve alignment problems in the forward pass, following Dykstra’s projection algorithm. We present a general convergence proof for Dykstra’s projection algorithm, sharing the form in Definition ( 1 ). First, we define d_{\Gamma} as a generic Bregman divergence on some convex set \mathcal{B} , and the proximal map of a convex function d_{\phi} according to this divergence is: \text{Prox}^{d_{\Gamma}}_{d_{\phi},\mathcal{B}}({\bf K}):=\arg\min_{\tilde{{\bf P}}\in\mathcal{B}}d_{\Gamma}(\tilde{{\bf P}}\|{\bf K})+d_{\phi}(\tilde{{\bf P}}). (48) \Gamma is a strictly convex function smooth on \text{int}(\mathcal{B}) , and \text{Prox}^{d_{\Gamma}}_{d_{\phi}}({\bf K})\in\text{int}(\mathcal{B}) is always uniquely defined by strict convexity. As \mathcal{B}=\text{dom}(\Gamma) , \forall({\bf P},{\bf K})\in\mathcal{B}\times\text{int}(\mathcal{B}),d_{\Gamma}({\bf P}\|{\bf K})=\Gamma({\bf P})-\Gamma({\bf K})-\langle\nabla\Gamma({\bf K}),{\bf P}-{\bf K}\rangle, which has its Legendre transform is also smooth and strictly convex: \Gamma^{*}(\rho)=\max_{{\bf P}\in\mathcal{B}}\langle{\bf P},\rho\rangle-\Gamma({\bf P}) In particular, one has that \nabla\Gamma and \nabla\Gamma^{*} are bijective maps between \text{int}(\mathcal{B}) and \text{int}(\text{dom}(\Gamma^{*})) such that \nabla\Gamma^{*}=(\nabla\Gamma)^{-1} . For \Gamma=||\cdot||^{2} , one recovers the squared Euclidean norm d_{\Gamma}=||\cdot||^{2} . One has KL = d_{\Gamma} for \Gamma({\bf P})=h({\bf P})=-\sum_{i,j=1}^{B}({\bf P}_{ij}(\log{\bf P}_{ij}-1)) . Dykstra’s algorithm starts by initializing {\bf P}^{(0)}:={\bf K} and \mathbf{U}^{(0)}=\mathbf{U}^{(-1)}:=0 . One then iterative defines, for k>0 , \displaystyle{\bf P}^{(k)} \displaystyle:=\text{Prox}^{d_{\Gamma}}_{d_{\phi_{[k]_{2}}}}(\nabla\Gamma^{*}(\nabla\Gamma({\bf P}^{(k-1)})+\mathbf{U}^{(k-2)})), (49) \displaystyle\mathbf{U}^{(k)} \displaystyle:=\mathbf{U}^{(k-2)}+\nabla\Gamma({\bf P}^{(k-1)})-\nabla\Gamma({\bf P}^{(k)}), (50) Proposition 2 . Giving d_{\phi_{1}},d_{\phi_{2}} are two proper, lower-semicontinuous convex functions defined on \mathcal{B} . We also assume that the following qualification constraint holds: ~\text{ri}(\text{dom}(d_{\phi_{1}}))\cap\text{ri}(\text{dom}(d_{\phi_{2}}))\cap\text{ri}(\text{dom}(d_{\Gamma}))=\emptyset, (51) where ri is the relative interior and \text{dom}(\phi)=\{\pi;\phi(\pi)=+\infty\} . Then the {\bf P}^{t} converges to the solution of the following equation: \text{prox}_{h,\mathcal{B}}^{d_{\Gamma}}({\bf K})=\arg\min_{{\bf P}\in\mathcal{B}}\{d_{\Gamma}({\bf P}\|{\bf K})+\lambda_{1}d_{\phi_{1}}({\bf P})+\lambda_{2}d_{\phi_{2}}({\bf P})\} (52) Proof of the Proposition 2 : Proof in [ 42 ] section 3.2. B.1.3 Convergence of Bregman projection This section aims to show for the continuous measure, the convergence of Bregman projection holds [ 4 ] . Finding the optimal transport map T could be derived in minimizes some functionals derived from the potential function f defined on \mathcal{X} , with Legendre transform in Equation ( 36 ) defined on \mathcal{Y} : ~J(f):=\int_{\mathcal{X}}f\mu(x)+\int_{\mathcal{Y}}f^{\mathsf{c}}\nu(y)=I_{\mu}-L (53) Lemma 3 ( Uniformly convergence ) . [ 4 ] When t_{1}\rightarrow\infty , f^{(t_{1})} converges uniformly to a fixed point f^{(\infty)} , with f^{(t_{1})}\leq f^{(\infty)} . Proof for the Lemma 3 ( Uniformly convergence ): We follow the procedures of methods in [ 4 ] . Giving push-forward measure T_{\mu} and T_{\nu} and a composed operator S=T_{\nu}\circ T_{\mu} , which yields an iteration on C(\mathcal{X}) as S:C(\mathcal{X})\rightarrow C(\mathcal{X}),f\rightarrow f\circ g\circ f , f^{(m+1)}=S(f^{(m)}) , and e^{S(f)-f}\mu is the probability measure on \mathcal{X} . Lemma 4 ( Existence and uniqueness ) . The following conditions are equivalent for a function f in the space C(X) , where C(X) denotes the space of continuous functions on a set X : • f is a critical point for the functional F on C(X) . • The function \exp(S(f)-f)=0 hold almost everywhere (a.e.) with respect to (w.r.t.) \mu . Moreover, if f is a critical point, then f^{*}:=S(f) is a fixed point for the operator S on C(X) . Proof of the Lemma 1 Consider the functional L defined in Equation ( 53 ), the differential of L at an element f\in C(X) is represented by the probability measure \exp({S(f)-f})\mu . For some iterations f^{(m+1)}-f^{(m)}=S(f^{(m)})-f^{(m)} , when f is a critical point (derivative is zero or undefined) for the functional J on C(X) , and f^{*}:=S(f) is a fixed point for the operator S on C(X) , proved by realizing for any \dot{f}\in C(X) : \left.\frac{d}{dt}L(f+t\dot{f})\right|_{t=0}=\int_{X}\dot{f}e^{(S(f)-f)}d\mu. (54) This follows readily from the definitions by differentiating t\mapsto g[(f+t\dot{f})] to get an integral over (X,\mu) and then switching the order of integration. As a consequence, f is a critical point of the functional F on C^{0}(X) if and only if e^{(S(f)-f)}\mu=\mu , i.e., if and only if e^{(S(f)-f)}=1 almost everywhere with respect to \mu . Finally, if this is the case, then S(f)=f almost everywhere with respect to \mu and hence S(S(f))=S(f) (since S(f) only depends on f viewed as an element in L^{1}(X,\mu) ). Lemma 5 . Given a point x_{0}\in X , the subset K_{x_{0}} of C(\mathcal{X}) defined as all elements f in the image of S satisfying f(x_{0})=0 is compact in C(\mathcal{X}) . Proof of the Lemma 5 : Based on the compactness of the product space \mathcal{X}\times\mathcal{Y} , the continuous function c is uniformly continuous on \mathcal{X} . So S(C(\mathcal{X})) is an equicontinuous family of continuous functions on X . By Arzelà-Ascoli theorem, it follows that the set K_{x_{0}} is compact in C(\mathcal{X}) . Proposition 3 . The operator S has a fixed point f^{*} in C(\mathcal{X}) . Moreover, f^{*} is uniquely determined a.e. wrt \mu up to an additive constant, and f^{*} minimizes the functional F . More precisely, there exists a unique fixed point in S(C(\mathcal{X}))/\mathbb{R} . Proof of the Proposition ( 3 ): Then based on the Jensen’s inequality, we have \displaystyle I_{\mu}(f^{(m+1)})-I_{\mu}(f^{(m)})=\int\log{\exp{(S(f^{(m)})-f^{(m)}}})d\mu\leq\log\int{\exp{(S(f^{(m)})-f^{(m)})}}{d\mu}=0, (55) \displaystyle L(f^{(m)})-L(f^{(m+1)})=\int\log{\exp{(S(g^{(m)})-g^{(m)})}}{d\nu}\leq\log\int{\exp{(S(g^{(m)})-g^{(m)})}}{d\nu}=0. (56) So we know the functionals are strictly decreasing at f^{(m)} unless S(f^{*})=f^{*} for f^{*}:=S(f^{(m)}) . Then based on the Lemma 5 , we know for each initial data f_{0} , the closure of its images denoted as K_{f_{0}} in C(\mathcal{X})/\mathbb{R} is compact, under the operator S . Hence, f^{(m)}\rightarrow f^{(\infty)} in C(\mathcal{X})/\mathbb{R} . And J is decreasing along the orbit but has lower bound: J(f^{(\infty)})=\inf_{K_{f^{(0)}}}J. By the condition for strict monotonicity, it must be that S(f^{(\infty)})=f^{(\infty)} a.e. wrt \mu . It then follows from the Proposition ( 3 ) that f^{(\infty)} is uniquely determined in C(\mathcal{X})/\mathbb{R} (by the initial data f^{(0)} ), i.e. the whole sequence converges in C(\mathcal{X})/\mathbb{R} . We first show that there exists a number \lambda\in\mathbb{R} such that \lim_{m\rightarrow\infty}I_{\mu}(f^{(m)})=\lambda. I_{\mu} is decreasing and hence it is enough to show that I_{\mu}(f^{(m)}) is bounded from below. By I_{\mu}=J+L , and J is bounded from below (by F(f^{(\infty)}) ). Moreover, by the first step L(f^{(m)})\geq L(f^{(0)}) . Next, decompose f^{(m)}=\tilde{f}^{(m)}+f^{(m)}(x_{0}), By the Lemma 5 the sequence (\tilde{f}^{(m)} is relatively compact in C(\mathcal{X}) and we claim that |f^{(m)}(x_{0})|\leq C for some constant C . Indeed, if this is not the case then there is a subsequence f^{(m_{j})} such that |f^{(m_{j})}|\rightarrow\infty uniformly on X . But this contradicts that I_{\mu}(f^{(m)}) is uniformly bounded. It follows that the sequence (f^{(m)}) is also relatively compact. Hence, by the previous step the whole sequence f^{(m)} converges to the unique minimizer f^{*} of F in S(C(\mathcal{X})) satisfying I_{\mu}(f^{*})=\lambda . B.2 GCA version of unbalanced optimal transport (GCA-UOT) In this section, we are going to introduce the relaxation of the EOT plan as Unbalanced optimal transport plan (UOT). And its relationship with the dual formula of EOT. Here we need to emphasize that the GCA-UOT not just add constraint to the proximal operators which computes the coupling matrix {\bf P}_{\theta} , but also add the penalty (i.e. KL-divergence) to the loss function d_{M} . For the specific function we used in the method of GCA-UOT in Table 2 , we employed a version with the loss in Equation ( 11 ) plus the loss in Equation ( 10 ) with a weight control parameter. B.2.1 Explanation of the unbalanced OT Unbalanced optimal transport (UOT) in Equation ( 9 ) seeks to generalize the OT problem in Equation ( 24 ) by allowing for the relaxation of these constraints [ 11 ] , as penalization by certain divergence measures d_{\phi_{1}} and d_{\phi_{2}} (e.g., Kullback-Leibler divergence). Here we provide the unbalanced OT for the entropic regularization optimal transport in Equation ( 3 ), which ensure that the transported mass respects the given source \mu and target distributions \nu : \mathbf{U}_{OT}(\mu,\nu)=\min_{\bf P}\left\langle{\bf P},{\bf C}\right\rangle+\lambda_{1}d_{\phi_{1}}({\bf P}\mathbbm{1}||\mu)+\lambda_{2}d_{\phi_{2}}({\bf P}^{\top}\mathbbm{1}||\nu)+\varepsilon H({\bf P}) (57) Here \left\langle P,C\right\rangle represents the total transport cost. \lambda_{1} and \lambda_{2} are regularization parameters that control the trade-off between the transport cost and the divergence penalties. B.2.2 Connection to dual formula of EOT Lemma 6 . The entropy regularized OT problem is a special case of a structured convex optimization problem of Equation ( 57 ) the by giving functions h_{\mathcal{F}} and h_{\mathcal{G}} , h_{\mathcal{F}}=\iota_{\{C_{1}^{\mu}\}} and h_{\mathcal{G}}=\iota_{\{C_{2}^{\nu}\}} , as the indicator function of a closed convex set C_{1}^{\mu}\coloneqq\{{\bf P}:{\bf P}\mathbbm{1}_{m}=\mu\},C_{2}^{\nu}\coloneqq\{{\bf P}:{\bf P}^{\top}\mathbbm{1}_{n}=\nu\}. \min_{{\bf P}}\langle{\bf P},{\bf C}\rangle+\varepsilon H({\bf P})+h_{\mathcal{F}}({\bf P}\mathbbm{1}_{m})+h_{\mathcal{G}}({\bf P}^{\top}\mathbbm{1}_{n}).\quad\iota_{C}(x)=\begin{cases}0&\text{if }x\in C,\\ +\infty&\text{otherwise},\end{cases} (58) Proof of the Lemma 6 : Let’s start with the dual formula of the Equation ( 3 ) with \mathcal{B}=C_{1}^{\mu}\cap C_{2}^{\nu} , we can introduce the Lagrangian \mathcal{E}({\bf P},f,g) of Equation ( 3 ) reads: \displaystyle\text{Prox}^{\text{KL}}_{\mathcal{B}}({\bf K}) \displaystyle:=\min_{{\bf P}\in\mathcal{B}}\langle{\bf P},{\bf C}\rangle-\varepsilon H({\bf P})=\mathcal{E}({\bf P},f,g) (59) \displaystyle=\min_{P}\max_{f\in\mathbb{R}^{n},g\in\mathbb{R}^{m}}\langle{\bf P},{\bf C}\rangle-\varepsilon H({\bf P})-\langle f,{\bf P}\mathbbm{1}_{m}-\mu\rangle-\langle g,{\bf P}^{T}\mathbbm{1}_{n}-\nu\rangle.\qed (60) To solve this problem, we can use the first order condition: \frac{\partial\mathcal{E}({\bf P},f,g)}{\partial{\bf P}_{ij}}={\bf C}_{ij}+\varepsilon\log({\bf P}_{ij})-f_{i}-g_{j}=0\quad\Rightarrow\quad log{\bf P}=\frac{1}{\varepsilon}(f\mathbbm{1}_{m}^{T}+\mathbbm{1}_{n}g^{T}-{\bf C}) (61) The solution to the Equation ( 3 ) is unique with scaling variabl ({\bf u},{\bf v})\in\mathbb{R}^{n}_{+}\times\mathbb{R}^{m}_{+} in Equation ( 23 ). And each items in the optimal transport matrix {\bf P} is, and optimal (f,g) are linked to non-negative vectors ({\bf u},{\bf v}) through ({\bf u},{\bf v})=(e^{f/\varepsilon},e^{g/\varepsilon}) . {\bf P}_{ij}=e^{f_{i}/\varepsilon}e^{-{\bf C}_{ij}/\varepsilon}e^{g_{j}/\varepsilon}={\bf u}_{i}{\bf K}_{ij}{\bf v}_{j},\quad(f^{(t)},g^{(t)})=\varepsilon(\log({\bf u}^{(t)}),\log({\bf v}^{(t)})), (62) B.3 Equivalence of INCE objective with single step Bregman projection In this section, we are going to discuss how to build the equivalence between minimizing the KL-divergence d_{M} between the {\bf P}^{(1)} and the {\bf P}_{\text{t}gt} with respect to \theta in GCA objective: \min_{\theta}\text{KL}\big({\bf I}||\text{Prox}_{C_{1}^{\mu}}^{KL}({\bf K}_{\theta})), with the INCE loss minimization in Equation ( 1 ). Here {\bf P}^{(1)} is the nearest point of {\bf K}_{\theta} on constraint set C_{1}^{\mu} measured by the KL-divergence d_{\Gamma} defined in Equation ( 18 ), through one step of proximal operator (Bregman projection). And {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ) with cosine similarity. B.3.1 Proof of the Theorem 10 Suppose we had a encoder f_{\theta} with parameter \theta in INCE, with \widetilde{f}_{\theta} to represent its normalized form,then we can use the following proposition to assist our proof: Proposition 4 . Given the cost matrix as {\bf C}_{i,j}=1-\widetilde{f}_{\theta}({\bf x}_{i}^{\prime})^{\top}\widetilde{f}_{\theta}({\bf x}_{j}^{\prime\prime}) , and Gibbs kernel {\bf K}_{\theta}=\exp(-{\bf C}_{i,j}/\varepsilon) , based on the cosine dissimilarity scores of the inner products \langle{\bf z}_{\theta i},{\bf z}_{\theta j}\rangle , with {\bf z}_{i}=\frac{f_{\theta}({\bf x}^{\prime}_{i})}{\|f_{\theta}({\bf x}^{\prime}_{i})\|} and {\bf z}_{j}=\frac{f_{\theta}({\bf x}^{\prime\prime}_{j})}{\|f_{\theta}({\bf x}^{\prime\prime}_{j})\|} . Set d_{M} and d_{\Gamma} to KL-divergence, and the target transport plan {{\bf P}}_{\text{tgt}}=\bf I . The probability matrix {\bf P} after one-step Bregman iteration of entropy optimal transport problem could be represented as: {\bf P}_{ij}=\frac{{\bf K}_{\theta ij}}{\sum_{j=1}^{B}{\bf K}_{\theta ij}}=\frac{\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{j}\rangle\right)}{\sum_{j=1}^{B}\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{k}\rangle\right)} (63) Proof of the Proposition ( 4 ) : We assume that gibbs kernel {\bf K}_{\theta} is a matrix which can be expressed as: {\bf K}_{\theta ij}=\exp\left(-\varepsilon^{-1}{\bf C}_{i,j}\right)=\exp\left(-\varepsilon^{-1}|1-\langle{\bf z}_{i},{\bf z}_{j}\rangle|\right), with a temperature parameter \varepsilon . \mu , \nu , {\bf u}^{(0)} and {\bf v}^{(0)} can be initialized as a vector of ones with the same size as B, the batch size, \mu=\mathbbm{1},\nu=\mathbbm{1}\quad{\bf u}^{(0)}=\mathbbm{1},{\bf v}^{(0)}=\mathbbm{1}. For t iterations of the Sinkhorn algorithm, {\bf u}^{(t)} is updated as: \mathbf{{\bf u}}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{\mu}{{\bf K}_{\theta}\mathbf{{\bf v}}^{(t)}},\quad\mathbf{{\bf v}}^{(t+1)}\stackrel{{\scriptstyle\text{ def }}}{{=}}\frac{\nu}{{\bf K}_{\theta}^{T}\mathbf{{\bf u}}^{(t)}}. So we know that: {\bf u}^{(1)}=\frac{1}{\sum_{j=1}^{b}\mathbf{K}_{\theta ij}}. Thus, half-step sinkhorn iteration or one-step Bregman interation for {\bf P} can be expressed as: {\bf P}_{ij}={\bf u}^{(1)}_{i}\mathbf{K_{\theta}}_{ij}{\bf v}^{(0)}_{j}=\frac{{\bf K}_{\theta ij}}{\sum_{j=1}^{b}{\bf K}_{\theta ij}}=\frac{\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{j}\rangle\right)}{\sum_{j=1}^{b}\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{k}\rangle\right)}\qed This concludes the expressions of {\bf P} at half-step iteration. Reminds us the formula of the KL divergence \text{KL}(\mathbf{I}\|{\bf P}) and the entropy H({\bf P}) : \text{KL}(\mathbf{I}\|{\bf P})\ \stackrel{{\scriptstyle\text{ def }}}{{=}}\sum_{i,j}\mathbf{I}_{i,j}\log\frac{\mathbf{I}_{i,j}}{{\bf P}_{i,j}}-\mathbf{I}_{i,j}+{\bf P}_{i,j},\quad\text{where}\quad\mathbf{I}_{i,j}\log\frac{\mathbf{I}_{i,j}}{{\bf P}_{i,j}}=0,\quad\text{if}\quad\mathbf{I}_{i,j}=0. (64) And after the batch normalization of {\bf P} , the value of \sum_{i,j}{\bf P}_{i,j} is equal to the batch size B and exactly the same as the \sum_{i,j}\mathbf{I}_{i,j} , we can obtain: \operatorname{KL({\bf I}\|{\bf P}})=\sum_{i}\log\left(\frac{1}{{\bf P}_{ii}}\right)=-\sum_{i}\log\frac{\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{i}\rangle\right)}{\sum^{b}_{j=1}\exp\left(\varepsilon^{-1}\langle{\bf z}_{i},{\bf z}_{j}\rangle\right)} j represents the elements on the diagonal of the similarity matrix, which is the same structure as the INCE loss as: \mathcal{L}_{\text{INCE}}=-\sum_{i}\log\left(\frac{\exp(f_{\theta}({\bf x}^{\prime}_{i})^{\top}f_{\theta}({\bf x}^{\prime\prime}_{i}))}{\sum_{j=1}^{b}\exp(f_{\theta}({\bf x}^{\prime}_{i})^{\top}f_{\theta}({\bf x}^{\prime\prime}_{j}))}\right)\qed B.4 Proximal operator version of RINCE In this section, we are going to discuss how to build the equivalence between minimizing the some convex function of d_{M} with adjustable parameters q and \lambda between the {\bf P}^{(1)} and the {\bf P}_{\text{t}gt} as: d_{M}(\mathbf{I},{\bf P})=-\frac{1}{q}\left(\left(\frac{\operatorname{diag}({\bf P}^{(1)}_{\theta})}{{\bf u}^{(1)}}\right)^{q}-\left(\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}}\right)^{q}\right) (65) with respect to \theta in GCA objective: L_{\text{RINCE}}^{\lambda,q}=\min_{\theta}-\frac{1}{q}\bigg(\frac{\operatorname{diag}({\bf P}^{(1)}_{\theta})}{{\bf u}^{(1)}}\bigg)^{q}+\frac{1}{q}\bigg(\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}}\bigg)^{q},~~\text{with}~~{\bf P}^{(1)}_{\theta}=\text{Prox}_{C_{1}^{\mu}}^{\text{KL}}({\bf K}_{\theta}),~~{\bf u}^{(1)}=\text{diag}\left(\frac{\mu}{{\bf P}^{(0)}\mathbbm{1}}\right) with the RINCE loss minimization in Equation ( 2 ). Here {\bf P}^{(1)} is the nearest point of {\bf K}_{\theta} on constraint set C_{1}^{\mu} measured by the KL-divergence d_{\Gamma} defined in Equation ( 18 ), through one step of proximal operator (Bregman projection). And {\bf K}_{\theta} denote the augmentation kernel as in Definition ( 3 ) with cosine similarity. Also, we are going to discuss when the q=1, RINCE loss is the symmetry loss, which provides the robustness in the noisy view. B.4.1 Proof of the Theorem 2 The loss function of RINCE looks like: \mathcal{L}^{\lambda,q}_{\text{RINCE}}=\frac{1}{q}\big(-e^{qs_{ii}}+\lambda^{q}(e^{s_{ii}}+\sideset{}{{}_{i\neq j}}{\sum}e^{s_{ij}})^{q}\big) (66) For the specific parameters \theta , we record the normalized latent of the {\bf z}^{i}_{\theta+}=s_{ii} , and {\bf z}^{i}_{\theta-}=s_{ij},j\neq i . The positive pairs are stored in the diagonal of the gibbs kernel {\bf K}_{\theta} , and the negative pairs are stored in the off-diagonal elements, which means: \displaystyle{\bf K}_{ii}=\exp\left(-\varepsilon^{-1}{\bf C}_{i,i}\right)=\exp\left(-\varepsilon^{-1}|1-\langle{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i}\rangle|\right)=\exp\left(\varepsilon^{-1}{\langle{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i}\rangle}-\varepsilon^{-1}\right)\propto e^{{\bf z}^{i}_{\theta+}}. (67) \displaystyle{\bf K}_{ij}=\exp\left(-\varepsilon^{-1}{\bf C}_{i,j}\right)=\exp\left(\varepsilon^{-1}{\langle{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta j}\rangle}-\varepsilon^{-1}\right)\propto e^{{\bf z}^{i}_{\theta-}},j\neq i. (68) By solving the \langle{\bf u}^{(1)}{\bf K},\mathbbm{1}\rangle=\mu in the Equation ( 21 ), we have the ith column elements \sum_{j=1}{\bf K}_{\theta ij}=\frac{\mu}{{\bf u}^{(1)}_{i}} , in which {\bf u}^{(1)} is given in 21 : \displaystyle\frac{\mu}{{\bf u}^{(1)}_{i}}=\sum^{B}_{j=1}{\bf K}_{\theta ij}=\frac{1}{e^{\varepsilon^{-1}}}(e^{\varepsilon^{-1}\langle{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i}\rangle}+\sum_{j=1,j\neq i}^{B}e^{\varepsilon^{-1}{\langle{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta j}\rangle}}),i\neq j, (69) \displaystyle\operatorname{diag}({\bf K}_{\theta})=\frac{e^{{\bf z}^{i}_{\theta+}}}{e^{\varepsilon^{-1}}}=\frac{\operatorname{diag}({\bf P}^{(1)})}{{\bf u}^{(1)}}. (70) The diagonal of K matrix contains the positive views and the marginal distribution of the u contains the negative view, we have: L^{\lambda,q}_{\text{RINCE}}(s_{\theta}^{i})=-\frac{e^{qs^{i}_{\theta+}}}{q}+\frac{(\lambda\cdot(e^{s^{i}_{\theta+}}+\sum_{j=1,j\neq i}^{B}e^{s_{\theta-}^{ij}}))^{q}}{q}\propto-\frac{\operatorname{diag}({{\bf K}_{\theta}})_{ii}^{q}}{q}+\frac{(\lambda\cdot(\sum_{j=1}^{B}{{\bf K}_{\theta}}_{ij}))^{q}}{q} (71) Furthermore, we have: -E(L_{\text{RINCE}}^{\lambda,q}({\bf K}_{\theta}))=\frac{1}{q}\bigg(\operatorname{diag}({\bf K}_{\theta})\bigg)^{q}-\frac{1}{q}\bigg(\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}}\bigg)^{q}. (72) where {\bf P}^{(0)}=\operatorname{diag}(\mathbbm{1}){\bf K}_{\theta}\operatorname{diag}(\mathbbm{1}) , {\bf P}^{(1)}=\operatorname{diag}({{\bf u}^{(1)}}){\bf K}_{\theta}\operatorname{diag}(\mathbbm{1}) , we have: L_{\text{RINCE}}^{\lambda,q}({\bf P}_{\theta}^{(1)})=-\frac{1}{q}(\frac{\operatorname{diag}({\bf P}^{(1)})}{{\bf u}^{(1)}})^{q}+\frac{1}{q}(\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}})^{q}. (73) B.4.2 Proof of the Symmetry and robustness of RINCE Symmetry loss is said to be noise tolerant as the classifier will keep performance with the label noise in Empirical Risk Minimization (ERM) . In many practical machine learning scenarios, we aim to select a model or function f_{\theta} that minimizes the expected loss across all possible inputs and outputs from a distribution \mathcal{D} , which is typically unknown. Instead of minimizing the true risk, which is often not feasible due to the unknown distribution \mathcal{D} , we minimize what is called the empirical risk \hat{R}_{L}(\widetilde{f}_{\theta}) , which is defined as the average loss over the training dataset of size B , which consists of independently and identically distributed (iid) data points. Mathematically, it is given by the following formula: \hat{R}_{L}(f_{\theta})=\frac{1}{B}\sum_{i=1}^{B}L(\widetilde{f_{\theta}}({\bf x}_{i}),\mathbf{y}_{i}) (74) Here, L(\widetilde{f_{\theta}}({\bf x}_{i}),\mathbf{y}_{i}) represents the loss function, which measures the discrepancy between the predicted value \widetilde{f_{\theta}}({\bf x}_{i}) and the true value \mathbf{y}_{i} . The function \widetilde{f}_{\theta} that minimizes this empirical risk is chosen as the model for making predictions. This approach is based on the assumption that minimizing the empirical risk will also approximate the minimization of the true risk, especially as the size of the training set increases. First we show the symmetry loss is robust to the noisy view with the following Lemma [ 20 ] , which means they will achieve the same performance in ERM with the noisy labels. Then we show RINCE satisfy the symmetry condition when q=1 , so the lemma is: Lemma 7 . Give a loss function L(\widetilde{f}_{\theta}({\bf x}),\mathbf{y}) exhibits a certain symmetry for some positive constant K , with respect to the labels \mathbf{y}=1 and \mathbf{y}=-1 : ~L(\widetilde{f}_{\theta}({\bf x}),1)+L(\widetilde{f}_{\theta}({\bf x}),-1)=K,\quad\forall x,\forall f,\quad\text{(Symmetry)} (75) Symmetry loss is noise tolerant given the label noise \eta<0.5 , which corresponds to the flipped labels: P_{D}[\text{sign}(\widetilde{f}^{*}_{\theta}(x))=\mathbf{y}_{\bf x}]=P_{D}[\text{sign}(\widetilde{f}^{*}_{\theta\eta}({\bf x}))=\mathbf{y}_{\bf x}],\quad\text{(Noisy tolerant)} (76) Proof of the Lemma 7 is in [ 20 ] . Second we show the RINCE loss is a symmetry loss with q\rightarrow 1 , so we have the Equation ( 2 ): L^{\lambda,q=1}_{\text{RINCE}}=-e^{{\bf z}^{ii}_{\theta+}}+\lambda\cdot(e^{{\bf z}^{ii}_{\theta+}}+\sum_{j=1,j\neq i}^{B}e^{{\bf z}_{\theta-}^{ij}}) (77) As we know that this formula has the same structure as the exponential loss function: L({\bf z}_{\theta},\mathbf{y})=-\mathbf{y}e^{{\bf z}_{\theta}} . To check for symmetry, we define a new binary classification loss function as: \tilde{L}_{x}({\bf z}_{\theta}({\bf x}),\mathbf{y})=B+L_{x}(\widetilde{f}_{\theta}({\bf x}),\mathbf{y})=B-\mathbf{y}\cdot e^{\widetilde{f}_{\theta}({\bf x})}\geq 0 where the prediction score \widetilde{f}_{\theta}({\bf x}) is bounded by s_{\text{max}}=\log(B) . Then we can establish that the loss satisfies the symmetry property: \tilde{L}(\widetilde{f}_{\theta}({\bf x}),1)+\tilde{L}(\widetilde{f}_{\theta}({\bf x}),-1)=2B (78) So we prove that this loss function is symmetry. B.5 Proof for RINCE is the upper bound of the 1-Wasserstein distance In this section, we are trying to build the connection when change the d_{M} from the KL-divergence in Equation ( 10 ) to the 1-Wasserstein distance in Equation ( 12 ), when q=1 in the RINCE loss. B.5.1 Proof of the Theorem 3 WDM [ 12 ] is proposed as a replacement for the KL divergence by Wasserstein distance in Mutual Information estimation. The Wasserstein distance between the joint distribution \pi on \mathcal{X}\times\mathcal{Y} and the product of the marginal distributions \mu and \nu on \mathcal{X} and \mathcal{Y} , respectively, is given by: W(\pi,\mu\otimes\nu)=\sup_{f\in\mathcal{C}(\mathcal{X}\times\mathcal{Y})}\left(\mathbb{E}_{\pi(x,y)}[f(x,y)]-\mathbb{E}_{\mu\otimes\nu(x,y)}[f(x,y)]\right) where \mathcal{C}(\mathcal{X}\times\mathcal{Y}) denotes the set of all 1-Lipschitz functions from \mathcal{X}\times\mathcal{Y} to \mathbb{R} . A function f:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R} is defined to be 1-Lipschitz if, for any two points (x_{1},y_{1}),(x_{2},y_{2})\in\mathcal{X}\times\mathcal{Y} , the following condition is satisfied: |f(x_{1},y_{1})-f(x_{2},y_{2})|\leq d((x_{1},y_{1}),(x_{2},y_{2})) where d((x_{1},y_{1}),(x_{2},y_{2})) denotes the metric on \mathcal{X}\times\mathcal{Y} typically defined, for example, by the Euclidean distance: d((x_{1},y_{1}),(x_{2},y_{2}))=\sqrt{(x_{1}-x_{2})^{2}+(y_{1}-y_{2})^{2}} Based on the Lipschitz continuity and inner product, it is easy to know for two given point (x_{1},y_{1}) , (x_{2},y_{2}) , the following properties hold with -\frac{1}{\varepsilon}\leq s\leq\frac{1}{\varepsilon} , which implies \lvert\nabla_{s}e^{s}\rvert\leq e^{1/\varepsilon} . Therefore, by the mean value theorem, we have: ~\begin{split}&\lvert e^{x_{1}^{T}y_{1}/\varepsilon}-e^{x_{2}^{T}y_{2}/\varepsilon}\rvert\leq e^{1/\varepsilon}\frac{1}{\varepsilon}\lvert\langle x_{1},y_{1}\rangle-\langle x_{2},y_{2}\rangle\rvert=e^{1/\varepsilon}\frac{1}{\varepsilon}\lvert\langle x_{1}-x_{2},y_{1}\rangle+\langle x_{2},y_{1}-y_{2}\rangle\rvert\\ &\leq e^{1/\varepsilon}\frac{1}{\varepsilon}\left(\|x_{1}-x_{2}\|\|y_{1}\|+\|y_{1}-y_{2}\|\|x_{2}\|\right)=e^{1/\varepsilon}\frac{1}{\varepsilon}\left(\lVert x_{1}-x_{2}\rVert+\lVert y_{1}-y_{2}\rVert\right)\end{split} (79) Consider two pairs of views, ({\bf z}^{\prime}_{\theta 1},{\bf z}^{\prime\prime}_{\theta 1}) and ({\bf z}^{\prime}_{\theta 2},{\bf z}^{\prime\prime}_{\theta 2}) , sampled from the joint distribution \pi of \mu and \nu . Thus, each pair ({\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i}) for i=1,2 represents a sample from the joint distribution \pi , where {\bf z}^{\prime}_{\theta i}\sim\mu and {\bf z}^{\prime\prime}_{\theta i}\sim\nu . The RINCE loss is a symmetry loss with q=1 , so we have the Equation ( 2 ): L^{\lambda,q=1}_{\text{RINCE}}=-e^{{\bf z}^{ii}_{\theta+}}+\lambda\cdot(e^{{\bf z}^{ii}_{\theta+}}+\sum_{j=1,j\neq i}^{B}e^{{\bf z}_{\theta-}^{ij}}),\begin{cases}{\bf z}^{ii}_{\theta+}=\varepsilon^{-1}\widetilde{f_{\theta}}({\bf x}^{\prime}_{i})^{\top}\widetilde{f_{\theta}}({\bf x}^{\prime\prime}_{i}),&\text{for }i=i,\\ {\bf z}_{\theta-}^{ij}=\varepsilon^{-1}\widetilde{f_{\theta}}({\bf x}^{\prime}_{i})^{\top}\widetilde{f_{\theta}}({\bf x}^{\prime\prime}_{j}),&\text{for }i\neq j.\end{cases} (80) So we know that: \displaystyle-\mathbb{E}(L_{\text{RINCE}}^{\lambda,q=1}(z_{\theta}))=\mathbb{E}_{\begin{subarray}{c}{\bf z}^{\prime}_{\theta i}\sim\mu\\ {\bf z}^{\prime\prime}_{\theta i}\sim\nu|\mu={\bf z}^{\prime}_{\theta i}\\ {\bf z}^{\prime\prime}_{\theta j}\sim\nu\end{subarray}}\left[(1-\lambda)e^{\varepsilon^{-1}{\bf z}^{\prime T}_{\theta i}{\bf z}^{\prime\prime}_{\theta i}}-\lambda\sum_{j=1}^{B-1}e^{\varepsilon^{-1}{\bf z}^{\prime T}_{\theta i}{\bf z}^{\prime\prime}_{\theta j}}\right] \displaystyle=\mathbb{E}_{({\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i})\sim\pi}\left[(1-\lambda)e^{\frac{{\bf z}^{\prime T}_{\theta i}{\bf z}^{\prime\prime}_{\theta i}}{\varepsilon}}\right]-\lambda(B-1)\mathbb{E}_{{\bf z}^{\prime T}_{\theta i}\sim\mu,{\bf z}^{\prime\prime T}_{\theta j}\sim\nu}\left[e^{\frac{{\bf z}^{\prime T}_{\theta i}{\bf z}^{\prime\prime}_{\theta j}}{\varepsilon}}\right] \displaystyle\leq(1-\lambda)\left(\mathbb{E}_{({\bf z}^{\prime}_{\theta},{\bf z}^{\prime\prime}_{\theta})\sim\pi}\left[e^{\frac{{\bf z}^{\prime T}_{\theta}{\bf z}^{\prime\prime}_{\theta}}{\varepsilon}}\right]-\mathbb{E}_{{\bf z}^{\prime T}_{\theta}\sim\mu,{\bf z}^{\prime\prime}_{\theta}\sim\nu}\left[e^{\frac{{\bf z}^{\prime T}_{\theta}{\bf z}^{\prime\prime}_{\theta}}{\varepsilon}}\right]\right)(\text{Giving setting}~\lambda(B-1)>1-\lambda) If we give two couples of two views ( {\bf z}^{\prime}_{\theta 1},{\bf z}^{\prime\prime}_{\theta 1} ) and ( {\bf z}^{\prime}_{\theta 2},{\bf z}^{\prime\prime}_{\theta 2} ) from joint distribution \pi of \mu and \nu , {\bf z}^{\prime}_{\theta}\sim\mu and {\bf z}^{\prime\prime}_{\theta}\sim\nu , which means to maximize: \displaystyle\lvert e^{\varepsilon^{-1}{\bf z}^{\prime T}_{\theta 1}{\bf z}^{\prime\prime}_{\theta 1}}-e^{\varepsilon^{-1}{\bf z}^{\prime T}_{\theta 2}{\bf z}^{\prime\prime}_{\theta 2}}\rvert \displaystyle\leq(1-\lambda)e^{\frac{1}{\varepsilon}}\frac{1}{\varepsilon}\left(\|{\bf z}^{\prime}_{\theta 1}-{\bf z}^{\prime}_{\theta 2}\|\|{\bf z}^{\prime\prime}_{\theta 1}\|+\|{\bf z}^{\prime\prime}_{\theta 1}-{\bf z}^{\prime\prime}_{\theta 2}\|\|{\bf z}^{\prime}_{\theta 2}\|\right)\text{(Mean value theorem from Equation~\eqref{eq:lip})} \displaystyle=(1-\lambda)e^{\frac{1}{\varepsilon}}\frac{1}{\varepsilon}\left(\lVert{\bf z}^{\prime}_{\theta 1}-{\bf z}^{\prime}_{\theta 2}\rVert_{2}+\lVert{\bf z}^{\prime\prime}_{\theta 1}-{\bf z}^{\prime\prime}_{\theta 2}\rVert_{2}\right) \displaystyle=(1-\lambda)e^{\frac{1}{\varepsilon}}\frac{1}{\varepsilon}d\left(({\bf z}^{\prime}_{\theta 1},{\bf z}^{\prime\prime}_{\theta 1}),({\bf z}^{\prime}_{\theta 2},{\bf z}^{\prime\prime}_{\theta 2})\right) \displaystyle\leq(1-\lambda)e^{1/\varepsilon}\frac{1}{\varepsilon}W_{1}(\pi,\mu\ \otimes\nu). B.6 Proof of connection with BYOL In this section, we are going to who how the change of the augmetation kernel from the {\bf K}_{\theta} in Definition ( 3 ) into the BYOL kernel \mathbf{S}_{\theta} would lead to the BYOL loss. Proof for the Theorem 4 BYOL has the online network parameterized by \theta and target network parameterized by \xi , where {\bf z}^{\prime}_{\theta}=\widetilde{f}_{\theta}({\bf x}^{\prime}) and {\bf z}^{\prime\prime}_{\xi}=\widetilde{f}_{\xi}({\bf x}^{\prime\prime}) are the normalized outputs of the online and target networks, respectively. The kernel of BYOL looks like: \mathbf{S}_{\theta}({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{j})=\exp(-\langle\widetilde{q}_{\theta}(\widetilde{f}_{\theta}({\bf x}^{\prime}_{i})),\widetilde{f}_{\xi}({\bf x}^{\prime\prime}_{j})\rangle), The kernel here involves both the parameters \theta and \xi , however, the target network has the stop gradient. Therefore, the only \theta needs to be updated, so we can rewrite the kernel as \mathbf{S}_{\theta}({\bf x}^{\prime}_{i},{\bf x}^{\prime\prime}_{j}) as we show in the main text. As we give in the equation, the corresponding proximal operators evolving with d_{\Gamma} is equal to L2-distance has the formula, and h(x)=0 for all {\bf P}\in\mathcal{R}^{B\times B} : \text{Prox}_{\mathcal{R}^{B\times B}}^{\|\cdot\|^{2}}(\mathbf{\mathbf{S}_{\theta}})=\arg\min_{\mathbf{P}\in\mathcal{R}^{B\times B}}\left\{h(\mathbf{P})+\frac{1}{2}\|\mathbf{P}-\mathbf{S}_{\theta}\|^{2}_{2}\right\}\Rightarrow{\bf P}=\mathbf{S}_{\theta} The BYOL loss can be written as normalized L2-distance between the normalized output after online network \widetilde{q}_{\theta}({\bf z}^{\prime}_{\theta}) in which \widetilde{q}_{\theta} is predictor and the stop gradient results for the target network \widetilde{q}_{\theta}({\bf z}^{\prime}) , and the formula of BYOL object reads as L_{\text{BYOL}}=\|\widetilde{q}_{\theta}({\bf z}^{\prime}_{\theta})-{\bf z}^{\prime\prime}_{\xi}\|_{2}^{2} . In this case, there exists equivalence between \text{KL}(\mathbf{I}\|\mathbf{S}_{\theta})=-\sum_{i}^{B}\log\mathbf{S}_{\theta ii}=\sum_{i}^{B}\|\widetilde{q}_{\theta}({\bf z}^{\prime}_{\theta})-{\bf z}^{\prime\prime}_{\xi}\|_{2}^{2} (81) which is the BYOL loss. B.7 Complexity Analysis for GCA In the forward pass, iteratively running the GCA does not involve inner optimization for gradient back-propagation. In the Sinkhorn algorithm, the transport plan {\bf P}_{\theta} is computed as: {\bf P}_{\theta}=\exp(f+g-{\bf C}_{\theta})/\epsilon, where f and g are dual variables iteratively updated in the Sinkhorn algorithm but do not involve gradients with respect to \theta . The Sinkhorn optimization primarily entails scaling the rows and columns of {\bf P} to satisfy the marginal constraints, which can be viewed as element-wise operations (scaling and exponentiation) on the cost matrix {\bf C}_{\theta} . Since {\bf P}_{\theta} is computed through the fixed-point iteration of f and g that depend only on the current values of {\bf C}_{\theta} , the gradient back-propagation process is simplified. Specifically, the gradient of the loss with respect to the cost matrix {\bf C}_{\theta} is the key part that needs to be differentiated, rather than through each iterative update of f and g . A typical workflow of these algorithms was shown in Figure 2 of [ 17 ] , the gradient flow primarily involves differentiating through {\bf C}_{\theta} , which is done only once, and not through each step of the Sinkhorn iterations. This approach reduces computational complexity and avoids the need for back-propagation through every iterative update within the Sinkhorn algorithm, which might otherwise be computationally expensive. Appendix C Proofs that GCA methods improve the alignment and uniformity C.1 Improving Alignment In this section, we are going to show the GCA methods minimize the difference between the target alignment plan with the coupling matrix on latent. The uniformity and alignment loss have been used to exam the quality of the representation in self-supervised learning, which is defined as the following [ 57 ] : Definition 9 (Alignment loss) . Given \pi as joint distribution of positive samples on the latent, ({\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i}) are the normalized positive pairs sampled from the joint distribution \pi with encoder parameterized by \theta , the alignment loss is: \mathcal{L}_{\text{align}}=\min_{\theta}~\mathbb{E}_{({\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta i})\sim\pi}\left[\|{\bf z}^{\prime}_{\theta i}-{\bf z}^{\prime\prime}_{\theta i}\|_{2}^{2}\right]=\min_{\theta}~\sum_{i}\operatorname{diag}({\bf C}_{ii}), (82) where {\bf C} is the cost matrix defined in Equation ( 24 ). We can alter the constraint sets of proximal operators to provide the better alignment plans, i.e. GCA-INCE changes the constraint sets by considering both row and column normalization in coupling matrix Rather than just the row normalization. Such change will not affect the alignment loss in forward pass, it will benefit the alignment loss in the backward pass through a tighter bound of empirical risk minimization with the identity matrix. C.1.1 Proof of the tighter bound of GCA in ERM In this section, we provide the evidence for using the converged coupling plan {\bf P}^{(\infty)}_{\theta} is better than the {\bf P}^{(1)}_{\theta} or {\bf P}^{(t)}_{\theta} in Equation ( 18 ) for the GCA-methods loss in table 1 . This loss function will correspond to different alignment loss on the latent. And here the ERM is the definition as we provided in Appendix B.4.2 . Lemma 8 . Denote f^{(t_{1})} and g^{(t_{2})} the two dual variables in their t_{1} and t_{2} iterations, respectively. Then the objective loss in Equation ( 10 ) could be written as \operatorname{KL({\bf I}\|{\bf P}_{\theta}})=\varepsilon^{-1}(\operatorname{diag}({\bf C})-(f^{(t_{1})}+g^{(t_{2})})) . Proof of the Lemma 8 : The above Lemma 8 be derived form Equation ( 23 ). Recall that {\bf u}=\exp{(f/\varepsilon)},\quad{\bf v}=\exp{(g/\varepsilon)},\quad{\bf K}=\exp{(-{\bf C}/\varepsilon)} \displaystyle\operatorname{KL({\bf I}\|{\bf P}_{\theta}}) \displaystyle=-\sum_{i}\log\left({\bf P}_{ii}\right)=-\sum_{i}(\log\operatorname{diag}({\bf u})_{ii}+\log{\bf K}_{ii}+\log\operatorname{diag}({\bf v})_{ii}) \displaystyle=-\varepsilon^{-1}\sum_{i}(f_{i}-C_{ii}+g_{i})=\varepsilon^{-1}(\operatorname{diag}(C)-(f+g))\qed Here we provide the proof of the Best Alignment in Theorem 5 : Proof of the Theorem 5 Based on the Lemma 8 , to show \operatorname{KL({\bf I}\|{\bf P}^{(\infty)}})\leq\operatorname{KL({\bf I}\|{\bf P}^{(1)}}) . We have to show: \varepsilon^{-1}(\operatorname{diag}(C)-(f^{(\infty)}+g^{(\infty)}))\leq\varepsilon^{-1}(\operatorname{diag}(C)-(f^{(1)}+g^{(1)})). Then give the Lemma 3 , we know the f^{(t_{1})} and g^{(t_{2})} increase and converge weakly to their upper bound. As the \operatorname{diag}({\bf C}) will be unchanged in each proximal operations, we know the objective function \text{KL}({\bf I}\|{\bf P}_{\theta}) have lower upper bound with f^{(t_{1})} and g^{(t_{2})} increase and finally converged. ∎ Based on the Lemma 8 , We have to show: \varepsilon^{-1}(\operatorname{diag}(C)-(f^{(t)}+g^{(t)}))\geq\varepsilon^{-1}(\operatorname{diag}(C)-(f^{(\infty)}+g^{(\infty)})). Then give the Lemma 3 , when t_{1}\rightarrow\infty , f^{(t_{1})} converge uniformly to a fixed point f^{(\infty)} with f^{(t_{1})}\leq f^{(\infty)} . So we know the f^{(t_{1})} and g^{(t_{2})} increase and converge weakly to their upper bound. As the \operatorname{diag}({\bf C}) will be unchanged in each proximal operations, we know the objective function finally converged to f^{(\infty)} and g^{(\infty)} (Similarly, we prove g^{(t_{1})}\leq g^{(\infty)} in Appendix B.1.3 ).∎ C.1.2 Proof of the Theorem 6 : To show: L_{\text{GCA-RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(t)})\leq L_{\text{RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(1)}) . we know that: L_{\text{GCA-RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(t)})=-\frac{\operatorname{diag}({\bf P}^{(t)})}{{\bf u}^{(t)}}+\frac{\lambda\mathbf{I}}{{\bf u}^{(t)}}, (83) L_{\text{RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(1)})=-\frac{\operatorname{diag}({\bf P}^{(1)})}{{\bf u}^{(1)}}+\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}}, (84) Given that the Lemma 3 , we know \{{\bf u}^{(t)}\} and \{{\bf v}^{(t)}\} are a monotonically increasing sequence where \displaystyle{\bf u}^{(1)}\leq{\bf u}^{(t)} \displaystyle\Rightarrow\frac{\lambda\mathbf{I}}{{\bf u}^{(1)}}\geq\frac{\lambda\mathbf{I}}{{\bf u}^{(t)}} (85) \displaystyle-\operatorname{diag}({\bf K}_{\theta}){\bf v}^{(0)}\geq-\operatorname{diag}({\bf K}_{\theta}){\bf v}^{(t)} \displaystyle\Rightarrow-\frac{\operatorname{diag}({\bf P}^{(1)})}{{\bf u}^{(1)}}\geq-\frac{\operatorname{diag}({\bf P}^{(t)})}{{\bf u}^{(t)}} (86) Combine the above two items, we have the equation like L_{\text{GCA-RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(t)})\leq L_{\text{RINCE}}^{\lambda,q=1,\varepsilon}({\bf P}_{\theta}^{(1)}) . C.2 GCA methods improve the uniformity and benefit downstream classification tasks In this section, we provide theoretical evidence that the GCA approaches could improve the performance of downstream task, i.e. classification tasks, by providing the maximum uniformity through solving the EOT, as Theorem ( 7 ) stated. Here, the uniformity loss is defined as [ 50 ] : Definition 10 (Uniformity loss) . Let {\bf z}^{\prime}_{\theta i}\sim\mu and {\bf z}^{\prime\prime}_{\theta j}\sim\nu in which \mu and \nu are two distributions on the representation space, we define the uniformity loss as the following: L_{\text{uniform}}=\log\mathbb{E}_{{\bf z}^{\prime}_{\theta i},{\bf z}^{\prime\prime}_{\theta j}\,\text{i.i.d.}\sim p_{\text{data}}}[e^{-\varepsilon\left\|{\bf z}^{\prime}_{\theta i}-{\bf z}^{\prime\prime}_{\theta j}\right\|^{2}_{2}}] (87) , in which p_{\text{data}}(\cdot) is the sample distribution over latent space \mathbb{R}^{n} . Here, p_{\text{data}}(\cdot) should be the marginal distribution of the samples. As the {\bf z}^{\prime}_{\theta i} and {\bf z}^{\prime\prime}_{\theta}j are normalized latent variables, we have the right items of the uniformity loss e^{-\varepsilon\|{\bf z}^{\prime}_{\theta i}-{\bf z}^{\prime\prime}_{\theta j}\|^{2}_{2}} is the same as the entropy-regularized kernel {\bf K}_{ij}=e^{-\varepsilon{\bf C}_{ij}} with cost matrix items {\bf C}_{ij}=\|{\bf z}^{\prime}_{\theta i}-{\bf z}^{\prime\prime}_{\theta j}\|^{2}_{2} . C.2.1 Proof of the Theorem 7 Here we are going to compare two different coupling plans, {\bf P}_{\theta}^{(1)} and {\bf P}_{\theta}^{(\infty)} , and show the converged plan {\bf P}_{\theta}^{(\infty)} will achieve higher the uniformity after the forward pass. The general logic is that we show the equivalence for the solving EOT with the minimizing the uniformity loss objective. Then we use the convergence of iterative Bregman projections to show it could achieve higher uniformity. Based on the Entropy regularized OT defined in Definition 8 , we have: W_{c,\varepsilon}(\mu,\nu)\coloneqq\min_{\pi\in\Pi(\mu,\nu)}\int_{X\times Y}c(x,y)d\pi(x,y)+\varepsilon H(\pi|\mu\otimes\nu) (88) in which the entropy could be defined as: H(\pi|\mu\otimes\nu)\coloneqq\int_{X\times Y}\left(\log\left(\frac{d\pi(x,y)}{d\mu(x)d\nu(y)}\right)-1\right)d\pi(x,y)+1, (89) is the relative entropy of the transport plan \pi with respect to the product measure \mu\otimes\nu . So the corresponding dual problem of this EOT one is shown in the following formula: \displaystyle W_{c,\varepsilon}(\mu,\nu) \displaystyle=\max_{f\in C(\mathcal{X}),g\in C(\mathcal{Y})}\int_{\mathcal{X}}f(x)\,d\mu(x)+\int_{\mathcal{Y}}g(y)\,d\nu(y) (90) \displaystyle~~~~~~~~~~~-\varepsilon\int_{\mathcal{X}\times\mathcal{Y}}e^{\frac{f(x)+g(y)-c(x,y)}{\varepsilon}}\,d\mu(x)d\nu(y)+\varepsilon (91) \displaystyle=\max_{f\in C(\mathcal{X}),g\in C(\mathcal{Y})}\mathbb{E}_{\mu\otimes\nu}\left[f(x)+g(y)-e^{\frac{f(x)+g(y)-c(x,y)}{\varepsilon}}\right]+\varepsilon (92) The \mu(x) and \nu(x) are defined as the uniformly distribution with Dirac delta function we have on the two latent supports \{{\bf z}^{\prime}_{\theta i}\}_{i=1}^{B} and \{{\bf z}^{\prime\prime}_{\theta i}\}_{i=1}^{B} , so the function f(x) and g(y) could be pull out of the expectation operators. Since the \|{\bf z}^{\prime}_{\theta i}-{\bf z}^{\prime\prime}_{\theta j}\|^{2}_{2} is the element in the cost matrix {\bf C}_{ij} , which is computed through the cost function c(x,y) . As the {\bf z}^{\prime}_{\theta i} and {\bf z}^{\prime\prime}_{\theta j} are drawn independently from the latent distribution, so the remaining item \mathbb{E}_{\mu\otimes\nu}[e^{\frac{-c(x,y)}{\epsilon}}] is equivalent to the uniformity loss. The the above integral could be turned into the sum of the elements in matrix of dual variables of f^{(t_{1})} and g^{(t_{1})} in each iteration. Meanwhile, based on the convergence provided in the Lemma 3 , When t_{1}\rightarrow\infty , f^{(t_{1})} converge uniformly to a fixed point f^{(\infty)} with f^{(t_{1})}\leq f^{(\infty)} , which would provided the maximum value of the dual formula in the f^{(\infty)} , which corresponding to the coupling plan the {\bf P}^{(\infty)} . ∎ C.2.2 GCA benefits the downstream supervised classification task Here, we further show how the minimizing the uniformity loss is equivalent to minimize the downstream supervised loss in classification tasks under several assumptions [ 16 ] . Giving a labeled dataset \mathcal{D}=\{(\bar{{\bf x}}_{i},\mathbf{y}_{i})\}\in\bar{\mathcal{X}}\times\mathcal{Y} where \mathcal{Y}=[1..M] with M classes, we consider a fixed, pre-trained encoder f_{\theta}\in\mathcal{F}:\mathcal{X}\to\mathcal{S} with its representation f_{\theta}({\mathcal{X}}) and the input space \mathcal{X} contains both positive and negative views of n original samples (\bar{{\bf x}}_{i})_{i\in[1..n]}\in\bar{\mathcal{X}} , sampled from the data distribution p(\bar{{\bf x}}) . For each positive views \bar{{\bf x}}^{\prime}_{i} in \mathcal{X} , we sample from \bar{{\bf x}}_{i} using {\bf x}_{i}^{\prime}\sim\mathcal{A}(\cdot|\bar{{\bf x}}_{i}) , \mathcal{A}(\cdot|\bar{{\bf x}}_{i}) is augmentation distribution (e.g., by applying color jittering, flip, or crop with a given probability). For consistency, we assume \mathcal{A}(\bar{{\bf x}})=p(\bar{{\bf x}}) so that the distributions \mathcal{A}(\cdot|\bar{{\bf x}}) and p(\bar{{\bf x}}) induce a marginal distribution p({\bf x}) over \mathcal{X} . Given an anchor \bar{{\bf x}}_{i} , all views {\bf x}^{\prime\prime}\sim\mathcal{A}(\cdot|\bar{{\bf x}}_{j}),j\neq i from different samples \bar{{\bf x}}_{j} are considered as negatives. Proof of claim 1 : From assumption 1 we know that the representation ability of encoders is good enough via the augmented samples in the Reproducing Kernel Hilbert Space (RKHS) \mathcal{H}_{\bar{\mathcal{X}}} of the original sample spaces \bar{\mathcal{X}} . And the kernel K_{\bar{\mathcal{X}}} with any function g RKHS defined by (\mathcal{H}_{f_{\theta}},{\bf K}_{\theta}) also belongs to H_{\bar{\mathcal{X}}} when conditioned on the distribution \mathcal{A}({\bf x}|\cdot) . So based on the assumption we have, we can obtain a centroid estimator by [ 16 ] : Definition 11 (Kernel-based centroid estimator) . Let ({\bf x}_{i},\bar{{\bf x}}_{i})_{i\in[1..n]}\sim\mathcal{A}({\bf x},\bar{{\bf x}}) , asssuming a consistent estimator of \mu_{\bar{{\bf x}}} is. \forall\bar{{\bf x}}\in\bar{\mathcal{X}},\hat{\mu}_{\bar{{\bf x}}}=\sum_{i=1}^{n}\alpha_{i}(\bar{{\bf x}})f({\bf x}_{i}), where \alpha_{i}(\bar{{\bf x}})=\sum_{j=1}^{n}[({\bf K}_{n}+n\lambda\mathbf{I}_{n})^{-1}]_{ij}{\bf K}_{\bar{\mathcal{X}}}(\bar{{\bf x}}_{j},\bar{{\bf x}}) and {\bf K}_{n}=[{\bf K}_{\bar{\mathcal{X}}}(\bar{{\bf x}}_{i},\bar{{\bf x}}_{j})]_{i,j\in[1..n]} . It converges to \mu_{\bar{{\bf x}}} with the \ell_{2} norm at a rate \mathcal{O}(n^{-1/4}) for \lambda=\mathcal{O}(n^{-1/2}) . The above estimator allows us to use representations of images close to an anchor \bar{{\bf x}} to estimate \mu_{\bar{{\bf x}}} . From the assumption 2 , we assume that all the samples in the same class is achievable when give the ideal augmentation or at least close to the augmented points in an \epsilon region. Consequently, if the prior is “good enough” to connect intra-class images disconnected in the augmentation graph suggested by Assumption 1 , then this estimator allows us to tightly control the classification risk of the representation of f_{\theta} on a classification task with a linear classifier g(\bar{{\bf x}})={\cal{W}}f_{\theta}(\bar{{\bf x}}) (with f_{\theta} fixed) that minimizes the multi-class classification loss. First we show the cross-entropy could be transformed into centroid based distance (optimal supervised loss): The cross-entropy (CE) to measure the difference between the true distribution (actual labels) and the estimated probability distribution (predicted probabilities from the model), which usually computes logits {\bf z}_{k} from the model, then apply the softmax function to obtain probabilities p_{k} . The logits {\bf z}_{k} could be defined as negative distances between f(\bar{{\bf x}}) and class centroids \mu_{k} after the representation: {\bf z}_{k}=-\|f(\bar{{\bf x}})-\mu_{k}\|^{2},\quad\mu_{k}=\mathbb{E}_{p(\bar{{\bf x}}|\mathbf{y}=k)}\mu_{\bar{{\bf x}}} which encourages the model to reduce the distance to the correct class centroid while increasing distances to others. The probability of class k in M classes given input \bar{{\bf x}} is: p(\mathbf{y}=k|\bar{{\bf x}})=\frac{e^{{\bf z}_{k}}}{\sum_{j=1}^{M}e^{{\bf z}_{j}}},\quad p(\mathbf{y}|\bar{{\bf x}})\propto e^{-\|f(\bar{{\bf x}})-\mu_{\mathbf{y}}\|^{2}}. If the model predictions p(\mathbf{y}|\bar{{\bf x}}) are influenced by the distances between \bar{{\bf x}} and the class centroids \mu_{\mathbf{y}} , then minimizing cross-entropy indirectly affects these distances. The standard CE loss in supervised learning for classification tasks is: \displaystyle\mathcal{L}_{\text{CE}}(f_{\theta}) \displaystyle=-\mathbb{E}_{(\bar{{\bf x}},\mathbf{y})\sim\mathcal{D}}\left[\log p(\mathbf{y}|\bar{{\bf x}})\right] (93) \displaystyle=-\mathbb{E}_{(\bar{{\bf x}},\mathbf{y})\sim\mathcal{D}}\left[-\|f(\bar{{\bf x}})-\mu_{\mathbf{y}}\|^{2}-\log Z\right]=-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{M}\mathbf{y}_{i,k}\log(p_{i,k}) (94) which focuses on maximizing the likelihood \hat{\mathbf{y}}=\arg\max_{k}p(\mathbf{y}=k|\bar{{\bf x}}) of the correct class for each individual sample \bar{{\bf x}}_{i} , where y_{i,k} is the true label indicator for example i and class k , p_{i,k} is the predicted probability for example i and class k . Therefore, we can rewrite the CE loss as optimal supervised loss in [ 16 ] , which is defined as: Lemma 9 (Optimal supervised loss) . Let a downstream task D with M classes. We assume that M\leq d+1 (i.e., a big enough representation space), that all classes are balanced and the realizability of an encoder f^{*}=\arg\min_{f\in F}\mathcal{L}_{\text{sup}}(f_{\theta}) with \mathcal{L}_{\text{sup}}(f_{\theta})=\log\mathbb{E}_{y,y^{\prime}\sim p(y)p(y^{\prime})}\left[e^{-\|\mu_{y}-\mu_{y^{\prime}}\|^{2}}\right], and \mu_{y}=\mathbb{E}_{p(\bar{x}|y)}\mu_{\bar{x}} . Then the optimal centroids (\mu^{*}_{y})_{y\in Y} associated to f^{*} make a regular simplex on the hypersphere S^{d-1} and they are perfectly linearly separable, i.e., \min_{(w_{y})_{y\in\mathcal{Y}}\in\mathbb{R}^{d}}\mathbb{E}_{(\bar{x},y)\sim D}\mathbbm{1}(w_{y}\cdot\mu^{*}_{y}<0)=0. Proof of the Lemma 9 All "labeled" centroids \mu_{y}=\mathbb{E}_{p(\bar{x}|y)}\mu_{\bar{x}} are bounded by 1 ( \|\mu_{y}\|\leq\mathbb{E}_{p(\bar{x}|y)}\mathbb{E}_{A(x|x^{\prime})}\|f(x)\|=1 by Jensen’s inequality). Then, since all classes are balanced, we can re-write the supervised loss as: \mathcal{L}_{\text{sup}}(f_{\theta})=\log\frac{1}{C^{2}}\sum_{y,y^{\prime}=1}^{C}e^{-\|\mu_{y}-\mu_{y^{\prime}}\|^{2}}. We have: \Gamma_{\mathcal{Y}}(\mu):=\sum_{y,y^{\prime}}\|\mu_{y}-\mu_{y^{\prime}}\|^{2}=\sum_{y,y^{\prime}}\|\mu_{y}\|^{2}+\|\mu_{y^{\prime}}\|^{2}-2\mu_{y}\cdot\mu_{y^{\prime}}\leq\sum_{y,y^{\prime}}(2-2\mu_{y}\cdot\mu_{y^{\prime}})=2C^{2}-2\|\sum_{y}\mu_{y}\|^{2}\leq 2C^{2}, with equality if and only if \sum_{y=1}^{C}\mu_{y}=0 and \forall y\in[1..C],\|\mu_{y}\|=1 . By the strict convexity of u\to e^{-u} , we have: \sum_{y\neq y^{\prime}}\exp(-\|\mu_{y}-\mu_{y^{\prime}}\|^{2})\geq C(C-1)\exp\left(-\frac{\Gamma_{Y}(\mu)}{C(C-1)}\right)\geq C(C-1)\exp\left(-\frac{2C}{C-1}\right), with equality if and only if all pairwise distances \|\mu_{y}-\mu_{y^{\prime}}\| are equal (equality case in Jensen’s inequality for a strict convex function), \sum_{y=1}^{C}\mu_{y}=0 , and \|\mu_{y}\|=1 . Thus, all centroids must form a regular (C-1) -simplex inscribed on the hypersphere S^{d-1} centered at 0. Furthermore, since \|\mu_{y}\|=1 , we have equality in Jensen’s inequality: \|\mu_{y}\|=\|\mathbb{E}_{\mathcal{A}({\bf x}|\bar{{\bf x}}^{\prime})}f_{\theta}(x)\|\leq\mathbb{E}_{\mathcal{A}({\bf x}|\bar{{\bf x}}^{\prime})}\|f_{\theta}(x)\|=1, so f must be perfectly aligned for all samples belonging to the same class: \forall x,\bar{x}^{\prime}\sim p(\cdot|y),f_{\theta}(\bar{x})=f_{\theta}(\bar{x}^{\prime}) . ∎ Seond we show optimizing the uniformity loss is equivalent to the supervised loss: As we have uniformity Loss defined in Equation ( 87 ) L_{\text{uniform}}(f_{\theta})=\log\mathbb{E}_{{\bf z}^{\prime}_{i},{\bf z}^{\prime\prime}_{j}\sim p_{\text{data}}}\left[e^{-\varepsilon\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}}\right], (95) where {\bf z}^{\prime}_{i}=f({\bf x}_{i}) and {\bf z}^{\prime\prime}_{j}=f({\bf x}_{j}) . Supervised Loss: \mathcal{L}_{\text{sup}}(f_{\theta})=\log\mathbb{E}_{y,y^{\prime}\sim p(y)p(y^{\prime})}\left[e^{-\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2}}\right], where \mu_{y}=\mathbb{E}_{p(\bar{{\bf x}}|y)}\hat{\mu}_{\bar{{\bf x}}} . Express the expectation over all pairs in terms of class labels: \mathbb{E}_{{\bf z}^{\prime}_{i},{\bf z}^{\prime\prime}_{j}}=\mathbb{E}_{y,y^{\prime}}\mathbb{E}_{{\bf z}^{\prime}_{i}\sim p({\bf z}|y),{\bf z}^{\prime\prime}_{j}\sim p({\bf z}|y^{\prime})}. So the uniformity loss could be decomposed into intra-class and inter-class components: L_{\text{uniform}}(f_{\theta})=\log\left(\underbrace{\mathbb{E}_{y}\left[\mathbb{E}_{{\bf z}^{\prime}_{i},{\bf z}^{\prime\prime}_{j}\sim p({\bf z}|y)}\left[e^{-\varepsilon\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}}\right]\right]}_{\text{Intra-Class Term}}+\underbrace{\mathbb{E}_{y\neq y^{\prime}}\left[\mathbb{E}_{{\bf z}^{\prime}_{i}\sim p({\bf z}|y),{\bf z}^{\prime\prime}_{j}\sim p({\bf z}|y^{\prime})}\left[e^{-\varepsilon\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}}\right]\right]}_{\text{Inter-Class Term}}\right). Based on the assumption 2 , we can approximate the Intra-Class term by: \displaystyle\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2} \displaystyle=\left\|(\mu_{y}+\delta_{i})-(\mu_{y^{\prime}}+\delta_{j})\right\|^{2}=\left\|\mu_{y}-\mu_{y^{\prime}}+\delta_{i}-\delta_{j}\right\|^{2}\approx\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2} \displaystyle\implies\mathbb{E}_{{\bf z}^{\prime}_{i}\sim p({\bf z}|y),{\bf z}^{\prime\prime}_{j}\sim p({\bf z}|y^{\prime})}\left[e^{-\varepsilon\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}}\right]\approx e^{-\varepsilon\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2}} for y=y^{\prime} , {\bf z}^{\prime}_{i} and {\bf z}^{\prime\prime}_{j} are close to \mu_{y} \left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}\approx\left\|(\mu_{y}+\delta_{i})-(\mu_{y}+\delta_{j})\right\|^{2}=\left\|\delta_{i}-\delta_{j}\right\|^{2}. Since \delta_{i} and \delta_{j} are small deviations: \mathbb{E}_{{\bf z}^{\prime}_{i},{\bf z}^{\prime\prime}_{j}\sim p({\bf z}|y)}\left[e^{-\varepsilon\left\|{\bf z}^{\prime}_{i}-{\bf z}^{\prime\prime}_{j}\right\|^{2}}\right]\approx 1,\quad e^{-\varepsilon\left\|\delta_{i}-\delta_{j}\right\|^{2}}\approx 1. Then with M terms for y=y^{\prime} and M(M-1) terms of y\neq y^{\prime} , we have: L_{\text{uniform}}=\log(\frac{1}{M}e^{-\varepsilon\left\|\delta_{i}-\delta_{j}\right\|^{2}}+\frac{1}{M^{2}}\sum_{y\neq y^{\prime}}e^{-\varepsilon\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2}}) (96) The supervised loss is: \mathcal{L}_{\text{sup}}(f_{\theta})=\log(\frac{1}{M^{2}}\sum_{y,y^{\prime}}e^{-\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2}})=\log(\frac{1}{M}e^{-\left\|\mu_{y}-\mu_{y}\right\|^{2}}+\frac{1}{M^{2}}\sum_{y\neq y^{\prime}}e^{-\left\|\mu_{y}-\mu_{y^{\prime}}\right\|^{2}}) Since e^{-\left\|\mu_{y}-\mu_{y}\right\|^{2}}=1 (for y=y^{\prime} ), the difference will be mainly dependent on the inter-class term. Therefore, a tighter (smaller) uniformity loss leads to smaller values of the supervised loss. This supports the idea that improving uniformity in representations can benefit downstream supervised classification tasks. ∎ C.3 Unbalanced OT assists to alleviate the feature suppression Although minimizing the uniformity loss can enhance downstream classification tasks, it may also lead the model to learn shortcut features that could impair the encoder’s generalization ability. To show this, we incorporate two propositions from previous work by Robinson et al. [ 48 ] . C.3.1 The uniformity loss causes feature suppression For an encoder f_{\theta}:\mathcal{X}\rightarrow\mathbb{S}^{d-1} to map input data {\bf x} to the surface of the unit sphere \mathbb{S}^{d-1}=\{u\in\mathbb{R}^{d}:||u||_{2}=1\} . Suppose we have the latent feature spaces \mathbf{Z}^{1},\dots,\mathbf{Z}^{n} with a distribution p_{j} on each latent space \mathbf{Z}^{j} with j\in[n] to model a distinct feature. We write \mathbf{Z} instead of \mathbf{Z}^{[n]} for the product as \mathbf{Z}^{S}=\prod_{j\in S}\mathbf{Z}^{j} , where [n]=\{1,\dots,n\} . So the latent sample {\bf z} could be represented as a set of feature vectors {\bf z}=({\bf z}^{1},{\bf z}^{2},\dots,{\bf z}^{n})=({\bf z}^{j})_{j\in S}\in\mathbf{Z} , where each {\bf z}^{j} comes from \mathbf{Z}^{j} . Further, let \lambda denote the measure on \mathbf{Z} induced by {\bf z} and \lambda(\cdot|{\bf z}^{S}) denote the conditional measure on \mathbf{Z} for fixed {\bf z}^{S} . For S\subseteq[n] we use {\bf z}^{S} to denote the projection of {\bf z} onto Z^{S} . Finally, an injective map g:\mathbf{Z}\rightarrow\mathcal{X} produces observations {\bf x}=g({\bf z}) . The feature suppression is defined as: Definition 12 . Consider an encoder f_{\theta}:\mathcal{X}\rightarrow\,\mathbb{S}^{d-1} and features S\subseteq[n] . For each {\bf z}^{S}\in Z^{S} , let \mu(\cdot|{\bf z}^{S}) be the pushforward measure on S^{d-1} by f_{\theta}\circ g of the conditional \lambda(\cdot|{\bf z}^{S}) . 1. f_{\theta} suppresses S if for any pair {\bf z}^{S},\tilde{{\bf z}}^{S}\in Z^{S} , we have \mu(\cdot|{\bf z}^{S})=\mu(\cdot|\tilde{{\bf z}}^{S}) . 2. f_{\theta} distinguishes S if for any pair of distinct {\bf z}^{S},\tilde{{\bf z}}^{S}\in\mathbf{Z}^{S} , measures \mu(\cdot|{\bf z}^{S}),\mu(\cdot|\tilde{{\bf z}}^{S}) have disjoint support. If one feature is uniformly distributed on the latent space, it might cause feature suppression due to different features could both achieve the minimization of the uniformity loss as the following propositions [ 48 ] : Proposition 5 (Feature suppression) . For a set S\subseteq[n] of features let L_{S}(f_{\theta})=L_{\text{align}}(f_{\theta})+\mathbb{E}_{{\bf x}^{+}}\left[-\log\mathbb{E}_{\bf x}\left[e^{f({\bf x}^{+})^{\top}f({\bf x}^{-})}\middle|{\bf z}^{S}={\bf z}^{S^{-}}\right]\right] denote the (limiting) InfoNCE conditioned on {\bf x}^{+},{\bf x}^{-} having the same features S . Suppose that p_{j} is uniform on Z^{j}=S^{d-1} for all j\in[n] . Then the infimum \inf L_{S} is attained, and every f_{\theta}\in\arg\min_{f}L_{S}(f_{\theta}^{\prime}) suppresses features S almost surely. Proof of proposition 5 is in [ 48 ] . C.3.2 How the GCA methods and unbalanced OT and alleviates the feature suppression Here we extended the unbalanced OT in the Equation ( 9 ) as the following: \min_{\theta}~d_{M}({\bf P}_{\text{t}gt}\|{\bf P}_{\theta})+\lambda_{1}d_{\phi_{1}}({\bf P}_{\theta})+\lambda_{2}d_{\phi_{2}}({\bf P}_{\theta})+\cdots+\lambda_{n}d_{\phi_{n}}({\bf P}_{\theta}) (97) The UOT equation can be converted with finding the transport plan {\bf P}_{\theta} that minimizes the transportation cost between two probability measures \mu and \nu . Here we only need to show that the relaxation or adding penalties will change the optimal transport plan {\bf P}_{\theta} , which is empirically exhibited in the Figure A4 . Suppose we have empirical samples \{{\bf z}^{\prime}_{i}\}_{i=1}^{n} from \mu and \{{\bf z}^{\prime\prime}_{j}\}_{j=1}^{m} from \nu . We can approximate the measures using empirical distributions: \mu\approx\frac{1}{n}\sum_{i=1}^{n}\delta_{{\bf z}^{\prime}_{i}},\quad\nu\approx\frac{1}{m}\sum_{j=1}^{m}\delta_{{\bf z}^{\prime\prime}_{j}}, where \delta_{{\bf z}} is the Dirac delta function at point {\bf z} . The standard UOT objective can be written as: \displaystyle\min_{{\bf P}\geq 0}\sum_{i=1}^{n}\sum_{j=1}^{m}{\bf C}({\bf z}^{\prime}_{i},{\bf z}^{\prime\prime}_{j}){\bf P}_{ij}+\lambda_{1}d_{\phi_{1}}\left(\sum_{j=1}^{m}{\bf P}_{ij}\Big\|\frac{1}{n}\right)+\lambda_{2}d_{\phi_{2}}\left(\sum_{i=1}^{n}{\bf P}_{ij}\Big\|\frac{1}{m}\right) (98) \displaystyle=\min_{{\bf P}\geq 0}\sum_{i=1}^{n}\sum_{j=1}^{m}\left[{\bf C}_{ij}{\bf P}_{ij}+\lambda_{1}{\bf P}_{ij}\left(\log\frac{{\bf P}_{ij}}{r_{i}}-1\right)+\lambda_{2}{\bf P}_{ij}\left(\log\frac{{\bf P}_{ij}}{c_{j}}-1\right)\right] (99) where {\bf C} is the cost matrix d_{\phi} could be any divergence (e.g., Kullback-Leibler divergence) with respect to a convex function \phi . {\bf P}\mathbf{1}_{\mu} and {\bf P}^{\top}\mathbf{1}_{\nu} are the marginal distributions. \lambda_{1},\lambda_{2} are regularization parameters controlling the unbalancedness and r_{i}=\frac{1}{n} (source marginal mass for {\bf z}^{\prime}_{i} ), c_{j}=\frac{1}{m} (target marginal mass for {\bf z}^{\prime\prime}_{j} ). Based on the UOT, here we can choose the divergence as \mathcal{L} : \mathcal{L}(\mathbf{P})=\sum_{i,j}\left[C_{ij}\mathbf{P}_{ij}+\lambda_{1}\mathbf{P}_{ij}\left(\log\frac{\mathbf{P}_{ij}}{r_{i}}-1\right)+\lambda_{2}\mathbf{P}_{ij}\left(\log\frac{\mathbf{P}_{ij}}{c_{j}}-1\right)\right] To find the minimizer, we take the partial derivative of L(\mathbf{P}) with respect to \mathbf{P}_{ij} and set it to zero: \displaystyle\frac{\partial\mathcal{L}}{\partial\mathbf{P}_{ij}}=C_{ij}+\lambda_{1}\left(\log\frac{\mathbf{P}_{ij}}{r_{i}}\right)+\lambda_{2}\left(\log\frac{\mathbf{P}_{ij}}{c_{j}}\right)=0 (100) \displaystyle\implies\lambda_{1}\left(\log\mathbf{P}_{ij}-\log r_{i}\right)+\lambda_{2}\left(\log\mathbf{P}_{ij}-\log c_{j}\right)=-C_{ij} (101) \displaystyle\implies(\lambda_{1}+\lambda_{2})\log\mathbf{P}_{ij}-\lambda_{1}\log r_{i}-\lambda_{2}\log c_{j}=-C_{ij} (102) \displaystyle\implies\log\mathbf{P}_{ij}=\frac{-C_{ij}+\lambda_{1}\log r_{i}+\lambda_{2}\log c_{j}}{\lambda_{1}+\lambda_{2}} (103) \displaystyle\implies\mathbf{P}_{ij}=\exp\left(\frac{-C_{ij}+\lambda_{1}\log r_{i}+\lambda_{2}\log c_{j}}{\lambda_{1}+\lambda_{2}}\right) (104) The minimizer \mathbf{P}_{ij} depends on \lambda_{1} and \lambda_{2} and the weights of r_{i} and c_{j} , which determine the influence of the marginals r_{i} and c_{j} , and through the scaling of the cost C_{ij} by \lambda_{1}+\lambda_{2} . This explicit relationship shows how \lambda_{1} and \lambda_{2} determine the minimizer. Appendix D Details of Experiments The following experiments involving with the GPU was set up on NVIDIA GeForce RTX 3090. D.1 Experimental details on image classification task In Table 2 standard settings, we used two different experimental setups. The first setup, referred to as the C0 or standard settings, was applied specifically to the CIFAR10 and CIFAR100 tasks. The second setup was used for the SVHN and ImageNet100 tasks, respectively. Below, we present the settings for CIFAR10 and CIFAR100, followed by the setups for SVHN and ImageNet100. Here is the setups for CIFAR10 and CIFAR100: • The SSL model has 512 feature dimensions with the base model (ResNet-18), which first convolutional changed as a layer with 3 input channels, 64 output channels, kernel size 3, stride 1, padding 1, and no bias. We replace the max-pooling layer as the identity. • A sequential projector comprising a linear layer mapping from feature dimension to 2048, ReLU activation, and another linear layer mapping from 2048 to 128. • For SSL training, an SGD optimizer is used with a learning rate of 0.6, momentum 0.9, and a weight decay of 1.0e-6. A LambdaLR scheduler is employed with linearly decay the learning rate to 1.0e-3 over total steps, which equals the length of the SSL training loader times the maximum epochs. The SSL model is trained for a maximum of 500 epochs, without loading a pre-trained model. The parameters of encoders are frozen after training. Temperature or epsilon: 0.5. • For supervised training, an Adam optimizer is also used with a learning rate of 0.2, momentum 0.9 and a weight decay of 0. A same LambdaLR scheduler is applied, where the learning rate is reduced by a factor of 1.0e-3. For supervised training, the model is trained for a maximum of 200 epochs using the specified train and test loaders. The setups for SVHN and ImageNet100 are: • The SSL model has number of feature dimensions equal to the fc layer incoming features of base model (ResNet-50). We replace the max-pooling layer as the identity. • A sequential projector comprising a linear layer mapping from feature dimension to 2048, ReLU activation, and another linear layer mapping from 2048 to 128. • For SSL training, an Adam optimizer is used with a learning rate of 3e-4. The SSL model is trained for a maximum of 200 epochs for ImageNet100 and 500 epochs for the SVHN, without loading a pre-trained model. The parameters of encoders are frozen after training. Temperature or epsilon: 0.5. • For supervised training, an Adam optimizer is also used with a learning rate of 3e-4. The model is trained for a maximum of 100 epochs using the specified train and test loaders. D.2 Settings for extreme data augmentations There is the "extreme DA" (Ex DA) column in Table 2 , which is the average of the following three settings: • C1: Large Erase Settings: Here, we first employed the same standard augmentation as C0 in Appendix D.1 does, than we apply the random erase with ’p=1’ (random erasing is applied every time), the ’scale=(0.10, 0.33)’. The large erase is applied before the normalization. • C2: Strong Crop Setting: This involves a strong cropping operation followed by resizing, which applied by ’transforms.RandomCrop’ and ’transforms.Resize’. The crop size varies based on the severity level, with values ranging from 96 to 224 pixels. We selected level 3 during our experiments, than Resizes the cropped image back to 32x32 pixels. • C3: Brightness settings: This augmentation alters the brightness of the images. We have ’severity’ determines the degree of brightness change, with predefined levels ranging from ‘.05‘ to ‘.3‘, corresponding to level 1 and level 5. And we chosse the level 5 as our C3 augmentation. The brightness is adjusted in the HSV color space, specifically altering the value channel to change the brightness. To evaluate performance on CIFAR10-C, we use a pretrained SSL model with frozen parameters. Fine-tuning is performed by training only the linear layer with 10\% of CIFAR10-C data for 50 epochs. We compute the final score by averaging results across all corruption types and severity levels in CIFAR10-C. And the details of each column are provided in Table A2 , Table A3 and Table A4 . Table A2: Test accuracy for contrastive methods on CIFAR-10. Test accuracy for different contrastive methods and their GCA equivalents on CIFAR-10 for ResNet-18 under extreme augmentation conditions, averaged over 5 seeds. Conditions INCE GCA-INCE RINCE GCA-RINCE SimCLR BYOL IOT IOT-uni GCA-UOT Standard 92.01 ± 0.40 93.02 ± 0.19 93.27±0.20 93.47±0.32 92.16 ±0.16 90.56 ± 0.59 92.10 ± 0.22 91.49 ± 0.11 93.50±0.31 Erase 88.40 ± 0.17 88.16 ± 0.89 88.80±1.01 89.21±0.59 88.44 ± 0.24 88.77 ± 0.58 87.02 ± 0.43 87.83 ± 0.30 89.84 ± 0.58 Crop 72.45 ± 0.40 72.79 ± 0.62 73.02±0.39 73.10±0.31 71.84 ± 1.02 70.78 ± 0.62 70.44 ± 0.64 70.78 ± 0.21 73.35 ± 0.41 Brightness 85.24 ± 0.41 85.60 ± 0.57 85.97±0.50 85.98 ± 0.58 85.32 ± 0.32 85.10 ± 0.29 84.31 ± 0.84 83.77 ± 0.21 86.36±0.34 Table A3: Test accuracy for contrastive methods on CIFAR-100. Test accuracy for different contrastive methods and their GCA equivalents on CIFAR-100 using ResNet-18 under extreme augmentation conditions, averaged over 5 seeds. Conditions INCE GCA-INCE RINCE GCA-RINCE SimCLR BYOL IOT IOT-uni GCA-UOT Standard 71.09 ± 0.31 71.55 ± 0.12 71.63 ± 0.36 71.95 ± 0.48 70.85 ± 0.50 69.75 ± 0.37 68.37 ± 0.42 68.62 ± 0.35 72.16 ± 0.38 Large Erase 62.54 ± 0.20 62.65 ± 0.17 63.55 ± 0.14 63.14 ± 0.41 62.94 ± 0.13 62.70 ± 0.31 62.69 ± 0.34 62.56 ± 0.22 63.62 ± 0.27 Strong Crop 45.67 ± 0.31 46.31 ± 0.22 46.47 ± 0.20 46.50 ± 0.35 46.05 ± 0.34 43.11 ± 0.41 45.36 ± 0.19 45.29 ± 0.12 46.60 ± 0.34 Brightness 59.87 ± 0.36 59.68 ± 0.33 60.38 ± 0.25 60.56 ± 0.25 60.46 ± 0.19 55.74 ± 0.63 57.52 ± 0.32 57.42 ± 0.18 61.02 ± 0.55 Table A4: Test accuracy for contrastive methods on CIFAR-10C. Test accuracy for different contrastive methods and their GCA equivalents on CIFAR-10C for ResNet-18 under extreme augmentation conditions, averaged over 5 seeds. Conditions INCE GCA-INCE RINCE GCA-RINCE SimCLR BYOL IOT IOT-uni GCA-UOT Standard 81.52 ± 1.04 82.63 ± 0.28 82.86 ± 0.21 82.87 ± 0.11 81.74 ± 1.54 82.43 ± 0.06 82.01 ± 0.80 81.18 ± 1.12 82.90 ± 0.49 Large Erase 82.67 ± 0.31 82.19 ± 0.35 83.11 ± 0.36 83.44 ± 0.55 82.49 ± 0.24 82.53 ± 0.34 82.30 ± 0.69 82.30 ± 0.48 83.09 ± 0.62 Strong Crop 43.55 ± 0.90 41.12 ± 3.28 42.51 ± 1.48 44.96 ± 1.46 42.74 ± 1.27 43.86 ± 0.91 42.36 ± 0.87 42.43 ± 1.03 45.20 ± 1.20 Brightness 84.00 ± 0.47 84.08 ± 0.26 84.24 ± 0.36 84.54 ± 0.26 83.86 ± 0.46 83.87 ± 0.19 82.33 ± 0.31 80.86 ± 0.72 84.88 ± 0.44 D.3 Experimental setting for domain generalization This section is going to show the settings of experiments in Figure 1 , which involves the domain generalization task. Training was executed under the DomainBed framework. Each model underwent training across multiple domains, with 5 distinct seeds (seed 71, 68, 42, 36, 15) used to ensure reproducibility: • For SSL model configuration, we employed a ResNet-18 architecture as the encoder, following with a 2048-dimensional, 3-layer projector equipped with BatchNorm1D and ReLU activations. We improved the framework of the SelfReg algorithm in Domainbed [ 23 ] by a self-supervised contrastive learning phase which involves the GCA-INCE, with regularized parameters \varepsilon=0.2 . • For SSL training hyperparameters, an Adam optimizer is used with a learning rate of 3e-4, and a weight decay of 1.5e-6. A Cosine Annealing learning rate scheduler is employed with a maximum number of 200 iterations equal to the length of the SSL training. The learning rate is scheduled to decrease to a minimum value of 0. The SSL model is trained for a maximum of 1500 epochs. • In the self-supervised learning phase, we utilized 20% of the data from each of the four datasets in the PACS dataset. The unsupervised holdout part employed contrastive learning augmentations to enhance generalization capabilities. Specifically, we implemented dual augmentation, including operations such as random resized crops, flips, color jitter, and grayscale conversion, standardized to an input shape of 3\times 224\times 224 . • The supervised learning rate was set at 5\times 10^{-5} using MSE loss, and the Adam optimizer with no weight decay. Training involved both domain and class labels over 3000 epochs, with checkpoints every 300 epochs to capture the model’s best performance. This approach was supplemented by fine-tuning the model post-unsupervised training phase. Domain labels were categorized into four types corresponding to the PACS dataset, and class labels were divided into five categories. In domain classification, all four domains are used for training, with 70% of the data held out for training and the remaining 30% used for testing. Four domains are utilized for class classification tasks. We train supervised models on three domains and test on the fourth. • The domain accuracy is computed as the average of the highest domain accuracies across five seeds, with each of the four test domains set sequentially as the test domain. The standard deviation for domain accuracy is calculated from the results across these five seeds. • Class label accuracy is determined by averaging the accuracies of the four test environments for each domain. The average of highest performance across the domain is taken as the mean accuracy. The standard deviation for each domain is computed from the five seeds, and these values are then averaged to obtain the final class standard deviation. Both the label classification tasks and the domain classification tasks use the Mean Squared Error (MSE) loss. Appendix E Additional Experiments E.1 Complexity Analysis of GCA Algorithms Time complexity analysis: The computational complexity of GCA including the forward pass and backward propagation phases. The complexity varies in different variants. For GCA-INCE, the computational complexity of forward pass is related to the speed of Sinkhorn when solving the EOT problem as O(n^{2}/\varepsilon^{3}) , in which \varepsilon is the regularization parameter . For GCA-UOT, the forward complexity is the Sinkhorn algorithm solving unbalanced OT, which is characterized by O(\tau(\alpha+\beta)^{2}/\varepsilon\log(n)[\log(\|C\|_{\infty})+\log(\log(n))+\log(1/\varepsilon)]), where C is the cost matrix, \alpha and \beta denote the total masses of the measures, and \tau is a regularization parameter related to KL divergences in the UOT framework [ 44 ] . Notably, the gradient backpropagation speed is not seriously affected by scaling operations in the EOT as we explained in Section B.7 . Moreover, the relaxations of penalties in UOT provide a even faster speed compared with the INCE and GCA-INCE (see Figure A2 ). Figure A2: Time complexity analysis (A) Time complexity analysis of different methods. Here, we provide the time complexity for different contrastive methods (INCE, RINCE) and GCA-based methods (GCA-INCE, GCA-RINCE, and GCA-UOT) on CIFAR-10. (B) Time complexity for INCE (GCA-INCE-1), and GCA-INCE with different number of iterations GCA-INCE-100 denotes GCA-INCE with 100 iterations. We ran the methods on the CIFAR-10 as self-supervised learning task for 50 epochs, and compared their run time. (C) Performance of the INCE (iteration=1) and GCA-INCE (iterations>1) on the CIFAR10 with different number of iterations. The shaded blue region is the standard deviation across 5 seeds. The complexity of the forward pass is affected by the choice of proximal operator, whereas the complexity of the gradient backward pass is influenced by the form of d_{M} [ 37 ] . Notably, utilizing Sinkhorn algorithms in GCA-UOT, GCA-RINCE, and GCA-INCE, only requires updating the coupling matrix {\bf P} (B\times B) without impacting the complexity of the backward pass, where B is the batch size. OT is known to have B^{2} complexity and in many cases can converge very quickly in fewer than 10 iterations. In practice, we use a simple stopping criterion for the multiple iterations using a convergence criterion. Upon analyzing the run time for the different methods (see Figure A2 ) we observe that the GCA-based variants of the different base approaches (INCE, or RINCE) achieve very similar run time as their equivalent loss, but different losses (RINCE vs INCE) exhibit more significant variability. Specifically, we find that RINCE and GCA-RINCE have lower time complexity than INCE and GCA-INCE. So the runnning speed is even quicker if we utillized different d_{M} in Equation ( 8 ). E.2 Measuring the representation quality using alignment and uniformity We study the uniformity and alignment of the representations learned by our GCA-INCE vs. INCE variants of GCA in Algorithms 1 .We train the model through the corresponding settings (C0: standard provided in the , C1: erase, C2: crop, C3: brightness) provided in the Appendix D.1 and Appendix D.2 . We find that in general, the GCA variants improve the representation quality evaluated by alignment and uniformity on both CIFAR-10 and CIFAR-10C datasets. Figure A3: Alignment and uniformity metrics on CIFAR-10. To visualize the ability of uniformity and alignment with different methods under different augmentation settings (C0: standard, C1: erase, C2: crop, C3: brightness). The bar above the x axis (zero line) represents the alignment loss, while the bar under the x axis represents the uniformity loss. The shorter the color bars i.e with lower alignment loss and higher uniformity loss, correspond to the better performance of SSL models. E.3 Visualizing transport plans of different methods after training Here we compared the optimal transport (OT) plans of different methods after training for 500 epochs under standard augmentation C0 settings in Appendix D.1 . Specifically, we analyzed the -\log(\mathbf{P}) matrices of INCE, GCA-INCE, GCA-RINCE, and GCA-UOT, as shown in Figure A4 . In these matrices, darker blue regions represent higher similarity, while lighter blue areas indicate less similarity. The matrices are rearranged based on class labels, so an effective model should display empty diagonals and block structures aligned along the main diagonal and sub-diagonals—reflecting high intra-class similarity and low inter-class similarity. Figure A4 (A) shows that INCE results in a matrix with only row normalization. In contrast, Figures A4 (B) and (C) demonstrate that GCA-INCE and GCA-RINCE achieve both row and column normalization, leading to more uniform distributions. Figure A4 (D) reveals that GCA-UOT produces a matrix highlighting greater differences between positive and negative pairs, underscoring its effectiveness in distinguishing them. Figure A4: Comparison of the -\log(\mathbf{P}) matrix across different methods. (A) The INCE matrix with row normalization. (B) The -\log(\mathbf{P}) matrix of GCA-INCE with five iterations in forward pass, both row and column normalization. (C) The -\log(\mathbf{P}) matrix of GCA-RINCE with five iterations in forward pass. (D) The -\log(\mathbf{P}) matrix of GCA-UOT with five iterations in forward pass E.4 Hyperparameter Tuning and Sensitivity Analysis In our hyperparameter modifying experiments, we investigate the influence of key parameters in transport plan regularization, iteration counts, and augmentation strengths on CIFAR-10 classification performance. Figure A5 visualizes transport plans under varying entropic regularization ( \epsilon values from 0.01 to 1) across INCE and GCA-UOT models, illustrating adjustments after five iterations using the same ResNet-18 weights. Figure A6 examines the impact of iteration number and entropic regularization on compactness—measured by the average L2 distance to class centers—and accuracy, with 20 pre-training epochs followed by fine-tuning. Figure A7 highlights the sensitivity of GCA-RINCE to the hyperparameters q and \lambda , testing classification accuracy for different settings under strong augmentation conditions; this includes a comparison against INCE with large erase augmentation after substantial pre-training and evaluation epochs. Figure A5: Visualization transport plan P for different amounts of entropic regularization. (Top) The transport plans for \epsilon from 0.01 to 1 for INCE and (Bottom) GCA-UOT after 5 iterations. To compute each plan, we took a mini-batch on CIFAR-10 with 1024 samples, and loaded the same weights of Resnet-18 for each subfigure. Figure A6: Hyperparameter sensitivity study. The compactness and accuracy as a function of the (A) number of iterations and the (B) entropic regularization parameter. In our experiments, we use the same weights and perform 20 pre-training epochs for each point, then evaluate their performance by fine-tuning linear classifiers for 20 epochs. Here the compactness is the average L2 distance of each point to their corresponding class center on the representation space after the encoder. Figure A7: Hyperparameter sensitivity for q and \lambda in GCA-RINCE. Both experiments are tested on the CIFAR-10 dataset with a ResNet-18 encoder and involve strong augmentation with large erase. (Left) Given q=0.98 , we change \lambda from 0 to 1. (Right) Given \lambda=0.01 , we change q from 0 to 1. The red threshold line is the INCE performance with the large erase augmentation. Each point represents the CIFAR-10 classification accuracy of the ResNet-18 model pre-trained for 400 epochs and evaluated after 300 epochs.