Supervised Distributional Reduction via Optimal Transport and Dependence Maximization

Sai-Aakash Ramesh, Archit Sood, Andrew Corbett, Tim Dodwell · 2026

Read Supervised Distributional Reduction via Optimal Transport and Dependence Maximization by Sai-Aakash Ramesh, Archit Sood, Andrew Corbett, Tim Dodwell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27619.txt.

Introduction

Sai-Aakash Ramesh, Archit Sood, Andrew Corbett, and Tim Dodwell propose Supervised Distributional Reduction, learning codes that track the target, with optimal transport and a dependence boost. The method aligns input link structure with sample points through a fused Gromov-Wasserstein objective plus a direct dependence term. The resulting compact representations feed adaptive kernels for Gaussian process modeling under shifting geometry.

Catalog record

CREATORSai-Aakash Ramesh, Archit Sood, Andrew Corbett, Tim Dodwell
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65536
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27619

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Supervised Distributional Reduction via Optimal Transport and Dependence Maximization Sai-Aakash Ramesh, Archit Sood, Andrew Corbett, Tim Dodwell Abstract Learning representations that capture both intrinsic data geometry and target-relevant structure remains a fundamental challenge, particularly in settings where data reduction must balance compression with predictive fidelity. While distributional reduction-encompassing joint clustering and dimensionality reduction-offers a principled way to summarize data, its supervised variants remain relatively under-explored, despite the importance of retaining task-relevant signal for downstream prediction and decision-making. We propose Supervised Distributional Reduction (SDR), an algorithm for learning target-aware representations by combining optimal transport with explicit dependence maximization. SDR builds on the Fused Gromov-Wasserstein (FGW) objective to align the relational structure of the input distribution with a set of representative points, while augmenting it with a direct dependence term that encourages the learned embeddings to capture predictive signal more explicitly. This results in compact representations that reflect both geometric structure and supervision. Beyond representation learning, SDR naturally induces a data-dependent, non-stationary geometry that can be leveraged for settings such as Gaussian Process (GP) modelling. By redefining distances through target-aware distributional alignment, SDR enables the construction of adaptive kernels that respond to local variations in both data geometry and supervision, offering an optimal transport-based perspective on non-stationary kernel design. 1 Introduction Optimal Transport (OT) provides a principled framework for comparing probability distributions, with Gromov–Wasserstein (GW) ( 31 ) enabling alignment across heterogeneous spaces by matching relational structure. Building on this, Distributional Reduction (DistR) ( 2 ) learns a low-dimensional set of representative points that preserves the geometry of a dataset, providing a framework for unifying clustering and dimensionality reduction. However, existing distributional reduction methods are largely unsupervised, which can lead to representations that fail to retain task-relevant information and lack a clear mechanism for out-of-sample generalization. This prevents these methods from being used for downstream prediction tasks. We address these limitations by proposing Supervised Distributional Reduction (SDR), a framework for learning target-aware latent representations by combining optimal transport with explicit dependence maximization. SDR builds on the Fused Gromov–Wasserstein (FGW) objective ( 37 ) to align the relational structure of the input distribution with representative points, while incorporating supervision. Crucially, rather than relying solely on the coupling matrix—where supervision is mediated indirectly and may be diluted by structural constraints—we augment the objective with a direct representation-level dependence term based on Centered Kernel Alignment (CKA). This enables explicit alignment between embeddings and targets while preserving intrinsic data geometry. Beyond representation learning, SDR induces a data-dependent, non-stationary geometry that can be exploited for kernel learning. By constraining embeddings to admit an approximate kernel-based out-of-sample extension, we obtain a stable projection mechanism for unseen data. This allows SDR to define adaptive covariance functions for Gaussian Processes (GPs), where the learned geometry translates into non-stationary kernel behaviour that responds to both structure and supervision in the data. 1.1 Contributions We make the following contributions in this work: 1. Supervised distributional reduction via OT and dependence maximization: We propose SDR, an algorithm that combines OT-based alignment with explicit dependence maximization to learn compact, target-aware representations that preserve both geometry and predictive signal. 2. Out-of-sample extension via RKHS projection with applications to GPs: We formulate the mapping from inputs to embeddings as a regularized kernel ridge regression problem, yielding an out-of-sample transformation map that allows SDR to be used as part of predictive pipelines and for non-stationary kernel construction in GPs. Figure 1 : Supervised Distributional Reduction (SDR) with Fused Gromov-Wasserstein OT formulation and explicit dependence maximization for target-aware distributional reduction. 2 Related Work 2.1 Supervised Dimensionality Reduction and Clustering Supervised dimensionality reduction (DR) methods incorporate label information to learn representations that are more discriminative for downstream tasks. Classical approaches such as Linear Discriminant Analysis (LDA) ( 11 ) maximize class separability through linear projections, while metric learning methods, including Neighbourhood Components Analysis (NCA) ( 15 ) and Large Margin Nearest Neighbour (LMNN) ( 40 ) , shape the embedding space using supervised distance objectives. Extensions of unsupervised DR techniques have also been proposed in supervised settings; for instance, Supervised UMAP ( 24 ) integrates label information to guide the preservation of local structure in a target-aware manner. In parallel, dependence-based approaches maximize statistical dependence between representations and targets. For example, using the Hilbert Schmidt Independence Criterion (HSIC) ( 16 ) in conjunction with PCA gives rise to a version of Supervised PCA ( 4 ) . On the other hand, supervised and semi-supervised clustering methods introduce constraints such as must-link and cannot-link pairs to guide cluster formation ( 39 ) . 43 suggest providing supervision in the form of distances (similarities) for metric-informed clustering. Despite these advances, most approaches either focus solely on embedding or clustering, or rely on heuristic combinations of the two, lacking a unified formulation that captures both aspects at the distributional level. 2.2 Optimal Transport, GW/FGW, and Distributional Reduction Optimal transport (OT) provides a principled framework for comparing probability distributions by minimizing the cost of transporting probability mass between them ( 26 ; 19 ) . Beyond classical Wasserstein distances, Gromov–Wasserstein (GW) extends OT to settings where only relational information, such as pairwise distances, is available, enabling comparisons across heterogeneous spaces ( 31 ) . The Fused Gromov–Wasserstein (FGW) formulation further combines feature-level and structural alignment by integrating Wasserstein and GW objectives into a single framework ( 37 ) . These formulations provide a natural mechanism for geometry-aware representation learning. Building on these ideas, OT-based approaches have been proposed for distributional summarization, including Wasserstein barycenters ( 8 ) and reduced-support approximations ( 12 ) . Methods such as DistR ( 2 ) leverage GW to jointly perform clustering and dimensionality reduction by learning a reduced set of representative points that preserve dataset geometry. However, these approaches are largely unsupervised and do not explicitly incorporate target information during the reduction process. SDR builds on this line of work by integrating supervision directly into the distributional reduction objective, enabling the learning of compact representations that are both geometry-preserving and target-aware. 3 Method 3.1 Distributional Reduction with OT Distributional Reduction (DistR) ( 2 ) formulates joint clustering and DR as an optimal transport problem between the original dataset and a reduced set of representative embeddings. Given input data X\in\mathbb{R}^{n\times d} , DistR seeks a reduced representation Z\in\mathbb{R}^{m\times p} with m\leq n and p\leq d by solving a Gromov–Wasserstein (GW) OT problem between their respective metric structures. Let C and \bar{C} denote pairwise distance matrices in the input and embedding spaces respectively and \textbf{h}_{X} denote the weight vector typically assumed to be uniform, that is \textbf{h}_{X}=\frac{1}{n}\textbf{1}_{n} . DistR solves: \underset{\begin{subarray}{c}\textbf{T}\in\mathcal{U}(\textbf{h}_{X})\\ \textbf{Z}\in\mathbb{R}^{m\times p}\\ \textbf{h}_{Z}\in\Sigma_{m}\end{subarray}}{\text{min}}\Sigma_{ijkl}\ L_{u}(C_{ij},\bar{C}_{kl})T_{ik}T_{jl} (DistR) where T is a transport plan in \mathcal{U}(\textbf{h}_{X})=\{T\in\mathbb{R}_{+}^{n\times m}:T\textbf{1}_{m}=\textbf{h}_{X}\} between the input and reduced representations, and L_{u} is an unsupervised, structural loss function comparing pairwise distances. The problem is optimized via alternating minimization over T and Z , where T is computed using standard GW solvers and Z is updated using gradient-based methods. When m<n , this yields a reduced set of representative points weighted non-uniformly with h_{Z} , unifying clustering and dimensionality reduction within a single framework. 3.2 Supervised Distributional Reduction In this section, we extend the Distributional Reduction Algorithm to the supervised setting. SDR optimizes a single unified objective over (Z,T)

Read the complete text