Deep Kuratowski Embedding Neural Networks for Wasserstein Metric Learning

Andrew Qing He · 2026

Read Deep Kuratowski Embedding Neural Networks for Wasserstein Metric Learning by Andrew Qing He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04343.txt.

Introduction

He proposes two neural designs that learn to approximate the Wasserstein-2 distance from data, easing a core bottleneck in data analysis. DeepKENN pools distances across all intermediate feature maps of a convolutional network using learnable positive weights. ODE-KENN swaps the layer stack for a Neural ODE, embedding each input into an infinite-dimensional Banach space with implicit regularization from trajectory smoothness. On MNIST with exact precomputed distances, ODE-KENN beats the single-layer baseline by 28 percent lower test error and DeepKENN by 18 percent, with a smaller generalization gap.

Catalog record

CREATORAndrew Qing He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22659
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep Kuratowski Embedding Neural Networks for Wasserstein Metric Learning Andrew Qing He Abstract Computing pairwise Wasserstein distances is a fundamental bottleneck in data analysis pipelines such as Wassmap [ 6 ] and LOT Wassmap [ 3 ] . Motivated by the classical Kuratowski embedding theorem, we propose two neural architectures for learning to approximate the Wasserstein-2 distance ( W 2 W_{2} ) from data. The first, DeepKENN , aggregates distances across all intermediate feature maps of a CNN using learnable positive weights. The second, ODE-KENN , replaces the discrete layer stack with a Neural ODE, embedding each input into the infinite-dimensional Banach space C 1 ​ ( [ 0 , 1 ] , ℝ d ) C^{1}([0,1],\mathbb{R}^{d}) and providing implicit regularization via trajectory smoothness. Experiments on MNIST with exact precomputed W 2 W_{2} distances show that ODE-KENN achieves a 28% lower test MSE than the single-layer baseline and 18% lower than DeepKENN under matched parameter counts, while exhibiting a smaller generalization gap. The resulting fast surrogate can replace the expensive W 2 W_{2} oracle in downstream pairwise distance computations. 1 Introduction The Wasserstein distance, or Earth Mover’s Distance, is a metric on the space of probability measures that has found widespread application in machine learning, image analysis, and data science [ 11 , 10 ] . Unlike divergences such as KL or total variation, the Wasserstein distance takes the underlying geometry of the sample space into account, making it well-suited for comparing structured objects such as images. Despite its theoretical appeal, the practical computation of Wasserstein distances presents a significant bottleneck. Solving the exact linear program for W_{2} between two discrete measures supported on n points requires O(n^{3}\log n) time [ 1 ] , and even approximate methods such as Sinkhorn iteration [ 4 ] remain expensive at scale. This computational cost is particularly acute in algorithms that require the pairwise W_{2} distance matrix between a dataset of measures. A prominent example is Wassmap [ 6 ] , a nonlinear dimensionality reduction technique that embeds a dataset of images into a low-dimensional Euclidean space by applying Isomap to the pairwise W_{2} distance matrix. Wassmap has been shown to exactly recover latent parameters of image manifolds generated by translations or dilations, but its scalability is limited by the cost of constructing the pairwise distance matrix. The subsequent LOT Wassmap [ 3 ] addresses this by replacing exact W_{2} with linearized optimal transport (LOT) approximations, providing approximation guarantees while avoiding the full pairwise matrix. Nevertheless, even LOT-based methods incur significant per-pair computational cost that could be alleviated by a fast learned surrogate. Our contribution. We propose and experimentally validate two neural architectures for learning the W_{2} distance from a training set of measure pairs with precomputed ground-truth distances: 1. DeepKENN : A convolutional neural network that computes distances as a weighted sum of squared Euclidean distances across all intermediate feature maps, with learnable positive weights enforced via a softplus reparameterization. This is motivated by the Kuratowski embedding theorem, which guarantees that any metric space embeds isometrically into a Banach space. 2. ODE-KENN : An extension that replaces the discrete multi-layer sum with a Neural ODE on the feature space, producing a trajectory-integral distance. The embedding space is now infinite-dimensional ( C^{1}([0,1],\mathbb{R}^{d}) ), and the ODE’s smoothness constraint acts as a natural regularizer. Both architectures are compared to a naive baseline (Euclidean distance in the final feature space) under matched parameter counts on MNIST W_{2} distance learning. Relationship to existing work. Our multi-layer distance formulation is closely related to the perceptual loss of Johnson et al. [ 7 ] , which also aggregates feature-map differences across layers of a pretrained CNN for image synthesis tasks. Our contribution differs in that the weights are learned end-to-end alongside the backbone, and the goal is metric approximation rather than image generation. The Neural ODE component follows Chen et al. [ 2 ] , with the key novelty that the distance is defined via the trajectory integral rather than the terminal state. 2 Background 2.1 Wasserstein Distance Let (\mathcal{X},c) be a metric space and let \mu,\nu\in\mathcal{P}_{2}(\mathcal{X}) be probability measures with finite second moments. The Wasserstein-2 distance between \mu and \nu is W_{2}(\mu,\nu)=\left(\inf_{\gamma\in\Pi(\mu,\nu)}\int_{\mathcal{X}\times\mathcal{X}}c(x,y)^{2}\,d\gamma(x,y)\right)^{1/2}, (1) where \Pi(\mu,\nu) denotes the set of all couplings (joint distributions with marginals \mu and \nu ). For discrete measures supported on finite grids, this reduces to a linear program solvable by the network simplex method. 2.2 Kuratowski Embedding The theoretical motivation for our approach is the following classical result. Theorem 1 (Kuratowski–Wojdysławski) . Every bounded metric space (M,d) embeds isometrically as a closed subset of a convex set in the Banach space \ell^{\infty}(M) . In practice, learning an isometric embedding into \ell^{\infty} is computationally intractable due to its infinite dimensionality. The Johnson–Lindenstrauss lemma [ 8 ] provides a finite-dimensional substitute: any N -point metric space embeds into \mathbb{R}^{k} with distortion 1+\varepsilon for k=O(\varepsilon^{-2}\log N) . However, both approaches are passive — they describe existence of embeddings, not how to learn one. Our work replaces the existence argument with a trainable embedding. 2.3 Neural ODEs A Neural ODE [ 2 ] defines the hidden state h(t) of a network as the solution to the initial value problem \frac{dh}{dt}=\Phi_{\boldsymbol{\vartheta}}(h(t),t),\quad h(0)=h_{0}, (2) where \Phi_{\boldsymbol{\vartheta}} is a neural network. The output is h(T) for some final time T . Neural ODEs can be viewed as the continuous-depth limit of ResNets, and gradients are computed efficiently via the adjoint method. A key property is that the map h_{0}\mapsto h(T) is a diffeomorphism when \Phi_{\boldsymbol{\vartheta}} is Lipschitz, implying injectivity of the learned embedding. 3 Deep Kuratowski Embedding Neural Network (DeepKENN) 3.1 Architecture Let F_{\boldsymbol{\vartheta}}:\mathcal{X}\to\mathbb{R}^{d_{L}} be a neural network with L layers, so that F_{\boldsymbol{\vartheta}}=f^{(L)}_{\boldsymbol{\vartheta}}\circ\cdots\circ f^{(1)}_{\boldsymbol{\vartheta}}. Denote the output of the k -th layer as F^{(k)}_{\boldsymbol{\vartheta}}(x):=f^{(k)}_{\boldsymbol{\vartheta}}\circ\cdots\circ f^{(1)}_{\boldsymbol{\vartheta}}(x)\in\mathbb{R}^{d_{k}} . Naive baseline. The simplest approach defines the distance as \hat{d}(x,\tilde{x})=\left\|F^{(L)}_{\boldsymbol{\vartheta}}(x)-F^{(L)}_{\boldsymbol{\vartheta}}(\tilde{x})\right\|_{2}. (3) This uses only the final layer representation of dimension d_{L} , discarding all intermediate structure. DeepKENN. We propose instead to use all intermediate representations: \hat{d}(x,\tilde{x})=\sqrt{\sum_{k=1}^{L}\lambda_{k}\left\|F^{(k)}_{\boldsymbol{\vartheta}}(x)-F^{(k)}_{\boldsymbol{\vartheta}}(\tilde{x})\right\|_{2}^{2}}, (4) where \lambda_{k}\geq 0 are learnable scalars. To enforce positivity, we parameterize \lambda_{k}=\text{softplus}(\theta_{k}) for unconstrained \theta_{k}\in\mathbb{R} . Embedding interpretation. The distance ( 4 ) corresponds to a Euclidean distance in the product space \mathbb{R}^{d_{1}}\times\cdots\times\mathbb{R}^{d_{L}} , equipped with the weighted inner product \langle u,v\rangle=\sum_{k}\lambda_{k}\langle u_{k},v_{k}\rangle . The effective embedding dimension is d_{\text{tot}}=\sum_{k=1}^{L}d_{k} , which can be substantially larger than d_{L} alone. This is the discrete analog of the Kuratowski embedding. 3.2 Triangle Inequality An important structural property is that ( 4 ) is guaranteed to satisfy the triangle inequality for any fixed backbone F_{\boldsymbol{\vartheta}} and any \lambda_{k}\geq 0 , since it is a Euclidean norm in the product embedding space. The positive definiteness (i.e., \hat{d}(x,\tilde{x})=0\Leftrightarrow x=\tilde{x} ) depends on the injectivity of the joint map (F^{(1)}_{\boldsymbol{\vartheta}},\ldots,F^{(L)}_{\boldsymbol{\vartheta}}) , which holds generically for sufficiently expressive networks. 4 ODE-KENN 4.1 Motivation In DeepKENN, the effective embedding dimension d_{\text{tot}}=\sum_{k}d_{k} is finite and grows only with the number of layers. The Kuratowski theorem, however, requires the embedding space to be infinite-dimensional for perfect isometry with an arbitrary metric space. This motivates replacing the discrete layer stack with a continuous-depth analog. 4.2 Architecture ODE-KENN consists of two components: a CNN encoder E_{\boldsymbol{\vartheta}}:\mathcal{X}\to\mathbb{R}^{d} that maps an input image to a fixed-dimensional feature vector, followed by

Read the complete text