Distance-informed Neural Processes
Aishwarya Venkataramanan, Joachim Denzler · 2025
Read Distance-informed Neural Processes by Aishwarya Venkataramanan, Joachim Denzler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18903.txt.
Introduction
Aishwarya Venkataramanan and Joachim Denzler propose the Distance-informed Neural Process, a new member of the neural process family that aims to judge its own errors more honestly. Standard neural processes lean on a single global hidden code, and they tend to misjudge how unsure they are. They also miss the way nearby inputs should behave alike.
The new model splits the code in two. A global part tracks how tasks vary as a whole, while a local part captures how close inputs sit to one another. A distance-keeping rule limits how far the coding step can distort those gaps, so points that start out close stay close in the code. Tests on regression and classification tasks show strong scores and error bars that match reality more closely. The model also separates familiar data from strange new data more cleanly.
Catalog record
| CREATOR | Aishwarya Venkataramanan, Joachim Denzler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18903
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distance-informed Neural Processes Aishwarya Venkataramanan, Joachim Denzler Abstract We propose the Distance-informed Neural Process (DNP), a novel variant of Neural Processes that improves uncertainty estimation by combining global and distance-aware local latent structures. Standard Neural Processes (NPs) often rely on a global latent variable and struggle with uncertainty calibration and capturing local data dependencies. DNP addresses these limitations by introducing a global latent variable to model task-level variations and a local latent variable to capture input similarity within a distance-preserving latent space. This is achieved through bi-Lipschitz regularization, which bounds distortions in input relationships and encourages the preservation of relative distances in the latent space. This modeling approach allows DNP to produce better-calibrated uncertainty estimates and more effectively distinguish in- from out-of-distribution data. Empirical results demonstrate that DNP achieves strong predictive performance and improved uncertainty calibration across regression and classification tasks. 1 Introduction Deep neural networks have achieved remarkable success in fields such as computer vision chai2021deep ; venkataramanan2023integrating , natural language processing vaswani2017attention ; otter2020survey , and reinforcement learning franccois2018introduction . However, they often produce overconfident and unreliable predictions, particularly when encountering data that fall outside the support of the training set kendall2017uncertainties ; venkataramanan2023gaussian ; guo2017calibration ; abdar2021review ; venkataramanan2025probabilistic . Stochastic models such as Gaussian Processes (GPs) williams2006gaussian address this limitation by defining a prior over functions using kernel functions, which encode the assumption that similar inputs produce similar outputs. Kernel functions measure similarity using a distance metric, assigning higher scores to inputs closer to the training data. Consequently, when the model encounters data points that are far from the training set, the predictive distribution reverts to the prior. In many cases, this prior has a high variance, indicating a high uncertainty about the output. However, the practicality of vanilla GPs is limited by their high computational cost, scaling cubically with the size of the training data and by their lack of flexibility in high-dimensional problems. This paved way to the development of Neural Processes (NPs) garnelo2018conditional ; garnelo2018neural , that leverage meta-learning principles hospedales2021meta to learn a distribution over functions from observed data. Unlike traditional kernel-based methods, NPs employ neural encoders to infer a global latent variable that summarizes the underlying function. The NP then conditions its predictions on the inputs and the global latent variable, facilitating rapid adaptation to new functions. However, the reliance on a single global latent variable limits representational expressiveness, and standard NPs often produce uncertainty estimates that are not well calibrated gordon2019convolutional ; kim2019attentive ; wang2020doubly . (a) Input (b) Vanilla MLP embeddings (c) Bi-Lipschitz MLP embeddings Figure 1: Visualization of input data and learned representations on a 2D synthetic data. (a) shows the original data. (b) shows representations from a 3-layer vanilla MLP, where the input structure is distorted. (c) shows representations from the MLP with bi-Lipschitz regularization, which better preserves the input structure and relative distances, important for computing reliable input similarity. To address these limitations, we propose the Distance-informed Neural Process (DNP), which integrates both global and local priors for better uncertainty estimation and generalization. The global prior functions like standard NPs, capturing task-specific patterns. The local prior models distance relationships between observed data and test inputs through a latent space, introducing an inductive bias similar to GPs. However, neural networks used to parameterize these latent spaces can warp the input geometry causing similar points to be mapped far apart and unrelated points to appear close together in the learned representation liu2020simple ; van2021feature ; zha2023rank . To mitigate this, we add a bi-Lipschitz regularization on the local latent network’s weight matrices. This constrains each layer’s singular values within a fixed range, which minimizes distortions and encourages the latent space to approximately preserve input distances. The distance-preserving latent space is then used to define a local prior that encodes data similarity. As shown in Fig. 1 , the latent representations learned by a vanilla neural network distorts input geometry, while bi-Lipschitz regularization helps preserve the input structure by minimizing such distortions 1 1 1 For illustrative purposes, we used a special case where the bi-Lipschitz bounds were set to 1 , resulting in an isometric mapping. . By conditioning on both global and distance-aware local latent variables, DNP yields better-calibrated uncertainty estimates and improved distinction between in- and out-of-distribution (OOD) data. We demonstrate its effectiveness in regression and classification through experiments on synthetic and real-world datasets. Code is available: https://github.com/cvjena/DNP.git . 2 Background Let \mathcal{D}={(\mathbf{x}_{i},\mathbf{y}_{i})}_{i=1}^{N} represent a dataset consisting of input-output pairs, where \mathbf{x}_{i}\in\mathbb{R}^{d_{x}} is an input and \mathbf{y}_{i}\in\mathbb{R}^{d_{y}} is the corresponding output. NPs define a family of conditional distributions over functions f\in\mathcal{F} , mapping \mathbf{x} to \mathbf{y} . The dataset is split into a context set (\mathbf{x}_{C},\mathbf{y}_{C})=(\mathbf{x}_{i},\mathbf{y}_{i})_{i=1}^{M} , where M\leq N is an arbitrary subset of \mathcal{D} , and a target set (\mathbf{x}_{T},\mathbf{y}_{T})=(\mathbf{x}_{i},\mathbf{y}_{i})_{i=1}^{N} , which includes all data points. The model uses the context set to create a summary that conditions how it predicts the target outputs \mathbf{y}_{T} from the target inputs \mathbf{x}_{T} . Different variants of NPs differ in how this context-based conditioning is performed and how the predictive distribution is defined. The Conditional Neural Process (CNP) garnelo2018conditional models the conditional distribution as p(\mathbf{y}_{T}|\mathbf{x}_{C},\mathbf{y}_{C},\mathbf{x}_{T})=p(\mathbf{y}_{T}|\mathbf{r}_{C},\mathbf{x}_{T}), where \mathbf{r}_{C}=\mathrm{Agg}((\mathbf{x}_{c},\mathbf{y}_{c})_{c\in C}) is a permutation-invariant summary (e.g. mean) of the context points. Each (\mathbf{x}_{c},\mathbf{y}_{c}) is first encoded by a neural network, after which \mathbf{r}_{C} and \mathbf{x}_{T} parameterize a factorized Gaussian distribution over \mathbf{y}_{T} . The model is trained by maximizing the log-likelihood of the target set conditioned on the context set. In addition to the deterministic encoding of the context set, Neural Process (NP) garnelo2018neural introduces a global latent variable \mathbf{z}_{G}\in\mathbb{R}^{d_{z}} to capture task-level uncertainty. A separate neural network encodes (\mathbf{x}_{C},\mathbf{y}_{C}) by first aggregating the context set into a global summary \mathbf{s}_{C} , which is then used to parameterize a Gaussian distribution over \mathbf{z}_{G} . The decoder models the conditional distribution over \mathbf{y}_{T} using \mathbf{x}_{T} , \mathbf{r}_{C} , and \mathbf{z}_{G} : p(\mathbf{y}_{T}|\mathbf{x}_{T},\mathbf{r}_{C},\mathbf{s}_{C})=\int p(\mathbf{y}_{T}|\mathbf{x}_{T},\mathbf{r}_{C},\mathbf{z}_{G})\,p(\mathbf{z}_{G}|\mathbf{s}_{C})\,d\mathbf{z}_{G} (1) The Attentive Neural Process (AttnNP) kim2019attentive extends NP by replacing the shared context representation \mathbf{r}_{C} with a target-specific representation \mathbf{r}_{t} computed using self-attention over the context and cross-attention with each target input \mathbf{x}_{t} . The decoder then conditions on \mathbf{x}_{t} , \mathbf{r}_{t} and \mathbf{z}_{G} to model the target distribution. While AttnNP uses attention to capture input-specific patterns, it does not explicitly preserve distance relationships among inputs. Its attention weights, computed over learned embeddings, may distort input relations zha2023rank ; van2021feature , hindering the capture of local dependencies crucial for similarity-based predictions. In contrast, while DNP also employs cross-attention to obtain target-specific latent variables, its key contribution is the construction of a distance-aware latent space via bi-Lipschitz regularization, from which attention weights are computed. 3 Distance-informed Neural Process Figure 2: The encoder consists of global and local latent paths, each learning a distribution over its respective latent variables. The local path incorporates bi-Lipschitz regularization to approximately preserve input distances to model similarity relations among the data. The decoder then conditions on the sampled latents and the target embedding to predict the output. The Distance-informed Neural Process (DNP) is illustrated in Fig. 2 . The encoder consists of both global and local latent paths. The global path models a distribution over the global latent variable \mathbf{z}_{G}\in\mathbb{R}^{d_{z}} , conditioned on