Dimension Agnostic Neural Processes
Hyungi Lee, Chaeyun Jang, Dong bok Lee, Juho Lee · 2025
Read Dimension Agnostic Neural Processes by Hyungi Lee, Chaeyun Jang, Dong bok Lee, Juho Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20661.txt.
Introduction
Hyungi Lee and co-authors teach meta-learning models to swallow data of any width. Their 2025 paper builds on neural processes, which learn a stochastic map from a few examples and own up to their uncertainty. Older versions choke when tasks arrive with different input dimensions. Their model, DANP, first squeezes each input into a fixed-size code through a dimension aggregator block.
A transformer with latent layers then reads wider, more portable features across tasks. Tests on synthetic and real regression jobs show DANP beating prior neural process variants. The authors pitch it as a general regressor for few-shot settings, where new tasks show up in shapes the model has never seen.
Catalog record
| CREATOR | Hyungi Lee, Chaeyun Jang, Dong bok Lee, Juho Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20661
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dimension Agnostic Neural Processes Hyungi Lee, Chaeyun Jang, Dong bok Lee, Juho Lee Abstract Meta-learning aims to train models that can generalize to new tasks with limited labeled data by extracting shared features across diverse task datasets. Additionally, it accounts for prediction uncertainty during both training and evaluation, a concept known as uncertainty-aware meta-learning. Neural Process ( np ) is a well-known uncertainty-aware meta-learning method that constructs implicit stochastic processes using parametric neural networks, enabling rapid adaptation to new tasks. However, existing np methods face challenges in accommodating diverse input dimensions and learned features, limiting their broad applicability across regression tasks. To address these limitations and advance the utility of np models as general regressors, we introduce Dimension Agnostic Neural Process ( danp ) . danp incorporates Dimension Aggregator Block ( dab ) to transform input features into a fixed-dimensional space, enhancing the model’s ability to handle diverse datasets. Furthermore, leveraging the Transformer architecture and latent encoding layers, danp learns a wider range of features that are generalizable across various tasks. Through comprehensive experimentation on various synthetic and practical regression tasks, we empirically show that danp outperforms previous np variations, showcasing its effectiveness in overcoming the limitations of traditional np models and its potential for broader applicability in diverse regression scenarios. 1 Introduction In real-world datasets, there are many tasks that come with various configurations (such as input feature dimensions, quantity of training data points, correlation between training and validation data, etc.). However, each task has a limited number of data points available, making it difficult to train a model capable of robust generalization solely based on the provided training data. To tackle this issue, meta-learning aims to train a model capable of generalizing to new tasks with few labeled data by learning generally shared features from diverse training task datasets. In cases of limited labeled data for new target tasks, ensuring model trustworthiness involves accurately quantifying prediction uncertainty, which is as critical as achieving precise predictions. A meta-learning strategy that considers prediction uncertainty during training and evaluation is known as uncertainty-aware meta-learning ( Nguyen & Grover, 2022 ; Almecija et al., 2022 ) . One of the well-known uncertainty-aware meta-learning methods is Neural Process ( np ) ( Garnelo et al., 2018a ; Garnelo et al., 2018b ) . Np employs meta-learning to understand the data-generation process governing the relationship between input-output pairs in meta-training and meta-validation data. Unlike the traditional approach to learning stochastic processes, where model selection from a known class, e.g. Gaussian Processes ( gp s) , precedes computing predictive distributions based on training data, np constructs an implicit stochastic process using parametric neural networks trained on meta-training data. It then optimizes parameters to maximize the predictive likelihood for both the meta-train and meta-validation data. Consequently, when np effectively learns the data-generation process solely from data, it can quickly identify suitable stochastic processes for new tasks. Thus, np can be viewed as a data-driven uncertainty-aware meta-learning method for defining stochastic processes. However, previous works ( Gordon et al., 2020 ; Foong et al., 2020 ; Lee et al., 2020 ; Nguyen & Grover, 2022 ; Lee et al., 2023 ) in np literature lack two crucial attributes essential for broad applicability across different regression tasks: 1) the ability to directly accommodate diverse input and output dimensions, and 2) the adaptability of learned features for fine-tuning on new tasks that exhibit varying input and output dimensions. Due to the absence of these two properties, it is necessary to train each np model separately for different dimensional tasks. These limitations hinder the utility of NP models as general regressors across diverse datasets compared to traditional stochastic processes ( Lee et al., 2021 ) . Traditional stochastic processes naturally accommodate varying input dimensions, particularly in regression tasks involving high-dimensional input features with limited training data, such as hyperparameter optimization tasks. To tackle these limitations and advance the utilization of np models as general regressors for a wide range of regression tasks, we introduce a novel extension of np called Dimension Agnostic Neural Process ( danp ) . In danp , we propose a module called Dimension Aggregator Block ( dab ) , which transforms input features of varying dimensions into a fixed-dimensional representation space. This allows subsequent np modules to effectively handle diverse datasets and generate predictive density for the meta-validation data. We also add the Transformer architecture ( Vaswani et al., 2017 ) alongside latent encoding layers based on the architecture of Transformer Neural Processes ( tnp ) ( Nguyen & Grover, 2022 ) to enhance the model’s ability to learn a wider range of features and effectively capture functional uncertainty, which can be applied across different tasks. Through experimentation on a variety of synthetic and real-world regression tasks with various situations, we show that danp achieves notably superior predictive performance compared to previous np variations. 2 Background 2.1 Problem settings Let \mathcal{X} be an input space defined as \bigcup_{i\in\mathbb{N}}\mathcal{X}_{i} with each \mathcal{X}_{i}\subseteq\mathbb{R}^{i} for all i\in\mathbb{N} . Similarly, let \mathcal{Y}=\bigcup_{i\in\mathbb{N}}\mathcal{Y}_{i} represent the output space, where each \mathcal{Y}_{i}\subseteq\mathbb{R}^{i} for all i\in\mathbb{N} . Let \mathbf{T}=\{\tau_{j}\}_{j\in\mathbb{N}} be a task set drawn in i.i.d. fashion from a task distribution p_{\text{task}}(\tau) . Given two dimension mapping functions u,v:\mathbb{N}\rightarrow\mathbb{N} , each task \tau_{j} comprises a dataset {\mathcal{D}}_{j}=\{\mathbf{d}_{j,k}\}_{k=1}^{n_{j}} , where \mathbf{d}_{j,k}=(\mathbf{x}_{j,k},\mathbf{y}_{j,k})\in\mathcal{X}_{u(j)}\times\mathcal{Y}_{v(j)} represents an input-output data pair, along with an index set c_{j}\subsetneq[n_{j}] where [m]:=\{1,\dots,m\} for all m\in\mathbb{N} . We assume elements in \mathcal{D}_{j} are i.i.d. conditioned on some function f_{j} . Here, the set of indices c_{j} defines the context set {\mathcal{D}}_{j,c}:=\{\mathbf{d}_{j,k}\}_{k\in c_{j}} . Similarly, the target set is defined as \mathcal{D}_{j,t}:=\{\mathbf{d}_{j,k}\}_{k\in t_{j}} where t_{j}:=[n_{j}]\setminus c_{j} . We aim to meta-learn a collection of random functions f_{j}:\mathcal{X}_{u(j)}\to\mathcal{Y}_{v(j)} , where each function within this set effectively captures and explains the connection between input x and output y pairs. For any given meta-training task \tau_{j} , we can regard its context set {\mathcal{D}}_{j,c} as the meta-training set and its target set {\mathcal{D}}_{j,t} as the meta-validation set. 2.2 Neural processes For the previous np variants, their objective was to meta-learn a set of random functions f_{j}:\mathcal{X}_{d_{\text{in}}}\to\mathcal{Y}_{d_{\text{out}}} , for some fixed d_{\text{in}},d_{\text{out}}\in\mathbb{N} , which is equal to the situation where dimension mapping functions u,v are constant functions, i.e., u(j)=d_{\text{in}} and v(j)=d_{\text{out}} for all j\in\mathbb{N} . In this context, to select a suitable random function f_{j} for the task \tau_{j} , np s meta-learns how to map the context set {\mathcal{D}}_{j,c} to a random function f_{j} that effectively represents both the context set {\mathcal{D}}_{j,c} and the target set {\mathcal{D}}_{j,t} . This entails maximizing the likelihood for both meta-training and meta-validation datasets within an uncertainty-aware meta-training framework. The process involves learning a predictive density that maximizes the likelihood using the following equation: \displaystyle p(\mathbf{Y}_{j}|\mathbf{X}_{j},{\mathcal{D}}_{j,c})=\int\bigg[\prod_{k\in[n_{j}]}p(\mathbf{y}_{j,k}|f_{j},\mathbf{x}_{j,k})\bigg]p(f_{j}|{\mathcal{D}}_{j,c})\mathrm{d}f_{j}, (1) where \mathbf{X}_{j}=\{\mathbf{x}_{j,k}\}_{k=1}^{n_{j}} and \mathbf{Y}_{j}=\{\mathbf{y}_{j,k}\}_{k=1}^{n_{j}} . In line with our discussion in Section 2.1 , we make the assumption that given the random function f_{j} , the outputs collection \mathbf{Y}_{j} are i.i.d. Employing the Gaussian likelihood and parameterizing f_{j} with latent variable r_{j}\in\mathbb{R}^{d_{j}} , Eq. 1 reduces to, \displaystyle p(\mathbf{Y}_{j}|\mathbf{X}_{j},{\mathcal{D}}_{j,c})=\int\bigg[\prod_{k\in[n_{j}]}{\mathcal{N}}\left(\mathbf{y}_{j,k}|\mu_{r_{j}}(\mathbf{x}_{j,k}),\text{diag}(\sigma_{r_{j}}^{2}(\mathbf{x}_{j,k}))\right)\bigg]p(r_{j}|{\mathcal{D}}_{j,c})\mathrm{d}r_{j}, (2) where \mu_{r_{j}}:{\mathcal{X}}_{d_{\text{in}}}\rightarrow{\mathcal{Y}}_{d_{\text{out}}} and \sigma_{r_{j}}^{2}:{\mathcal{X}}_{d_{\text{in}}}\rightarrow\mathbb{R}_{+}^{d_{\text{out}}} . Then different np variants aim to effectively design the model structures of the encoder, denoted as f_{\text{enc}} , and the decoder, denoted as f_{\text{dec}} . These components are responsible for describing