Amortized Spectral Kernel Discovery via Prior-Data Fitted Network
Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek · 2026
Read Amortized Spectral Kernel Discovery via Prior-Data Fitted Network by Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21731.txt.
Introduction
Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, and Parikshit Pareek extract readable kernels from networks fitted on synthetic data. Such prior-data fitted networks do fast, learned inference, but their priors and kernels stay hidden inside. That opacity blocks uses like surrogate optimization, which need an explicit covariance model.
The authors build a framework that recovers the spectral story of the learned kernel. A decoupled attention design helps separate the pieces. Making the kernel visible turns a black-box predictor into a tool other methods can build on. It also lets users check whether the learned prior matches the problem at hand.
Catalog record
| CREATOR | Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Amortized Spectral Kernel Discovery via Prior-Data Fitted Network Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek Abstract Prior-Data Fitted Networks (PFNs) enable efficient amortized inference but lack transparent access to their learned priors and kernels. This opacity hinders their use in downstream tasks, such as surrogate-based optimization, that require explicit covariance models. We introduce an interpretability-driven framework for amortized spectral discovery from pre-trained PFNs with decoupled attention. We perform a mechanistic analysis on a trained PFN that identifies attention latent output as the key intermediary, linking observed function data to spectral structure. Building on this insight, we propose decoder architectures that map PFN latents to explicit spectral density estimates and corresponding stationary kernels via Bochner’s theorem. We study this pipeline in both single-realization and multi-realization regimes, contextualizing theoretical limits on spectral identifiability and proving consistency when multiple function samples are available. Empirically, the proposed decoders recover complex multi-peak spectral mixtures and produce explicit kernels that support Gaussian process regression with accuracy comparable to PFNs and optimization-based baselines, while requiring only a single forward pass. This yields orders-of-magnitude reductions in inference time compared to optimization-based baselines. † † affiliationnotice: Equal contribution 1 Introduction The central question in modern probabilistic modeling is the tradeoff between structural inductive bias and computational scalability . Kernel-based methods, particularly Gaussian Processes (GPs), offer exact uncertainty quantification and, through the covariance kernel, an explicit language to describe properties like periodicity, smoothness, and stationarity ( 15 ) . This explicit structure is indispensable in scientific discovery, where the goal is often to identify the data-generating mechanism (e.g., the spectral density of a signal) rather than merely fitting points. However, beyond the use of standard kernels ( 15 ) , identifying the correct kernel structure is computationally prohibitive. Methods such as Deep Kernel Learning (DKL) ( 21 ) or the Automatic Statistician ( 8 ) rely on expensive iterative marginal likelihood optimization for every new dataset, creating a bottleneck that prevents real-time deployment. Conversely, amortized Bayesian inference methods, including Prior-Data Fitted Networks (PFNs) ( 9 ) and Neural Processes ( 5 ) , address the scalability challenge. By pre-training on a distribution of priors, PFNs learn to map datasets directly to posterior predictive distributions in a single forward pass. These models achieve strong predictive performance and can approximate Bayesian inference procedures across a wide range of tasks ( 10 ; 9 ; 12 ; 1 ) . However, this amortization comes at the cost of structural opacity . The spectral density and covariance structure that define the Bayesian prior are no longer explicitly represented, but instead are implicitly encoded in the network parameters, limiting interpretability and downstream structured reasoning ( 10 ) . This work bridges this gap by proposing a framework for amortized spectral and kernel discovery . We address two limitations emphasized in recent foundational perspectives on PFNs ( 10 ) : (1) the lack of mechanistic transparency and (2) the absence of explicit, portable representations of the latent structure learned during amortized inference. We show that PFNs do not merely memorize predictions; rather, they construct structured internal representations that encode properties of the data-generating process. We introduce a decoding framework that extracts this implicit information in the form of an explicit spectral density, from which a stationary covariance kernel is obtained via Bochner’s theorem. Our approach reframes the PFN from a black-box predictor into a zero-shot spectral inference engine . Instead of performing iterative kernel optimization at test time like DKL, the model processes a context dataset and directly outputs an explicit spectral representation in a single forward pass. This spectral representation is then deterministically converted into a stationary kernel via Bochner’s theorem. We study this process in two complementary regimes: single-function identification , where only one realization of a function is available and only partial spectral information is statistically identifiable, and multi-function identification , where multiple realizations from a shared generative process permit full recovery of the spectral density. Positioning and Contributions: We advance recent progress on PFNs ( 10 ) by asking a mechanistic question: if a PFN approximates GP inference, where and how is spectral information represented? Focusing on PFNs trained with Decoupled-Value Attention (DVA) ( 16 ) , we show that the attention mechanism induces a separation between spatial similarity and signal amplitude. This architectural property yields an identifiable spectral manifold within the latent representation. Conceptually, this induces a simple inference pipeline having contextual data set \mathcal{D}_{\mathrm{ctx}}=\{X_{ctx},Y_{ctx}\} \mathcal{D}_{\mathrm{ctx}}\;\xrightarrow[\text{Attention}]{\text{DVA}}\;H\&V\;\xrightarrow[\text{Decoders}]{\text{Proposed}}\;S(\omega)\;\xrightarrow[\text{Theorem}]{\text{Bochner}}\;k(\tau) where, value encodings is given as V , the PFN maps data to a latent representation H which encodes an implicit estimate of the spectral density S(\omega) , and the stationary kernel k(\tau) follows directly from this spectrum. Building on this observation, we develop an amortized spectral decoding framework that makes this pipeline explicit, yielding stationary surrogate kernels without iterative test-time optimization. Our main contributions are: • Mechanistic evidence of spectral encoding: Through probing and ablation analyses, we show that DVA-PFN attention representations ( H ) contains the spectral content of the input function, indicating that PFNs implicitly perform spectral estimation as part of inference. • Amortized spectral decoding across regimes: We propose a decoder that extracts explicit spectral densities from frozen PFNs 1 1 1 Pre-trained PFN with no weights update during inference. in both single-function and multi-function realization settings, accounting for the distinct statistical identifiability properties of each regime. • Fundamental limits of spectral recovery: We characterize what aspects of the spectral density are identifiable from data. We show that spectral weights are non-identifiable from a single normalized realization, but become identifiable with multiple independent realizations, and we derive an unbiased estimator to recover the missing global scale. • Empirical comparison with iterative baselines: We show that kernels constructed from decoded spectral representations via Bochner’s theorem achieve GP regression performance comparable to PFNs, Deep Kernel Learning and Random Fourier Feature methods, while avoiding iterative test-time optimization and enabling substantial inference speedups. Scope of the decoded kernel: Firstly, we are not claiming that PFNs are the only way to predict kernels. Our idea is to show that PFNs, which are becoming foundational tools for AutoML, already contain this information. Thus, the value of our method is Interpretability and Reuse, not just raw performance. Further, the kernels recovered in this work are explicit stationary surrogate kernels obtained by applying Bochner’s theorem to decoded spectral densities. They are not exact reconstructions of the PFN’s internal attention mechanism nor guaranteed to match the true data-generating GP kernel. Rather, they capture the dominant second-order statistics encoded in the PFN representation. We evaluate kernel quality by using the decoded kernels as plug-in covariances for GP regression, treating predictive performance as a proxy for the fidelity of the recovered spectral structure. 2 Background and Related Works 2.1 Prior-Data Fitted Networks with DVA Prior-Data Fitted Networks (PFNs) ( 9 ) enable amortized Bayesian inference by training a neural set-predictor on a vast distribution of synthetic datasets. Formally, given a prior p(\mathcal{D}) over supervised learning tasks, we sample datasets \mathcal{D}_{k}=\{(x_{i},y_{i})\}_{i=1}^{N}\sim p(\mathcal{D}) . Each dataset is partitioned into a context set \mathcal{D}_{\text{ctx}}=(X_{\text{ctx}},Y_{\text{ctx}}) containing observed pairs, and a query set \mathcal{D}_{\text{q}}=(X_{\text{q}},Y_{\text{q}}) containing targets to predict. The model parameters \theta are optimized to minimize the expected Negative Log-Likelihood (NLL): \mathcal{L}(\theta)=\mathbb{E}_{\mathcal{D}\sim p(\mathcal{D})}\left[-\sum_{(x,y)\in\mathcal{D}_{\text{q}}}\log q_{\theta}(y\mid x,\mathcal{D}_{\text{ctx}})\right]. (1)