Continuum Transformers Perform In-Context Learning by Operator Gradient Descent

Abhiti Mishra *, Yash Patel, Ambuj Tewari · 2025

Read Continuum Transformers Perform In-Context Learning by Operator Gradient Descent by Abhiti Mishra *, Yash Patel, Ambuj Tewari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17838.txt.

Introduction

Abhiti Mishra, Yash Patel, and Ambuj Tewari prove that function-space transformers learn by gradient descent in context. Their 2025 paper extends a known result beyond finite inputs. Ordinary transformers were shown to run descent steps inside their forward pass. Continuum versions eat whole functions, as PDE solvers do.

The proof works in a reproducing kernel Hilbert space of operators. A generalized representer result and gradient flow math carry the load. In the deep limit, the in-context answer equals the Bayes optimal predictor. Experiments check the claim and recover the descent settings used. Prompt-style learning for equations now has a firm footing. Theory finally matches what the models plainly do.

Catalog record

CREATORAbhiti Mishra *, Yash Patel, Ambuj Tewari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continuum Transformers Perform In-Context Learning by Operator Gradient Descent Abhiti Mishra *, Yash Patel, Ambuj Tewari Abstract Transformers robustly exhibit the ability to perform in-context learning, whereby their predictive accuracy on a task can increase not by parameter updates but merely with the placement of training samples in their context windows. Recent works have shown that transformers achieve this by implementing gradient descent in their forward passes. Such results, however, are restricted to standard transformer architectures, which handle finite-dimensional inputs. In the space of PDE surrogate modeling, a generalization of transformers to handle infinite-dimensional function inputs, known as “continuum transformers,” has been proposed and similarly observed to exhibit in-context learning. Despite impressive empirical performance, such in-context learning has yet to be theoretically characterized. We herein demonstrate that continuum transformers perform in-context operator learning by performing gradient descent in an operator RKHS. We demonstrate this using novel proof strategies that leverage a generalized representer theorem for Hilbert spaces and gradient flows over the space of functionals on a Hilbert space. We further show the operator learned in context is the Bayes Optimal Predictor in the infinite depth limit of the transformer. We then provide empirical validations of this result and demonstrate that the parameters under which such gradient descent is performed are recovered through pre-training. 1 Introduction LLMs, and transformers more broadly, have demonstrated a remarkable ability to perform in-context learning, in which predictive performance improves without the need for parameter updates, merely by providing training samples in the LLM context window [ 1 , 2 ] . With workflows increasingly relying on such fine-tuning in place of traditional weight updates due to its computational efficiency, much interest has gone into providing a theoretical explanation of this phenomenon [ 3 , 4 , 5 ] . Such works have demonstrated that, with particular choices of transformer parameters, an inference pass through such models is equivalent to taking steps of gradient descent for the in-context learning task. While LLMs were the first setting in which in-context learning abilities of transformers were exploited, interest is increasing in its broader application to other sequence prediction tasks. In particular, a seemingly orthogonal subfield of machine learning is that of accelerating the solution of partial differential equations (PDEs) [ 6 , 7 , 8 , 9 , 10 , 11 ] . In this setting, the sequence is no longer of finite-dimensional vectors, but instead of infinite-dimensional functions. Nonetheless, transformers have been adapted to this setting, with an architecture known as “continuum transformers” [ 12 ] . More surprising still is that such transformers continue to exhibit in-context learning, whereby related PDEs can be efficiently solved with the similar placement of solution pairs in the continuum transformer context window [ 13 , 14 , 15 ] . Unlike in the finite-dimensional setting, this generalized, functional in-context learning yet remains to be theoretically characterized. We, therefore, herein extend this line of theoretical inquiry to characterize the continuum transformers that have been leveraged for in-context operator learning [ 12 ] . Our contributions, therefore, are twofold: the insights afforded by such theoretical analysis and the development of a mathematical framework for doing such analysis, as we highlight in greater detail in the main text. In particular, our contributions are as follows: • Proving continuum transformers perform in-context operator learning by performing gradient descent in a Reproducing Kernel Hilbert Space of operators and that the resulting in-context predictor recovers the Bayes Optimal Predictor under well-specified parameter choices of the transformer. Such proofs required the modeling of a generalized continuum attention mechanism and subsequent novel usages of a generalized representer theorem for Hilbert spaces and Gaussian measures over Hilbert spaces. • Proving that such parameters under which continuum transformers implement operator gradient descent are minimizers of the training process of such transformers, leveraging a novel gradient flow analysis over the space of functionals on a Hilbert space. • Empirically validating that continuum transformers perform in-context operator gradient descent upon inference with the exhibited parameters and that such parameters are recovered with transformer training across a diverse selection of operator RKHSs. 2 Background 2.1 Neural Operators Neural operator methods, while more broadly applicable, most often seek to amortize the solution of a spatiotemporal PDE. Such PDEs are formally described by spatial fields that evolve over time, namely some u:\Omega\times[0,\infty)\rightarrow\mathbb{R} , where x\in\Omega\subset\mathbb{R}^{d} are spatial coordinates and t\in[0,\infty) is a time coordinate. While the more abstract operator learning framework can be formulated as seeking to learn a map \widehat{\mathcal{G}}:\mathcal{A}\rightarrow\mathcal{U} between two function spaces \mathcal{A} and \mathcal{U} , we are most often interested in learning time-rollout maps, in which the input and output function spaces are identical. In such cases, it is assumed a dataset of the form \mathcal{D}:=\{(u^{(0)}_{i},u^{(T)}_{i})\} is available, where u^{(0)}:\Omega\rightarrow\mathbb{R} is the initial condition, for which there exists some true operator \mathcal{G} such that, for all i , u_{i}^{(T)}=\mathcal{G}(u_{i}^{(0)}) . While many different learning-based approaches have been proposed to solve this operator learning problem, they all can be abstractly framed as seeking \min_{\widehat{\mathcal{G}}}||\widehat{\mathcal{G}}-\mathcal{G}||^{2}_{\mathcal{L}^{2}(\mathcal{U},\mathcal{U})}=\int_{\mathcal{U}}||\widehat{\mathcal{G}}(u_{0})-\mathcal{G}(u_{0})||_{\mathcal{U}}^{2}\,du_{0}. (1) Within this broad family of operator learning, the most widely employed classes are the Deep Operator Networks (DeepONets) [ 16 , 17 , 18 , 19 ] and Fourier Neural Operators (FNOs) [ 20 , 21 , 6 , 22 ] . FNOs are parameterized as a sequence of layers of linear operators, given as kernel integral transforms, with standard intermediate ReLU nonlinearities. Formally, a single layer is then given by \sigma(\mathcal{F}^{-1}(R_{\ell}\odot\mathcal{F}(u))) , where \mathcal{F} denotes the Fourier transform and R_{\ell} the learnable kernel parameter. This layer encodes a translationally invariant kernel integral transform. 2.2 In-Context Learning and Continuum Transformers The standard attention mechanism is parameterized by \theta:=\{W_{k},W_{q},W_{v}\} , where W_{k}\in\mathbb{R}^{d_{k}\times d} , W_{q}\in\mathbb{R}^{d_{q}\times d} , and W_{v}\in\mathbb{R}^{d_{v}\times d} for sequential data X\in\mathbb{R}^{d\times n} [ 23 ] . Formally, \mathrm{Attn}(X):=(W_{v}X)MH\left((W_{q}X),(W_{k}X)\right), (2) where M\in\mathbb{R}^{n\times n} is a masking matrix and H:\mathbb{R}^{d_{q}\times n}\times\mathbb{R}^{d_{k}\times n}\rightarrow\mathbb{R}^{n\times n} is a nonlinear transform of key-query similarity measures. This resulting matrix H(Q,K) is referred to as the “attention weights” matrix. Most often, d_{k}=d_{q} and H:=\mathrm{softmax} , i.e. [H(Q,K)]_{i,j}=\exp(Q^{\top}_{i}K_{j})/\sum_{\ell}\exp(Q^{\top}_{i}K_{\ell}) [ 24 ] . Transformers are repeated compositions of such attention blocks with residual connections and feedforward and normalization layers. We now highlight the “continuum attention” mechanism proposed in [ 12 ] that generalizes transformers to operator learning; “continuum” here highlights that such an architecture models data in its discretization-agnostic function form as with other neural operator methods. In particular, in place of x_{i}\in\mathbb{R}^{d} , x_{i}\in\mathcal{X} for some Hilbert space \mathcal{X} . The natural generalization of the attention mechanism for function inputs then replaces the W_{k},W_{q} , and W_{v} matrices with linear operators. For a sequence x\in\mathcal{X}^{n} , the key, query, and value operators respectively map \mathcal{W}_{k}:\mathcal{X}^{n}\rightarrow\mathcal{K}^{n} , \mathcal{W}_{q}:\mathcal{X}^{n}\rightarrow\mathcal{Q}^{n} , and \mathcal{W}_{v}:\mathcal{X}^{n}\rightarrow\mathcal{V}^{n} , where \mathcal{K},\mathcal{Q}, and \mathcal{V} are Hilbert spaces. Notably, the original framing of continuum transformers [ 12 ] generalizes attention by considering an infinite index set, rather than considering infinite-dimensional tokens as we do here; we discuss how these two seemingly distinct notions are equivalent in Appendix A . In practice, such operators are implemented as kernel integral transforms, as \mathcal{W}_{q}x_{i}=\mathcal{F}^{-1}(R_{q}\odot\mathcal{F}x_{i}), where R_{q} is the Fourier parameterization of the query kernel, with R_{k} and R_{v} similarly defined for the key and value kernels.

Read the complete text