Is In-Context Universality Enough? MLPs are Also Universal In-Context Anastasis Kratsios, Takashi Furuya Abstract The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over 𝒳 ⊆ ℝ d \mathcal{X}\subseteq\mathbb{R}^{d} ) and a query x ∈ 𝒳 x\in\mathcal{X} . This raises the question: Does in-context universality explain their advantage over classical models ? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context . This suggests the transformer’s success is likely due to other factors like inductive bias or training stability. \coltauthor The Vector Institute and McMaster University and Shimane University 1 Introduction The undeniable success of transformers is attributed to their ability to learn in-context , unlike traditional multilayer perceptions (MLPs). This means that transformers can process sequences of tokens and predict the next relevant token. They often exhibit in-context learning (ICL) when trained on large, diverse datasets. This means that given a short sequence of input-output pairs (a prompt) from a specific task, the model can make predictions on new examples without updating its parameters. There are three primary pillars in which ICL can be studied: approximation theoretic, statistical, and optimization lenses. We focus on the former of these lenses by contrasting the approximation capacity of the transformer with the classical MLP model by asking “Does the transformer have an approximation-theoretic advantage over the MLP?” (Q) Due to the work of hornik1989multilayer ; yarotsky18a_verydeep_COLT ; suzukiadaptivity ; bolcskei2019optimal ; kidger2020universal ; kratsios2022universal ; ZuoweiHaizhaoZhang_2022_JMPA , and several others, it is by now well-known that MLPs are universal approximators in the classical “out-of-context” sense; i.e., meaning that any continuous functions from \mathbb{R}^{d} to \mathbb{R}^{D} can be uniformly approximated on compact sets to arbitrary precision by MLPs with enough neurons. These classical “out-of-context” universal approximation guarantees have since been established for the transformer model kim2024transformers ; fang2022attention with matching optimal rates, implying that the transformer is at least as expressive as the MLP “out-of-context”. More recently petrov2024universal showed that recurrent transformers can approximate any function in-context by leveraging prompt engineering. It was subsequently established by furuya2024transformers that transformers are universal approximators in-context , meaning that they can approximate any function which continuously maps context and queries to predictions uniformly on compact sets to arbitrary precision; again given enough neurons. These results suggest that the transformer may indeed have an advantage in expressivity over the vintage MLP model, hinting that ( Q ) can be hoped to be answered positively since the latter is currently not known to be universal in-context. Our paper is in the negative direction of ( Q ). Our main result ( Theorem ) matches the in-context universality of the MLP model in the setting of permutation invariant contexts (PICs) of castin2024smooth ; furuya2024transformers . We conclude that if the transformer’s superior empirical performance over the MLP model cannot be explained by in-context universality . This suggests that the empirically well-documented advantage of transformers over MLPs must stem from a statistical or optimization phenomenon unique to transformers rather than from in-context universality. Our main result is complemented by Corollary , which a quantitative version of furuya2024transformers , showing that for any target function, compact set of PICs, and approximation error, a transformer with multiple attention heads per block. Thus, the in-context approximation power of the transformer seems to match that of the MLP. 1.1 Secondary Contributions Our second main result (Corollary ) is deduced from our main result for ReLU MLPs, using our transformerification procedure (Proposition ), which converts any MLP to a multi-head transformer which exactly preserves its depth and width while only doubling its trainable (non-zero) parameters (thus has the same order of trainable parameters) and with a fixed number of attention heads per block. This type of “conversion” map was also recently obtained to prove quantitative universal approximation guarantees for convolutional neural networks in petersen2020equivalence and spiking neural networks in singh2023expressivity . A key step in our analysis shows that ReLU MLPs can exactly implement the 1 -Wasserstein distance on a broad class of finite probability measures, containing all empirical measures (Proposition ). This auxiliary result can be of independent interest to the neural optimal transport community, e.g. korotin2019wasserstein ; korotinneural ; gazdieva2024robust . 1.2 Related Literature Permutation-Invariant Contexts and their Geometry Transformer models generally cannot inherently detect the order of input tokens without using positional encodings vaswani2017attention ; chuconditional ; li2021learnable . These encodings capture the sequence in which tokens appear, addressing the transformer’s invariance to row permutations in the input context matrix. Since we are studying the transformer architecture and not a specific positional encoding scheme; then, we will adopt the permutation-invariant formulation of context of castin2024smooth ; furuya2024transformers , rather than the sequential formulation of garg2022can ; zhang2024trained ; akyurek2022learning ; von2023transformers , which does not capture the transformer’s permutation invariance. In this paper, we work with a refined version of the permutation-invariant setting introduced by furuya2024transformers . Their framework formalizes contexts as probability measures on a space of tokens within a dictionary \mathcal{X}\subseteq\mathbb{R}^{n} . The motivation is that any finite set of tokens x_{1},\dots,x_{N}\in\mathcal{X} can be represented as an empirical measure \mu=\sum_{n=1}^{N}w_{n}\delta_{x_{n}} , where 0\leq w_{1},\dots,w_{N}\leq 1 (summing to 1 ), reflecting the relative frequency of each token in the permutation-invariant context (PIC) \mu , which is unaffected by the indexing of these tokens. While their permutation-invariant setting is well-suited for real-world transformers, it overlooks the fact that real-world context windows ( C ), even when tokens are drawn from an infinite dictionary \mathcal{X} , still impose constraints. Our analysis rests on a quantitative refinement of the mathematical idealization in the setting of furuya2024transformers , which assumed that context can be arbitrarily large since any real-world LLM has a finite context window. Instead, our analysis operates on the realistic subspace \mathcal{P}_{C,N}(\mathcal{X}) (formalized in Section 2.1 ) comprised of probability measures of the form \sum_{n=1}^{N}\,w_{n}\delta_{x_{n}} where the weights \{w_{n}\}_{n=1}^{N} are all positive and divisible by the context window C . Thus, each distinctly observed token x_{n} cannot be observed more times than the context window C allows. I.e. we prohibit mathematical artifacts such as a relative frequency of 1/\sqrt{2} . Under this realism restriction, we can identify any PIC in \mu\in\mathcal{P}_{C,N}(\mathbb{R}^{d}) an equivalence class X^{\mu} of N\times(d+1) real-matrix \mu=\sum_{n=1}^{N}\,w_{n}\delta_{x_{n}}\leftrightarrow X^{\mu}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\big[(x_{n},w_{n})_{n=1}^{N}\big] (1) here, [A] denotes the equivalence class of matrices formed by permuting the rows of the N\times(d+1) matrix A . We equip \mathcal{P}_{C,N}(\mathbb{R}^{d}) with the 1 -Wasserstein metric, which, when restricted to the space of empirical measures \mathcal{P}_{1,N}(\mathbb{R}^{d}) , is shown to be equivalent to the natural quotient metric (see BridsonHaefliger_1999NPCBook ) on the corresponding subspace of matrices, quotiented by row permutations. Details are provided in Section 2.1 . Approximation Guarantees We highlight that there are alternative, weaker, approximation guarantees for transformers than the one in furuya2023globally , some of which consider contexts; e.g. petrov2024universal , without permutation-invariance, and some which are context-free; e.g. kim2024transformers ; fang2022attention . The conclusion remains the same when juxtaposed against Theorem ; in-context universality is not enough to explain the advantage of the transformer model over the classical MLP model. Although not directly related to our findings, there is a body of work that examines the expressivity of transformers when applied to discrete token sets as formal systems chiang2023tighter ; merrill2023expresssive ; strobl2024formal ; olsson2022context . Another line of research also explores how positional encoding affects transformer expressivity luo2022your . Nevertheless, the transformer’s advantage over the classical MLP model is not identified. We also mention results explaining the ability of transformers to represent specific structures, such as Kalman filtering updates relying on a Nadaraya–Watson kernel density estimator goel2024can and its ability to satisfy certain constraints zamanlooy2021trans . Statistical Guarantees We also mention the growing body of statistical guarantees for in-context learners, such as transformers. These study the training dynamics of transformers with linear attention mechanisms chen2024training ; lu2024asymptotic ; zhang2024context ; siyu2024training ; kim2024transformersdyna and non-asymptotic single-step variants thereof duraisamy2024finite , the minimax statistical optimality of pre-trained transformers trained in-context kim2024transformers , guarantees for transformers trained with non-i.i.d. time-series data limmer2024reality , PAC-Bayesian guarantees for transformers mitarchuk2024length , their infinite-width limits zhang2024trained in an NTK fashion jacot2018neural , statistical guarantees for in-context classification using transformers reddy2023mechanistic , and several other works examining the statistical foundations of in-context learning; e.g. akyurek2022learning ; li2023transformers ; bai2024transformers . We mention the recent line of work studying the efficiency of trains of thoughts generated by transformers trained in context kim2024transformersTOT . 2 Preliminaries This section contains the necessary background for the formulation of our main results. 2.1 Permutation-Invariant In-Context Learners We first review and add to the notions of permutation invariant in context-learners considered in furuya2024transformers . This relies on the introduction of some tools from probability theory, specifically from optimal transport and their refinements use ideas from metric geometry ; both of which are introduced now. Permutation-Invariant Context via Probability Measures Given a (non-empty) measurable subset {\mathcal{X}} of \mathbb{R}^{d} , for some d\in\mathbb{N}_{+} , we use \mathcal{P}({\mathcal{X}}) to denote the set of all Borel probability measures on {\mathcal{X}} which is then equipped with the topology of weak convergence of measures. When 1\leq p<\infty , we denote by \mathcal{P}_{p}({\mathcal{X}}) the subset of probabilities that finitely integrate x\mapsto\|x-x_{0}\|^{p} for some (and thus for any) x_{0}\in{\mathcal{X}} . Similarly, we equip \mathcal{P}_{p}({\mathcal{X}}) with the Wasserstein p -distance \mathcal{W}_{p} , that is, for \mu,\nu\in\mathcal{P}_{p}({\mathcal{X}}) , the metric defined by \mathcal{W}_{p}(\mu,\nu)^{p}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\inf_{\pi\in\operatorname{Cpl}(\mu,\nu)}\int d_{\mathcal{X}}(x,y)^{p}\,\pi(dx,dy), where \Pi(\mu,\nu)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\{\pi\in\mathcal{P}({\mathcal{X}}\times{\mathcal{X}})\colon\pi\text{ has first marginal }\mu,\text{ second marginal }\nu\} . Elements of \Pi(\mu,\nu) are called couplings, or transport plans, between the measures \mu and \nu . When p=1 the Kantorovich-Rubinstein duality allows us to re-express and extend the definition of \mathcal{W}_{1} to the class of finite signed (Borel) measures on \mathcal{X} via \mathcal{W}(\mu,\nu)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\sup_{f\in\operatorname{Lip}(\mathcal{X},1)}\int f(x)d(\mu-\nu)(x) where \operatorname{Lip}(\mathcal{X},1) denotes the set of 1 -Lipschitz functions f on \mathcal{X} with \|f\|_{\infty}+L_{f}\leq 1 , where L_{f} is the optimal Lipschitz constant of f and \|f\|_{\infty}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\sup_{x\in\mathcal{X}}\,|f(x)| . This extension to signed finite measures is typical in the non-linear theory of Banach space theory godefroy2015survey ; weaver2018lipschitz ; ambrosio2020linear and has applications in deep learning von2004distance ; kratsios2023approximation ; cuchiero2023global .