Transformers are almost optimal metalearners for linear classification

Roey Magen Weizmann Institute of Science [email protected] Gal VardiWeizmann Institute of Science [email protected] · 2025

Read Transformers are almost optimal metalearners for linear classification by Roey Magen Weizmann Institute of Science [email protected] Gal VardiWeizmann Institute of Science [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19797.txt.

Introduction

Roey Magen and Gal Vardi prove that a simplified transformer, trained by gradient descent, can act as a nearly optimal metalearner for linear classification. Metalearning means learning from many related tasks so a new one needs only a handful of examples. Their tasks are Gaussian mixture classification problems whose mean vectors live in a shared low-dimensional subspace.

After training on enough tasks, the transformer generalizes to a fresh task using a number of in-context examples that depends on the subspace dimension, not the full data dimension. That nearly matches an optimal learner told the subspace outright, and it crushes any learner restricted to the in-context examples alone. The required training effort is likewise independent of the ambient dimension.

Catalog record

CREATORRoey Magen Weizmann Institute of Science [email protected] Gal VardiWeizmann Institute of Science [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19797

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformers are almost optimal metalearners for linear classification Roey Magen Weizmann Institute of Science [email protected] Gal VardiWeizmann Institute of Science [email protected] Abstract Transformers have demonstrated impressive in-context learning (ICL) capabilities, raising the question of whether they can serve as metalearners that adapt to new tasks using only a small number of in-context examples, without any further training. While recent theoretical work has studied transformers’ ability to perform ICL, most of these analyses do not address the formal metalearning setting, where the objective is to solve a collection of related tasks more efficiently than would be possible by solving each task individually. In this paper, we provide the first theoretical analysis showing that a simplified transformer architecture trained via gradient descent can act as a near-optimal metalearner in a linear classification setting. We consider a natural family of tasks where each task corresponds to a class-conditional Gaussian mixture model, with the mean vectors lying in a shared k k -dimensional subspace of ℝ d \mathbb{R}^{d} . After training on a sufficient number of such tasks, we show that the transformer can generalize to a new task using only O ~ ​ ( k / R ~ 4 ) \widetilde{O}(k/\widetilde{R}^{4}) in-context examples, where R ~ \widetilde{R} denotes the signal strength at test time. This performance (almost) matches that of an optimal learner that knows exactly the shared subspace and significantly outperforms any learner that only has access to the in-context data, which requires Ω ⁡ ( d / R ~ 4 ) \Omega(d/\widetilde{R}^{4}) examples to generalize. Importantly, our bounds on the number of training tasks and examples per task needed to achieve this result are independent of the ambient dimension d d . 1 Introduction Transformer-based models are the dominant architecture in both natural language processing (NLP) and computer vision. Since their introduction by Vaswani et al. 2017 , transformers have been scaled to produce remarkable advances in language modeling [ 2 ] , image classification [ 3 ] , and multimodal learning [ 4 ] . Their strength lies in their ability to model complex dependencies through attention mechanisms and to generalize across diverse tasks with minimal task-specific supervision. One of the most intriguing emergent capabilities of large transformer models is in-context learning (ICL). In ICL, a model is given a short sequence of input-output pairs (called a prompt ) from a particular (possibly new) task, and is asked to make predictions on test examples from that task without any explicit parameter updates. This ability to rapidly adapt to new tasks from a small number of examples, solely by conditioning on the prompt, has been observed in large language models [ 5 ] , and is central to the ongoing shift toward prompt-based learning paradigms. The ICL phenomenon is closely connected to the broader framework of metalearning , or “learning to learn” [ 6 , 7 , 8 ] , which has been widely studied before. In metalearning, a learner is trained to perform well across a distribution of related tasks, thereby acquiring representations that allow for rapid adaptation to new tasks. It is often helpful to think of tasks as corresponding to individual users. For instance, generating personalized email completions. While each user provides limited task-specific data, such as writing style and personal preferences, there is a rich shared structure across users that can be exploited. Metalearning aims to leverage this structure to improve performance on each task beyond what would be possible if learned independently. In this paper, we consider binary classification, where we model each task as a distribution \mathcal{D} over labeled examples ({\bm{x}},y) in \mathbb{R}^{d}\times\{\pm 1\} . During training, the learner has access to B datasets, where the j -th dataset consists of N samples drawn i.i.d from task \mathcal{D}_{j} . We further assume that each task labeled by a classifier that relies on a common representation h:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k} , where typically k is smaller than the ambient dimension d . For each task \mathcal{D} , there is a classifier f_{\mathcal{D}}:\mathbb{R}^{k}\rightarrow\{\pm 1\} such that f_{\mathcal{D}}\circ h has high accuracy on \mathcal{D} . Our interest lies in studying families of tasks for which knowing this shared representation h substantially reduces the number of samples required to learn each task separately. Metalearning Objective. Assuming that the tasks \mathcal{D}_{1},\dots,\mathcal{D}_{B} are themselves drawn i.i.d. from an unknown metadistribution \Omega , the goal is to output a representation \hat{h} that can be specialized to a new unseen task \mathcal{D}\sim\textit{$\Omega$} . In the modern view, the model is first trained on tasks \mathcal{D}_{1},\dots,\mathcal{D}_{B} , each task contain N samples. Then, at test time, the learner is given M in-context labeled samples drawn i.i.d. from some new task \mathcal{D} , and needs to classify a new sample from \mathcal{D} , without further training. To evaluate performance, mostly the required number of in-context labeled samples M that required to ensure (with high probability) small error, we consider two benchmark baselines: • Single-task optimal learner: An optimal algorithm, in terms of the number of required samples, that has access only to samples from the new task \mathcal{D} . • Optimal learner with access to the ground-truth representation: An optimal algorithm that has access both to samples of the new task and to the ground-truth representation h . The above discussion naturally motivates the study of transformers from the lens of metalearning. In particular, we are interested in understanding whether, and under what conditions, transformer architectures can act as (optimal) metalearners . Specifically, Can a transformer outperform a single-task optimal learner (that only have access to the in-context data), and potentially approach the performance of an optimal learner with access to the ground-truth representation ? 1.1 Our contributation To address the question of whether transformers can serve as effective metalearners , we analyze their behavior in the well-studied Gaussian mixture framework. In this setting, each task is a random instance of a class-conditional Gaussian mixture model in \mathbb{R}^{d} , with identical spherical covariance and opposite means. The task-specific signal vector {\bm{\mu}} is sampled from a shared low-dimensional subspace of dimension k\leq d . Formally, for some semi-orthogonal matrix {\bm{P}}\in\mathbb{R}^{d\times k} and signal strength R>0 , each task \tau=1,\dots,B is defined by: \displaystyle{\bm{\mu}}_{\tau}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}{\bm{P}}\cdot\mathsf{Unif}(R\cdot{\mathbb{S}}^{k-1}), \displaystyle y_{\tau,i}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathsf{Unif}(\{\pm 1\}),\ \ \ \ \ \displaystyle{\bm{z}}_{\tau,i}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathsf{N}({\mathbf{0}},{\bm{I}}_{d}), \displaystyle{\bm{x}}_{\tau,i}=y_{\tau,i}{\bm{\mu}}_{\tau}+{\bm{z}}_{\tau_{i}}. At test time, in-context examples are also drawn from a class-conditional Gaussian mixture model, but potentially with a different signal-to-noise ratio determined by a test-time signal strength \tilde{R} . That is, the meta-distribution may shift between training and testing. We consider a simplified transformer with linear attention, a setup similar to many prior works on ICL [ 9 , 10 , 11 , 12 ] . The transformer is trained via gradient descent (GD) on the logistic or exponential loss over the above random linear classification tasks. Our main contributions are as follows: • We prove that if the transformer is trained on a sufficiently large number of tasks, then the number of in-context samples required at test time to achieve a small constant error on a new task (without parameters update) is \tilde{O}(k/\tilde{R}^{4}) , where \tilde{R} denote the test-time signal strength. In contrast, any single-task learner, which has access only to samples from the new task, require at least \Omega(d/\tilde{R}^{4}) samples (see our discussion in Remark

Read the complete text