Analyzing limits for in-context learning

Omar Naim, Nicholas Asher · 2025

Read Analyzing limits for in-context learning by Omar Naim, Nicholas Asher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03503.txt.

Introduction

A popular claim says that when a large language model learns from examples inside a prompt, it quietly runs a classic learning algorithm under the hood. Omar Naim and co-authors challenge that claim in a 2025 paper. They present test results that do not fit the claimed picture, then give a mathematical argument about what these models can and cannot do.

Their analysis concludes that the model's structure itself sets a ceiling: it cannot reach general prediction accuracy in this setting, no matter how the examples are chosen. The authors argue the field should rethink how in-context learning truly works.

Catalog record

CREATOROmar Naim, Nicholas Asher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03503

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analyzing limits for in-context learning Omar Naim, Nicholas Asher Abstract We examine limits of in-context learning (ICL) in transformer models trained from scratch, focusing on function approximation tasks as a controlled setting to uncover fundamental behaviors. While we show empirically that transformer models can generalize, approximating unseen classes of polynomial (non linear) functions, they cannot generalize beyond certain values. We provide both empirical and mathematical arguments explaining that these limitations stem from architectural components, namely layer normalization and the attention scoring function, softmax. Together, our findings reveal structural constraints on ICL that are often masked in more complex NLP tasks but that need to be understood to improve robustness and interpretability in transformer-based models. 1 Introduction In-context learning (ICL) Brown et al. 2020 is an emergent capability of large language models (LLMs), enabling them to learn a task from an instructional prompt and a few examples at inference time, without any modification of the model’s parameters from pretraining. Thus we can use LLMs without the computational and resource-intensive demands of pretraining or fine-tuning, challenging conventional distinctions between training and inference. ICL’s effectiveness has led to the adoption of LLMs in many applications. While theoretical research has focused on what these models can learn in context, little attention has been devoted to understanding the inherent limitations of ICL. We study here ICL’s structural constraints and failure modes. Understanding ICL mechanisms, which involve prompting and pretraining, requires precise control of experimental conditions. We study ICL in a well-defined setting using clean, structured data: our task is to in-context learn polynomial functions. These mathematical problems are well-suited for analyzing ICL due to their structured and predictable nature. We explore the ICL task on functions in \mathbb{R}^{n}[X] for 1\leq n\leq 6 , using a variety of training and testing distributions and a variety of transformer models. We examine the impact of architectural components such as feedforward layers and normalization, and conduct ablation experiments to understand the role of specific components in ICL. Our main results and organization are as follows. After related work and background sections, Section 4 details the experimental set up and shows that small transformer models with attention only architectures are both necessary and sufficient for ICL of several classes of polynomial functions, even classes whose class forms they did not encounter in training, as well as some continuous functions. This shows a generalization capacity across function classes, and the primacy of attention in our task reflects prior work on the importance of attention layers in NLP tasks. However, Section 5 shows that our models fail to generalize their predictions for all functions f and inputs x , where x,f(x) lie outside the training distributions. Section 6 provides a mathematical expression of the ICL problem and gives a derives generalization failures from normalization and scoring functions. This highlights a fundamental dilemma. On one hand, the normalization layer restricts the model’s ability to generalize beyond certain input ranges. On the other hand, while removing normalization unexpectedly improves performance in our setting, if the query input has a high embedding norm, the model’s output diverges toward a linear function determined mainly by its parameters fixed in pretraining, effectively precluding ICL. This reveals a distinct failure mode of ICL in the high-norm regime, both in the presence, and absence, of normalization. But the attention scoring function itself also introduces structural limitations that hinder the model’s ability to leverage contextual information. Taken together, these results suggest that generalization failures in ICL stem from a transformer’s basic architectural design. To alleviate some of these problems, we suggest constraining embedding functions to avoid high-norm vectors and reduce variability in weight magnitudes. Focusing on well-defined and structured mathematical problems with deterministic answers enables us to avoid many subtleties of prompt engineering and of managing the large scale pretraining involved in language tasks. We can more easily and confidently measure a model’s ability to generalize, recognize patterns and reason from a few examples. The consistent patterns within polynomial functions, such as the relationship between degree and curvature for instance, make them ideal for testing whether models are truly grasping mathematical relationships rather than just memorizing specific instances. Our findings are thus applicable to ICL tasks in general. We briefly discuss consequences for NLP in the last section. 2 Related Work Brown et al. 2020 introduced in-context learning (ICL) as a framework that occurs entirely at inference time; LLMs learn tasks by analogy based on examples presented in the prompt without updating their parameters. Despite its promise, the underlying mechanisms behind ICL remain only partially understood. Akyürek et al. 2022 ; Von Oswald et al. 2023 ; Fu et al. 2023 ; Xie et al. 2021 ; Wu et al. 2023 ; Zhang et al. 2023 ; Panwar et al. 2023 have suggested that LLMs perform implicit gradient-based updates, higher-order optimization, or approximate Bayesian inference when prompted with in-context examples. However, these remain speculative hypotheses grounded in what the architecture could, in principle, implement, rather than direct evidence of the mechanisms at play. As noted by Dong et al. 2022 , much of this analysis remains confined to simple tasks, such as linear regression or Boolean functions Bhattamishra et al. 2023 . Garg et al. 2022 showed that small transformers trained from scratch on synthetic data can acquire ICL abilities, providing early evidence that ICL is not solely a consequence of model size pretraining on language, but a trainable capability. Similarly, Raventós et al. 2024 studied how ICL performance changes as a function of the number of pretraining examples. Olsson et al. 2022 investigated architectural components responsible for ICL, proposing that induction heads , a learned copying and comparison mechanism, underlie ICL. Geva et al. 2021 and Bietti et al. 2024 explored the role of memory in transformers, showing that models heavily rely on memorization via attention matrices. This is further supported by findings from Yu et al. 2023 and Geva et al. 2023 , who argue that transformers prioritize memorized patterns during inference. Naim and Asher 2024 showed that transformers trained from scratch on linear functions fail to extrapolate, when evaluated out of distribution: performance degrades from accurate prediction, to constant outputs, to near-random behavior, introducing the notion of boundary values . 3 Training for In-context learning 3.1 Learning in-context Learning a task in-context means that the model is able to predict the output for a new input using only a few input-output examples provided within the same sequence. Specifically, given a prompt of few input-output examples of the form (x_{1},g(x_{1}),...x_{p},g(x_{p}),x) , a transformer is able to approximate the value of g(x) , regardless of which specific input samples x_{i} are included in the in-context examples. We distinguish two notions of "learning a a class of functions in context". The first, call it ICL 1 , states that a transformer model \hat{f}^{\theta} can ICL a function class \cal F on a given distribution sampling functions g\in D_{\cal F} and a distribution matching inputs x_{i}\in D_{\cal I} where both correspond to training distribution: \forall x_{i},y_{i},x\in D_{\cal I}~\forall g\in D_{\cal F}, (1) ||\hat{f}^{\theta}({x_{1},g(x_{1}),...x_{p},g(x_{p})},x)-g(x)||^{2}\approx ||\hat{f}^{\theta}({y_{1},g(y_{1}),...y_{p},g(y_{p})},x)-g(x)||^{2}<\epsilon A second definition of ICL is more restrictive: a model M can ICL 2 a function class \cal F if it is capable of approximating all functions

Read the complete text