Provable Low-Frequency Bias of In-Context Learning of Representations

Yongyi Yang, Hidenori Tanaka, Wei Hu · 2025

Read Provable Low-Frequency Bias of In-Context Learning of Representations by Yongyi Yang, Hidenori Tanaka, Wei Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13540.txt.

Introduction

Yongyi Yang, Hidenori Tanaka, and Wei Hu explain why large language models form smooth internal pictures of new tasks from examples alone. Their paper appeared in 2025. Models can pick up new behavior from a prompt, with no retraining. Past work showed their inner representations take on the shape of the task. The question was why.

The authors give the first proof-based account. They show the inner representations settle in two ways at once: across the prompt and across the model's layers. This double settling pushes the representations toward smooth, low-frequency forms. The theory explains why the learned shapes look ordered overall yet warped up close, and why their energy fades without dying out. It also predicts that the models shrug off high-frequency noise, which tests confirmed.

Catalog record

CREATORYongyi Yang, Hidenori Tanaka, Wei Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Provable Low-Frequency Bias of In-Context Learning of Representations Yongyi Yang, Hidenori Tanaka, Wei Hu Abstract In-context learning (ICL) enables large language models (LLMs) to acquire new behaviors from the input sequence alone without any parameter updates. Recent studies have shown that ICL can surpass the original meaning learned in pretraining stage through internalizing the structure the data-generating process (DGP) of the prompt into the hidden representations. However, the mechanisms by which LLMs achieve this ability is left open. In this paper, we present the first rigorous explanation of such phenomena by introducing a unified framework of double convergence, where hidden representations converge both over context and across layers. This double convergence process leads to an implicit bias towards smooth (low-frequency) representations, which we prove analytically and verify empirically. Our theory explains several open empirical observations, including why learned representations exhibit globally structured but locally distorted geometry, and why their total energy decays without vanishing. Moreover, our theory predicts that ICL has an intrinsic robustness towards high-frequency noise, which we empirically confirm. These results provide new insights into the underlying mechanisms of ICL, and a theoretical foundation to study it that hopefully extends to more general data distributions and settings. 1 Introduction It has become a major challenge for today’s machine learning community to understand how large language models (LLMs) perform in-context learning (ICL), i.e. the ability to learn patterns or tasks solely from input sequences without any gradient updates ( Brown et al. 2020 ; Min et al. 2022 ; Garg et al. 2022 ; Akyürek et al. 2022a ) . Empirical studies have demonstrated that LLMs can carry out a variety of tasks, including logical reasoning ( Wei et al. 2022 ) , programming ( Gao et al. 2023 ) , and solving mathematical problems ( Hendrycks et al. 2021 ) ; recent work also suggests that LLMs are able to stay robust against noisy prompts ( Cheng et al. 2025 ; Alazraki et al. 2025 ) . However, the mechanisms underlying these capabilities remain largely elusive. A particularly striking phenomenon, recently highlighted by Park et al. 2024 , shows that when a pre-trained LLM is fed a sequence generated by a random walk on a planar graph, with each node corresponds to a word (see “Data Generating Process” part of Figure 1 ), the model’s hidden representations converge to a state that reflects the original graph structure, even though the graph itself was never explicitly provided. We refer to this emergent behavior as In-Context Learning of Representations (ICLR). The ICLR phenomenon suggests an important mechanism of ICL: the model (in-contextly) learns to embed the information of the data-generating process (DGP) into the hidden representations. Therefore, understanding the ICLR phenomenon is a crucial step towards a deeper understanding of the mechanisms of ICL. Moreover, Park et al. 2024 also shows that an energy function decays over the course of this process, suggesting there might be an underlying principle that drives the emergence of ICLR. However, it is left open what is the nature of this principle and how does it applies to the representations. In this paper, we present the first theoretical explanation of the ICLR phenomenon, showing that it arises as a consequence of an intrinsic bias in LLMs towards low-frequency hidden representations. The central idea of our theoretical framework is a process we call Double Convergence , wherein hidden representations converge both along the context length and across layers. Specifically, the low-frequency bias emerges from the interaction of the the Context-wise Process and the Layer-wise Process . 1. Context-wise Process : If the attention map “reflects” a function that is depends only on token identities (formally defined in Definition 2 and Definition 3 ), and the representations converge to a set of latent representations as the context length increases (formally defined in Definition 1 ), then the attention effectively applies the reflected function to the latent representations. This result is formally shown in Theorem 1 ; 2. Layer-wise Process : The latent representations across layers, and eventually converge to a state that captures the distributional properties of the input sequence. Under the specific DGP considered in Park et al. 2024 , we prove that this convergence exactly yields the representations characteristic of the ICLR phenomenon. This result is formally stated in Theorem 2 . Figure 1: Overview of the DGP and the Double Convergence Process. The input sequence is generated by a random walk over a graph defined on the vocabulary (top), and then passed into a pre-trained Transformer model. Context-wise convergence occurs within each layer: token representations associated with the same word converge into tight clusters. Layer-wise convergence occurs across layers: the cluster centers gradually evolve towards a structure aligned with the underlying graph. The concept of Double Convergence is illustrated in Figure 1 . While our main analysis focuses on the specific DGP used in Park et al. 2024 , the techniques and theoretical insights we develop are able to be extended beyond this particular setting. In Appendix D , we provide a generalized framework that is decoupled from any specific DGP, highlighting the broader applicability of our results. Our theoretical framework provides a comprehensive explanation for several phenomena and open questions raised in Park et al. 2024 . Furthermore, as both a validation and application of our theoretical results, we also demonstrate that ICL exhibits implicit robustness against high-frequency noise in the input data, which is consistent with recent empirical findings ( Cheng et al. 2025 ; Alazraki et al. 2025 ) . In summary, our main contributions in this paper are as follows: 1. We identify the double convergence process, which serves as a general framework for studying the evolution of the representation in ICL ( Sections 4 and 5 ); 2. We provide theoretical explanations for several previously unexplained phenomena in Park et al. 2024 , including why ICL can suppress the original meaning of each word learned in pre-training ( Section 6.1 ), why the learned representations form an apparently regular yet slightly distorted structure ( Sections 6.2 and 6.3 ), and why the energy decreases but does not converge to zero ( Section 6.4 ); 3. Our theory highlights an implicit low-frequency bias in ICL, and predicts that LLMs are naturally robust to errors in the input prompts, which we have verified empirically ( Section 6.5 ). This paper is organized as follows. In Section 2 , we review prior work on the mechanisms of in-context learning and the structure of hidden representations, emphasizing the core challenges and introducing core ideas of our approach. In Section 3 , we define notations used and the problems considered throughout the paper, as well as outline our method. In Section 4 , we present our results on context-wise convergence, and in Section 5 , we establish the main result by combining context-wise and layer-wise convergence. In Section 6 , we discuss the theoretical implications of our framework and address specific empirical observations. Finally, we conclude in Section 7 with a summary and outlook for future work. Due to space constraints, we defer the formal proofs of our theoretical results and the empirical validation of our assumptions to the appendix. 2 Background In this

Read the complete text