Scaling Sparse Feature Circuits For Studying In-Context Learning
Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda · 2025
Read Scaling Sparse Feature Circuits For Studying In-Context Learning by Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13756.txt.
Introduction
Dmitrii Kharlapenko and co-authors probe how language models learn from examples in the prompt, in a 2025 study of model internals. They use sparse autoencoders, tools that split model activity into a large set of rare, clear features. The question is whether those features can reveal the steps behind learning from context.
The team finds features that encode which task the model should do, and shows those features can trigger the task with no examples at all. Step-by-step tracing then links early features that spot that a task was done to later ones that carry it out. The tracing method was scaled up to a much larger model than past work had handled. Results back the view that prompt learning runs through compact task signals.
Catalog record
| CREATOR | Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Sparse Feature Circuits For Studying In-Context Learning Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda Abstract Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL). We identify abstract SAE features that (i) encode the model’s knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot. This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task-execution features. To explore the ICL mechanism, we adapt the sparse feature circuits methodology of Marks et al. 2024 to work for the much larger Gemma-1 2B model, with 30 times as many parameters, and to the more complex task of ICL. Through circuit finding, we discover task-detecting features with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers. † † affiliationnotice: Equal contribution 1 Introduction Sparse autoencoders (SAEs; Ng 2011 ; Bricken et al. 2023 ; Cunningham et al. 2023 ) are a promising method for interpreting large language model (LLM) activations. However, the full potential of SAEs in interpretability research remains to be explored, since most recent SAE research either i) interprets a single SAE’s features rather than the model’s computation as a whole ( Bricken et al. 2023 ) , or ii) performs high-level interventions in the model, but does not interpret the effect on the downstream computation caused by the interventions ( Templeton et al. 2024 ) . In this work, we address these limitations by interpreting in-context learning (ICL), a widely studied phenomenon in LLMs. In summary, we show that SAEs a) enable the discovery of novel circuit components ( task-detection features ; Section 4.2 ) and b) refine existing interpretations of ICL, by e.g. decomposing task vectors ( Todd et al. 2024 ; Hendel et al. 2023 ) into task-execution features ( Section 3 ). In-context learning (ICL; Brown et al. 2020 ) is a fundamental capability of large language models that allows them to adapt to new tasks without fine-tuning. ICL is a significantly more complex and important task than other behaviors commonly studied in circuit analysis (such as IOI in Wang et al. 2022 and Kissane et al. 2024 , or subject-verb agreement and Bias-in-Bios in Marks et al. 2024 ). Recent work by ( Todd et al. 2024 ) and Hendel et al. 2023 has introduced the concept of task vectors to study ICL in a simple setting, which we follow throughout this paper. 1 1 1 Task vectors ( Hendel et al. 2023 ) are also called “function vectors” ( Todd et al. 2024 ) , but we use “task vectors” throughout this paper for consistency. In short, task vectors are internal representations of simple operations performed by language models during the processing of few-shot prompts, e.g. as “hot \rightarrow cold, big \rightarrow small, fast \rightarrow slow”. We call these simple operations tasks . Task vectors can be extracted and added into different LLM forward passes to induce 0-shot task behavior, e.g. making LLMs predict that “slow” follows “fast \rightarrow ” without explicit context. Section 2.3 provides a full introduction. Unfortunately, it is not possible to decompose task vectors by naively applying Sparse Autoencoders to reconstruct them. This is because task vectors are out-of-distribution for SAEs and empirically, their SAE decompositions tend to be cluttered with irrelevant features. To address this limitation, we developed the Task vector cleaning (TVC) algorithm ( Section 3.1 ), which enabled us to extract task-execution features : SAE latents that preserve most of the task vector’s effect on task loss while being more interpretable. These task-execution features demonstrated two key properties: they could partially substitute for complete task vectors in steering experiments, and they often exhibited clear task-related patterns in their maximum-activating tokens. Through extensive testing across diverse tasks, we validated that these features play a causal role in the model’s ICL capabilities ( Section 3.2 ). To understand how these features operate within the broader ICL mechanism, we extended the Sparse Feature Circuits (SFC) discovery algorithm introduced by Marks et al. 2024 ( Section 4 ). Applying this algorithm to the Gemma-1 2B model ( Team et al. 2024 ) revealed a network of SAE latents crucial to ICL processing. This analysis uncovered a complementary set of features we call task-detection features , which identify the specific task being performed based on earlier prompt information. Importantly, our experiments demonstrated that disabling task-detection feature directions also disabled task-execution directions, revealing their fundamental interdependence in the ICL circuit ( Section 4.2 ). Our findings not only advance our understanding of ICL mechanisms but also demonstrate the potential of SAEs as a powerful tool for interpretability research on larger language models. By unifying the task vectors view with SAEs and uncovering two of the most important causally implicated feature families behind ICL, we pave the way for future work to control and monitor ICL further, to improve either the safety or capabilities of models. Our main contributions are as follows: 1. We demonstrate that SAEs can be effectively used to analyze a complex ICL mechanism in Gemma-1 2B, which has 10 – 35 \times more parameters than prior models typically studied at this depth in comparable, end-to-end circuits-style mechanistic interpretability research ( Wang et al. 2022 ; Marks et al. 2024 ) ( Section 4.1.3 ). 2. We identify two core components of the ICL circuit : task-detection features that identify required tasks from the prompt, and task-execution features that implement those tasks during generation ( Sections 3 and 4.2 ). 3. We uncover how these components interact: attention heads and MLPs process information from task-detection features to activate the appropriate task-execution features, revealing how the model integrates information across the prompt to perform ICL tasks ( Section 4.2 ). 4. We develop the Task Vector Cleaning (TVC) algorithm, a novel transformer-specific sparse linear decomposition method that enables precise analysis of ICL mechanisms by decomposing task vectors into their most relevant features ( Section 3.1 ). Figure 1 : A diagram of the in-context learning circuit, showing task detection features (yellow) causing task execution features (blue) which cause the model to output the antonym (left \to right). A more concrete circuit, along with texts these features activate on, can be seen in Figure 12 . 2 Background 2.1 Sparse Autoencoders (SAEs) Sparse autoencoders (SAEs) are neural networks designed to learn efficient representations of data by enforcing sparsity in the hidden layer activations ( Elad 2010 ) . In the context of language model interpretability, SAEs are used to decompose the high-dimensional activations of language models into more interpretable features ( Cunningham et al. 2023 ; Bricken et al. 2023 ). The basic idea behind SAEs is to train a neural network to reconstruct its input while constraining the hidden layer to have sparse activations. This