Localizing RL-Induced Tool Use to a Single Crosscoder Feature
Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman · 2026
Read Localizing RL-Induced Tool Use to a Single Crosscoder Feature by Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26474.txt.
Introduction
Andrii Shportko and five colleagues ask where a learned behaviour lives inside a language model. Reinforcement learning fine-tuning teaches models to call external tools correctly. Which internal features carry that new skill was unknown. The team used a tool called a crosscoder. It separates features shared by two models from features unique to one of them.
Their dedicated feature crosscoders isolated a small set of features created by the reinforcement learning. The work used the Qwen2.5-3B model. Across a wide sweep of settings, rebuilding the tuned model from those features improved its tool correctness greatly. Passing the features to the frozen base model also transferred some tool skill. The authors call that transfer a capability spillover. Their findings suggest the new skill sits in a compact, steerable feature set. That set could allow behaviour control at run time, without retraining.
Catalog record
| CREATOR | Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Localizing RL-Induced Tool Use to a Single Crosscoder Feature Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman Abstract Fine-tuning through RL reshapes the internal representations of language models to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that Dedicated Feature Crosscoders (DFC) isolate a compact set of RL-specific features that mediate tool-calling capability 1 1 1 see Limitations (section 7 ) in Qwen2.5-3B . Across a 48 48 -crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model’s tool correctness by + 31.1 ± 9.7 +31.1\pm{9.7} pp and passively transfers tool-calling ability to the frozen base model by + 6.8 ± 5.0 +6.8\pm 5.0 pp which we call a capability spillover . Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs. † † affiliationnotice: Equal contribution 1 Introduction Large language models are increasingly trained to perform agentic tasks such as invoking external tools and interacting with external systems ( 10 ) . RL enables these behaviors, yet how RL fine-tuning alters a model’s internal representations is not well understood. Mechanistic interpretability has introduced sparse autoencoders (SAEs) ( 3 ; 4 ; 14 ) to decompose model activations into interpretable features, and Crosscoders ( 9 ) extend this shared decomposition across two related models. Dedicated Feature Crosscoders ( 6 ) further partition the joint dictionary into A -exclusive, B -exclusive, and shared sub-dictionaries with gradient masking in order to enforce exclusivity. We test whether capability differences introduced by RL concentrate in the A -exclusive partition as intended and whether those features can be steered for retraining-free behavioral control. We make the following contributions: 1. We train and evaluate 48 crosscoder variants across a systematic hyperparameter sweep on a ToolRL-fine-tuned Qwen2.5-3B pair ( \lx@sectionsign\ref{sec:hyperparam} ). 2. We identify capability spillover : passing the frozen base model’s activations through a jointly-trained crosscoder passively transfers +6.8 pp tool-correctness with zero fine-tuning ( \lx@sectionsign\ref{sec:spillover} ). 3. We show that the DFC exclusive partition acts as a filter , concentrating the strongest model-specific features but not fully isolating the RL-induced capability difference ( sink ) 2 2 2 A sink is the limiting case of a filter. It is reached only when the capability is orthogonal to shared structure. ; penalizing it degrades RL-model fidelity ( \lx@sectionsign\ref{sec:filter} ). 4. We demonstrate that steering a single A -exclusive neuron (interchangeably: feature ) achieves +65.0 pp \Delta tool-correctness, generalizing across the supermajority of probed layers ( \lx@sectionsign\ref{sec:saturation} ). 5. We provide geometric and autointerp evidence that A -exclusive features occupy a separable "Tool Interaction" region which is distinct from shared and B-exclusive features under the DFC, but not under a matched CrossCoder ( \lx@sectionsign\ref{sec:feature_analysis} ). DFC-based model diffing is a promising tool for identifying and modulating the representations introduced by RL post-training. Our implications for mechanistic interpretability suggest runtime behavioral control of agentic LLMs. 2 Related Work (a) Crosscoders (b) Tied Crosscoders (c) Dedicated Feature Crosscoders Figure 1 : Three architectures for joint sparse decomposition of paired model activations Sparse autoencoders and mechanistic interpretability. SAEs are popular as a tool for decomposing LLM activations into interpretable features ( 3 ; 4 ; 14 ; 5 ) . These methods are built on the hypothesis that model representations are superpositions of many more features than there are neurons, and that sparse dictionary learning can recover them. Crosscoders and model diffing. Crosscoders ( 9 ) (fig. 1(a) ) extend SAEs to pairs of models by jointly encoding their activations into one shared sparse dictionary. Dedicated Feature Crosscoders ( 6 ) (fig. 1(c) ), like the Tied Crosscoders ( 1 ) (fig. 1(b) ), further partition this dictionary with gradient masking. It forces each model to read from and write to only its designated partition. Our work is the first systematic behavioral evaluation of DFC across a hyperparameter sweep, and the first to demonstrate single-neuron steering saturation. RL fine-tuning for tool use. ToolRL ( 12 ) demonstrates that RL from execution feedback substantially improves structured tool-call generation in Qwen2.5-3B . We use the ToolRL-fine-tuned model as Model A throughout our experiments. Activation steering. Our steering approach operates in the sparse dictionary of the crosscoder which allows for feature-level intervention ( 14 ) . We identify the minimal steerable subset of neurons, and a single A -exclusive neuron is enough to maximize the effect. Automated interpretability. Automated interpretability methods ( 2 ) use LLM judges to label and score SAE features. We apply the protocol to validate A -exclusive feature interpretations. 3 Methodology 3.1 Models and Training Data We study two models sharing the Qwen2.5-3B architecture ( d=2048 , 36 layers): Model B (Base): Qwen/Qwen2.5-3B ( 13 ) and Model A (RL): chengq9/ToolRL-Qwen2.5-3B ( 12 ) , RL-fine-tuned for structured <tool_call> invocation. Training uses 40{,}000 FineWeb ( 11 ) samples (general-domain) and 40{,}000 ToolRL instruction-output pairs. 3.2 DFC Architecture and Training Objective The DFC dictionary of size D is partitioned into A-exclusive [0,a_{\text{end}}) , B-exclusive [a_{\text{end}},b_{\text{end}}) , and shared [b_{\text{end}},D) features, with gradient masking enforcing exclusivity. The training objective is presented in the Equation [ 1 ]: \begin{split}\mathcal{L}={}&\mathrm{MSE}(h,\hat{h})+\lambda_{\mathrm{sh}}\cdot\overline{\|f_{\mathrm{sh}}\|_{1}}\\ &+\frac{\lambda_{\mathrm{excl}}}{2}\left(\overline{\|f_{A\cup\mathrm{sh}}\|_{1}}+\overline{\|f_{B\cup\mathrm{sh}}\|_{1}}\right)\end{split} (1) where h=(h_{A},h_{B}) is the concatenated residual stream post-MLP and \overline{|\cdot|_{1}} denotes the mean of |f_{i}| . Top- k sparsity is enforced in the encoder. 3.3 Hyperparameter Sweep The sweep varies five axes that determine the decomposition’s capacity, sparsity, and partition structure (Table 1 ). Architecture is the top-level choice between a standard Crosscoder (one unpartitioned dictionary jointly encoding both models) and a DFC (the same dictionary explicitly split into A-exclusive, B-exclusive, and shared sub-dictionaries via gradient masking). Dictionary size D is the total number of features the dictionary can represent. Top- k is the sparsity budget: at each forward pass, only the k highest-activating features are retained. Exclusive share p (DFC only) is the fraction of D allocated to each exclusive partition. \boldsymbol{\lambda_{\text{excl}}} is the magnitude penalty applied specifically to the exclusive partitions (in the CrossCoder rows of Table 8 we reuse the same coefficient on the equivalent unmasked features — see code release): setting it to 0 (“free”) lets exclusive features fire without extra cost, while 10^{-3} (“penalised”) pushes the model to route signal through shared features unless exclusivity is needed which allows to test whether the exclusive partition behaves as a sink or a filter. Axis Values Architecture CrossCoder (no exclusive partition), DFC Dictionary D 8,192 16,384 Top- k 45 90 160 Exclusive share p (DFC) 3% 5% 10% \lambda_{\text{excl}} (DFC; reused for CC variants) 0 (free) 10^{-3} (penalised) Table 1: 48 crosscoder variants ( 36 DFC and 12 crosscoders) trained for 9{,}000 steps, batch 1{,}024 , Adam lr=10^{-4} , \lambda_{\text{sh}}=10^{-3} . 3.4 Behavioral Scoring Rubric Each generated response is scored on three metrics ( 12 ) : • format_accuracy : <tool_call> present and JSON "name":"..." field present; • tool_correctness : called name fuzzy-matches a numbered tool in the prompt; • overall_score \in\{-1,0,+1,+2\} : +2 = both correct, 0 =