Enhancing Cross-task Transfer of Large Language Models via Activation Steering

Xinyu Tang, Zhihao Lv, Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, Jun Zhou · 2025

Read Enhancing Cross-task Transfer of Large Language Models via Activation Steering by Xinyu Tang, Zhihao Lv, Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, Jun Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13236.txt.

Introduction

Xinyu Tang and co-authors present a way to move skill between tasks in language models in this 2025 paper. Models can use what they know when prompted, but they often fail on new tasks where data are scarce. Showing examples from other tasks helps, yet that route can be fragile and slow.

Their method, called CAST, works inside the model instead of in the prompt. The authors found that examples boost inner activity in like patterns across tasks. CAST picks strong, varied samples from rich tasks and uses their boosted inner patterns to adapt the model to poor tasks. Tests across topics and languages beat rival methods, with lower computing cost.

Catalog record

CREATORXinyu Tang, Zhihao Lv, Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, Jun Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13236

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Cross-task Transfer of Large Language Models via Activation Steering Xinyu Tang, Zhihao Lv, Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, Jun Zhou Abstract Large language models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, particularly in data-scarce scenarios. While cross-task in-context learning offers a direct solution for transferring knowledge across tasks, it still faces critical challenges in terms of robustness, scalability, and efficiency. In this paper, we investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion. Through an analysis of activation patterns in the latent space of LLMs, we observe that the enhanced activations induced by in-context examples have consistent patterns across different tasks. Inspired by these findings, we propose CAST , a novel C ross-task A ctivation S teering T ransfer framework that enables effective transfer by manipulating the model’s internal activation states. Our approach first selects influential and diverse samples from high-resource tasks, then utilizes their contrastive representation-enhanced activations to adapt LLMs to low-resource tasks. Extensive experiments across both cross-domain and cross-lingual transfer settings show that our method outperforms competitive baselines and demonstrates superior scalability and lower computational costs. 1 Introduction Large language models (LLMs) LLM-survey ; openai-o1 ; deepseek-r1 have demonstrated remarkable capabilities to acquire and store knowledge during pretraining, which can be effectively accessed through prompting. However, as this usage becomes more prevalent, a crucial challenge emerges: these models often struggle with tasks that were not seen during pretraining, especially in data-scarce scenarios Inherent-limits . One common approach to tackle this issue is transfer learning transfer-learning1 ; transfer-learning2 ; transfer-learning3 , which uses knowledge from high-resource tasks to adapt to low-resource ones. Some studies continuous1 ; continuous2 ; continuous3 ; continuous4 focus on fine-tuning soft-prompts for cross-task transfer. They typically train soft prompts on data-sufficient tasks, and then apply them to data-scarce tasks by adding them to the input during inference. While effective, these methods still require training and can not generalize well across diverse tasks. An alternative approach is cross-task in-context learning (ICL) cross-lingual-icl ; discrete2 ; discrete3 ; cross-domain-ICL , which leverages labeled examples from high-resource tasks to improve performance on low-resource tasks without parameter updates. Despite its promise, cross-task ICL still suffers from several limitations: (1) Its performance is highly sensitive to the choice of demonstrations Demo-Selection1 ; Demo-Selection2 ; Demo-Selection3 , prompt templates Template1 ; Template2 , and source tasks cross-domain-ICL , which restricts its adaptability (robustness). (2) Few demonstrations can be included due to the limited context length of LLMs (scalability). (3) Higher computational costs with an increasing number of demonstrations, for the quadratic computational complexity of Transformers to the input length (efficiency). To address these challenges, one possible solution is to operate in the continuous space continuous-learning1 ; continuous-learning2 . This approach does not require adding explicit input tokens, thereby avoiding increasing the input context length. This leads us to raise a research question: Can we achieve effective cross-task transfer through latent space steering ? To answer this question, we conduct an empirical study of activation patterns in both zero-shot and few-shot settings. Our findings reveal that few-shot prompts yield higher matrix entropy than zero-shot prompts, suggesting that LLMs encode richer and more diverse features from in-context examples. Additionally, we observe that the activation differences between few-shot and zero-shot prompts are nearly parallel across tasks in the model’s latent space. This phenomenon indicates that the enhanced information induced by in-context examples follows consistent patterns across tasks, which provides strong support for the possibility of cross-task transfer in the latent space. Building on these insights, we propose a novel C ross-task A ctivation S teering T ransfer framework, namely CAST . We first select a representative subset of high-resource examples that balance influence and diversity, aiming to improve the efficiency of feature extraction. Specifically, we first construct a similarity-based graph of high-resource examples, then iteratively choose a sample with the highest combined score of influence and diversity at each step, and repeat this process to form the subset. Next, we compute the activation differences between few-shot and zero-shot prompts for these selected samples, which are subsequently injected into the forward pass of low-resource queries via activation steering to guide LLMs toward effective cross-task transfer. This approach is both efficient and scalable, as it leverages pre-extracted activations from high-resource tasks during inference without parameter updates or input expansion. To validate the effectiveness of our proposed method, we conduct extensive experiments in both cross-domain and cross-lingual scenarios. Experimental results demonstrate that our approach consistently outperforms competitive baselines while maintaining lower computational cost. Our main contributions can be summarized as follows: \bullet To the best of our knowledge, we are the first to analyze the zero-shot and few-shot prompts from the perspective of activations, and find that the enhanced representation from in-context examples exhibits consistent patterns across tasks in the latent space of LLMs. \bullet We propose a novel cross-task transfer learning framework based on activation steering without modifying the parameters or increasing the input length, which effectively mitigates the challenge of data scarcity in real-world scenarios. \bullet Extensive experiments validate the effectiveness of our proposed method in both cross-domain and cross-lingual scenarios, while also showing strong scalability and low computational cost. 2 Related Work Transfer Learning. In data-scarce scenarios, transfer learning offers a promising solution to alleviate the lack of labeled data in low-resource tasks by leveraging labeled samples from high-resource tasks. Existing transfer learning approaches for LLMs can be mainly categorized into two types: continuous and discrete cross-task transfer. Continuous cross-task transfer methods continuous1 ; continuous2 ; continuous3 learn a shared continuous soft prompt from source tasks and apply it to the target tasks. While effective, they require fine-tuning and generalize poorly across diverse tasks. In contrast, discrete cross-task transfer methods cross-lingual-icl ; discrete2 ; discrete3 ; cross-domain-ICL directly incorporate high-resource examples into LLM inputs to solve low-resource tasks without parameter updates. However, such an approach faces several limitations: its performance depends heavily on the quality of demonstration, the fixed context window constrains scalability, and the quadratic complexity reduces efficiency with longer inputs. To overcome these issues, we propose a novel approach to extract activations from high-resource tasks and inject them into low-resource tasks. This approach eliminates the need for fine-tuning or input expansion, while preserving robustness, scalability, and computational efficiency. Activation Steering. Activation steering is a well-established method that treats internal representations as the fundamental unit, focusing on analyzing and manipulating them within neural networks. This technique has been successfully applied in various scenarios, including alignment re-alignment , personality modeling re-person , instruction following re-instruction , hallucination mitigation re-halu1 ; re-halu2 , safety enhancement re-safety , and reasoning improvement re-cot1 ; re-cot2 . In this work, we leverage the activation steering method to transfer knowledge from the data-sufficient tasks to address data-scarce tasks. 3 Empirical Study The Hopfieldian view of cognition hopfield suggests that neural computation arises from dynamic transformations in population-level neural activity driven by external stimuli. This perspective has inspired mechanistic interpretations of artificial neural networks, where activation steering Representation-Engineer has emerged as a powerful method for analyzing the internal states of models. By treating intermediate

Read the complete text