Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning

Ziyu Ma, Chenhui Gou, Yiming Hu, Yong Wang, Xiangxiang Chu, Bohan Zhuang, Jianfei Cai · 2025

Read Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning by Ziyu Ma, Chenhui Gou, Yiming Hu, Yong Wang, Xiangxiang Chu, Bohan Zhuang, Jianfei Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08246.txt.

Introduction

Ziyu Ma and colleagues improve many shot multimodal in context learning by choosing carefully where and what task vectors to insert into a model. Showing a model many examples at once runs into context length and cost limits. Task vectors compress those demonstrations into activations, but earlier methods either ignored insertion position or could not pick good values for each spot.

Their sensitivity aware framework finds insertion points from consistent patterns in activation changes across query context pairs. A clustered bank of candidate activations is built for each point, and reinforcement learning selects the best candidate. Tests on several multimodal models and visual question tasks show steady improvements.

Catalog record

CREATORZiyu Ma, Chenhui Gou, Yiming Hu, Yong Wang, Xiangxiang Chu, Bohan Zhuang, Jianfei Cai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08246

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning Ziyu Ma, Chenhui Gou, Yiming Hu, Yong Wang, Xiangxiang Chu, Bohan Zhuang, Jianfei Cai Abstract Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization. Our code will be available at https://github.com/AMAP-ML/STV . 1 Introduction Large Multimodal Models (LMMs), such as GPT-4V ( 37 ) , LLaVA ( 25 ; 30 ) , and Qwen-VL ( 44 ; 4 ) , have demonstrated strong in-context learning (ICL) capabilities for vision-language tasks, particularly in few-shot scenarios ( 3 ; 24 ) . Recent findings from commercial LMMs including Gemini ( 11 ) and GPT-4o ( 19 ) , further highlight the potential of many-shot ICL, where increasing the number of in-context examples markedly enhances performance on multimodal tasks ( 2 ; 20 ) . However, leveraging many-shot ICL in open-source LMMs remains challenging due to two key limitations: (1) limited context length (e.g., Qwen-VL ( 3 ) supports a maximum of 8192 tokens, but encoding a single image already consumes up to 256 tokens), and (2) substantial inference overhead, as incorporating more examples increases both memory usage and inference latency ( 33 ) . Figure 1: Comparison of previous task-vector-based methods and our sensitivity-aware method that systematically determines both insertion locations and task vector values. Figure 2: Attention head sensitivity across datasets (VizWiz and OK-VQA), models (Qwen-VL-7B and idefics2-8B), and sample sizes (100 vs. 500). Context-sensitive locations (black boxes) consistently emerge within tasks, validating the stability and structural patterns of activation delta, which are computed by contrasting query–context activations with query-only activations. The paradigm based on task vectors has recently emerged as a promising solution, attracting increasing attention. Unlike conventional methods that concatenate all in-context demonstrations into the input sequence, task-vector techniques ( 15 ; 31 ; 42 ; 18 ) compress many-shot demonstrations into compact, semantically rich representations (i.e., task vectors). They are then integrated into the model’s attention heads to address downstream tasks. By reframing the problem, this paradigm shifts focus from managing long input sequences to addressing two fundamental challenges: (1) how to construct semantically meaningful task vectors, and (2) where within the model architecture to insert them for optimal effectiveness. Previous studies ( 15 ; 31 ; 42 ) have introduced the value-estimation-based methods (see Fig. 1 (a)), which primarily focus on deriving task vectors from demonstrations and inserting them into predefined locations within the model. A representative approach ( 31 ) employs Principal Component Analysis (PCA) ( 1 ) on hidden states from multiple demonstrations and a dummy query, producing compact vectors. Such vectors are then used to replace the representation of the real query at fixed attention layers during inference. While this strategy mitigates the challenge of processing long input sequences and demonstrates efficacy for simple tasks, it struggles to generalize to complex, multimodal tasks ( 39 ) . Furthermore, it does not account for the critical influence of insertion locations on task performance. In contrast, recent approaches ( 18 ) shift focus to optimizing insertion locations while using fixed task vectors, a framework known as the location-selection-based method (see Fig. 1 (b)). They typically compute the mean activation across demonstrations to create a task vector, which is then used by a policy network to identify the insertion point for maximizing performance. However, these methods often harm the representation richness of task information and lead to the loss of task-specific details due to averaging multiple in-context demonstrations into a single fixed vector. Moreover, insertion typically relies on sampling-based policies, and we observe that repeated runs yield inconsistent locations, and some locations have little or no impact on model predictions, indicating suboptimal insertion. The inherent limitations of existing methods raise a fundamental question: Is it possible to determine where and what to insert in a more systematic and reliable manner? Figure 3: Overview of the STV Framework. It consists of two stages: (1) Sensitivity-aware location identification, where we compare query–context activations with query-only activations to compute activation deltas, and determine attention heads that consistently respond to contextual information. (2) Task vector selection, where candidate task vectors are drawn from a pre-computed activation bank, and reinforcement learning is used to choose the most suitable one at the identified locations. To address this issue, we propose a novel framework, Sensitivity-aware Task Vector insertion (STV), which determines where to insert and what to insert when incorporating task vectors. This framework is inspired by our observation that the effective insertion locations exhibit consistent, task-dependent patterns (see Section 2.1 for details). Based on this, STV consists of two stages: (1) identifying locations that are sensitive to contextual information, and (2) selecting optimal task vectors for insertion. In the first stage, we quantify contextual sensitivity by computing activation deltas between queries with and without demonstrations. Specifically, averaging these deltas across multiple samples highlights stable patterns, and the top- k locations with the highest sensitivity scores are chosen as insertion points. In the second stage, we first construct a pre-clustered activation bank for each location by clustering activation values derived from multiple query-context forward passes. For each identified sensitive location, a candidate task vector is then sampled from this activation bank by a learnable discrete distribution. And this distribution is iteratively updated through the REINFORCE algorithm ( 45 ) , enabling the model to progressively identify the most effective vector for each location. We systematically evaluate our method across multiple LMM families, including Qwen-VL ( 3 ) , and Idefics-2 ( 23 ) , as well as a diverse set of multimodal benchmarks. The results demonstrate that our approach consistently outperforms the previous state-of-the-art task-vector-based method, MTV ( 18 ) , while reducing location-searching time by over 98%. Besides, compared to traditional fine-tuning approaches, such as LoRA ( 17 ) , our method achieves superior accuracy with significantly lower computational resource requirements. Notably, it operates in an end-to-end manner on a single GPU with less than 20GB of memory, outperforming both zero-shot and few-shot ICL baselines ( 6 ) without requiring any model finetuning. Our contributions can be summarized as follows: • We observe that activation deltas exhibit consistent, task-dependent patterns within a given model, revealing structurally stable sensitivity to contextual information. Building on this insight,

Read the complete text