Context Tuning for In-Context Optimization
Jack Lu, Ryan Teehan, Zhenbang Yang, Mengye Ren · 2025
Read Context Tuning for In-Context Optimization by Jack Lu, Ryan Teehan, Zhenbang Yang, Mengye Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04221.txt.
Introduction
Jack Lu, Ryan Teehan, Zhenbang Yang, and Mengye Ren present Context Tuning, a way to adapt language models to a few examples without touching their weights, in a paper from 2025. In-context learning builds a memory of examples in one pass, but cannot polish that memory. Prompt tuning can polish a prompt, yet starts it blind to the examples.
Context Tuning starts the trainable memory from the examples themselves, using the model's own in-context skill. Gradient steps then refine that memory. Tests spanned CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC. The method beat both plain in-context learning and prompt-based rivals. It also matched test-time training on accuracy while training far faster.
Catalog record
| CREATOR | Jack Lu, Ryan Teehan, Zhenbang Yang, Mengye Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context Tuning for In-Context Optimization Jack Lu, Ryan Teehan, Zhenbang Yang, Mengye Ren Abstract We introduce Context Tuning , a simple and effective method to significantly enhance few-shot adaptation of language models (LLMs) without fine-tuning model parameters. While prompt-based adaptation techniques have demonstrated the effectiveness of lightweight adaptation methods for large language models (LLMs), they typically initialize a trainable prompt or prefix with irrelevant tokens for the task at hand. In contrast, Context Tuning initializes the trainable prompt or prefix with task-specific demonstration examples, leveraging the model’s inherent In-Context Learning (ICL) ability to extract relevant information for improved few-shot learning performance. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms traditional prompt-based adaptation methods and achieves competitive accuracy to Test-Time Training with significantly higher training efficiency. 1 Introduction Large language models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing (NLP) tasks by leveraging knowledge acquired during large-scale pretraining ( Brown et al., 2020 ; Grattafiori et al., 2024 ; Jiang et al., 2023 ) . These models can adapt to new tasks using only a few input and output examples provided in context, a process known as In-Context Learning (ICL) ( Brown et al., 2020 ) . However, ICL often struggles with complex reasoning or domain shifts, as it relies solely on a forward pass to interpret the examples. While methods like Test-Time Training (TTT) ( Akyürek et al., 2024 ) have shown that effective adaptation is possible with limited data, they can still be computationally expensive. This highlights the need for more efficient and effective approaches to task adaptation in LLMs. Contrary to ICL’s reliance on a forward pass, prompt-based adaptation methods like Prompt Tuning ( Lester et al., 2021 ) and Prefix Tuning ( Li and Liang, 2021 ) prepend a set of trainable vectors to each example input and optimize them via gradient descent. At a conceptual level, ICL harnesses the model’s ability to extract task-relevant information from the context of few-shot examples, while prompt-based adaptation methods optimize randomly initialized vectors to guide the model’s behavior in solving each example. Given these complementary approaches, it is natural to ask whether we can bridge them by directly optimizing the context of few-shot examples to steer the model more effectively. In this work, we introduce Context Tuning , a simple and effective method for few-shot learning that initializes trainable vectors from the few-shot examples of a novel task, then optimizes them to solve each example. We develop two variants: CT-Prompt , which applies Prompt Tuning to a soft prompt initialized from few-shot examples, and CT-KV , which applies Prefix Tuning to optimize the key-value (KV) cache derived from those same examples. While CT-Prompt achieves strong performance, it suffers from a quadratic training-time cost in the number of examples. Similarly, the recently proposed Test-Time Training (TTT) ( Akyürek et al., 2024 ) method, which fine-tunes model parameters with LoRA ( Hu et al., 2022 ) on permutations of few-shot examples, also incurs quadratic cost. In contrast, CT-KV achieves linear training time complexity while also outperforming CT-Prompt and achieving competitive performance to TTT, thanks to the efficiency and per-layer conditioning of the KV cache. In addition, because Context Tuning tunes the context and TTT tunes the model, the two methods are complementary: applying CT-KV to refine the model context after TTT’s weight updates leads to additional performance gains. A high-level comparison in Figure 1 illustrates CT-KV ’s high efficiency and accuracy, whether used alone or in combination with TTT. Figure 1: Comparison of training-free, prompt-based adaptation, and In-Context Optimization methods on solving 26 NLP-LR tasks from Table 1 . Circles are baselines; stars are our methods; bolded methods attain the best performance-efficiency tradeoff. We situate these two approaches for few-shot learning with in-context examples – TTT that optimizes the model itself, and Context Tuning that optimizes the model’s context – within a broader framework we term In-Context Optimization ( ICO ). Under this framework, adaptation leverages the LLM’s ICL ability and either updates its parameters or its context representation. We evaluate ICL, prompt-based adaptation methods, and ICO techniques across a wide range of natural language and symbolic reasoning benchmarks, including CrossFit ( Ye et al., 2021 ) , UnifiedQA ( Khashabi et al., 2020 ) , BIG-Bench Hard (BBH) ( Srivastava et al., 2023 ; Suzgun et al., 2022 ) , MMLU ( Hendrycks et al., 2021 ) , and the Abstraction and Reasoning Corpus (ARC) ( Chollet, 2019b ) . CT-KV significantly outperforms both ICL and prompt-based adaptation methods, while remaining competitive with the more computationally intensive TTT approach. Furthermore, we show that CT-KV can serve as a post-hoc refinement step following TTT, leading to improved few-shot adaptation performance compared to either method used in isolation. 2 Related Work Prompt-Based Adaptation. Prompt-Based Adaptation steers pretrained language models to solve downstream tasks by learning task-specific inputs while keeping the model weights frozen. AutoPrompt ( Shin et al., 2020 ) was an early method that constructed discrete prompts via gradient-based search. Prefix Tuning ( Li and Liang, 2021 ) introduced trainable continuous vectors that serve as a prefix to the model’s key-value cache at each layer, achieving strong performance on generation tasks with only a small number of trainable parameters. P-Tuning ( Liu et al., 2022b ) appended soft prompts to the input and used an LSTM-based prompt encoder to model dependencies between prompt tokens. Prompt Tuning ( Lester et al., 2021 ) simplified the approach by learning soft prompts solely at the input layer and demonstrated that performance improves with model scale. P-Tuning v2 ( Liu et al., 2022a ) provided an optimized implementation of Prefix Tuning and extended it to natural language understanding tasks. While these works typically initialize their learnable prompts using high-level task instructions, random tokens, or unrelated words, Context Tuning leverages the pretrained LLM’s ability to extract meaningful task-specific information directly from in-context demonstration pairs. Finally, Singhal et al., 2023 proposed Instruction Prompt Tuning, in which expert-curated few-shot demonstrations are prepended to a learned soft prompt. In contrast, Context Tuning draws demonstration pairs directly from the dataset and uses them to initialize the prompt rather than prepending them as input. In-Context Learning. Introduced by Radford et al., 2019 , ICL has become a defining feature of large language models (LLMs), enabling them to perform novel tasks by conditioning on a few input-output demonstrations without any parameter updates. This behavior has been leveraged through various prompting strategies, such as Chain-of-Thought prompting to elicit reasoning ( Wei et al., 2022 ) and self-consistency decoding to reduce variance ( Wang et al., 2023 ) . Prior work has also explored selecting informative demonstrations ( Liu et al., 2021 ; Li and Qiu, 2023 ) , as well as meta-training over large sets of tasks to improve ICL generalization and inference-time efficiency ( Min et al., 2022a ; Chen et al., 2022 ; Muhtar et al., 2024 ) . From a theoretical perspective, Dai et al., 2023 and Deutch et al., 2024 interpret ICL as performing implicit gradient descent; Zhao, 2023 conceptualizes it as contextual retrieval within an associative memory framework; and Garg et al., 2022 demonstrates