PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning

Zeming Chen Angelika Romanou Gail Weiss Antoine Bosselut · 2025

Read PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning by Zeming Chen Angelika Romanou Gail Weiss Antoine Bosselut (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06415.txt.

Introduction

Zeming Chen and co-authors teach language models to absorb long contexts by learning at test time, in a method named PERK. Long documents bury the facts a question needs under noise, and standard fine-tuning handles that search poorly. PERK writes the context into a small low-rank adapter through quick gradient steps instead.

Their 2025 paper trains the trick with two nested loops: an inner loop that encodes the context into the adapter, and an outer loop that learns to recall and reason over it. On synthetic and real long-context tasks, PERK beats standard fine-tuning by wide margins and matches specialist long-context models. The gains hold across model sizes, longer inputs, and facts hidden at any depth.

Catalog record

CREATORZeming Chen Angelika Romanou Gail Weiss Antoine Bosselut
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06415

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning Zeming Chen Angelika Romanou Gail Weiss Antoine Bosselut Abstract Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. Previous research shows that using test-time learning to encode context directly into model parameters can effectively enable reasoning over noisy information. However, meta-learning methods for enabling test-time learning are prohibitively memory-intensive, preventing their application to long context settings. In this work, we propose PERK ( P arameter E fficient R easoning over K nowledge), a scalable approach for learning to encode long input contexts using gradient updates to a lightweight model adapter at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard prompt-based long-context baseline, achieving average absolute performance gains of up to 90% for smaller models (GPT-2) and up to 27% for our largest evaluated model, Qwen-2.5-0.5B. In general, PERK is more robust to reasoning complexity, length extrapolation, and the locations of relevant information in contexts. Finally, we show that while PERK is memory-intensive during training, it scales more efficiently at inference time than prompt-based long-context inference. 1 Introduction Large language models (LLMs) struggle with identifying and reasoning over relevant information in long, potentially noisy, input contexts ( i.e. , long-context reasoning [ 2 , 34 , 36 , 38 , 65 ] ), including modern models with substantial context windows [ 32 , 35 , 23 ] such as GPT-4 (claimed context window of 128k tokens [ 23 ] ). These challenges are not surprising, as longer contexts contain more irrelevant distractor information, and may also reflect more complex reasoning problems ( i.e. , multiple hops), both of which increase the challenge of reasoning tasks and degrade LLM performance [ 13 ] . Furthermore, positional biases in long-context models focus disproportionate attention on the beginning or end of contexts, neglecting intermediate information [ 39 ] . In this work, we address these challenges of long-context reasoning by reframing it as test-time learning, enabling models to encode context at inference time using gradient-based parameter adaptation. Our approach processes and compresses lengthy sequences as a batch (or multiple batches) of shorter segments of the long context, rather than processing the full sequence token-by-token. The updated model can then use the newly stored parametric information (and its initial parametric knowledge) to answer relevant questions. During training, we operate two nested optimization loops: the inner loop, which is also used at test time, encodes the long context segment batches into model parameters, while the outer loop learns to retrieve and reason over relevant information from the updated parameters that now encode the long context. However, meta-learning algorithms for training test-time learning methods entail prohibitive memory overhead [ 17 , 13 , 25 ] , particularly when updating the complete set of model parameters, because they require backpropagating through a complete multi-step optimization trajectory. To reduce the memory overhead, we introduce Parameter-Efficient Reasoning over Knowledge ( PERK ), a lighter-weight meta-learning algorithm that keeps training-time memory costs low by: (1) internalizing the context into a Low-Rank Adapter (LoRA [ 24 ] ), and (2) backpropagating the outer loop over only a truncated trajectory from the final few inner-loop adaptation steps [ 54 ] . By lowering the memory cost, PERK scales more efficiently at training, supporting larger models and longer contexts. We benchmark PERK relative to models finetuned for classical in-context reasoning (FT-ICR) on multiple Needle-in-a-Haystack benchmarks, demonstrating PERK ’s advantages. A small GPT-2 model (with 127M parameters and a native 1k context window) can be scaled by PERK to reason effectively over 8k token contexts, achieving a 60% absolute performance gain over fine-tuning GPT-2 for in-context reasoning. For larger models (Qwen-2.5-0.5B) with a native 32k context window, PERK surpasses FT-ICR by a 27% absolute improvement. As relevant facts in Needle-in-a-Haystack problems are often easily distinguishable from distractor text, we also propose a more challenging evaluation setting where relevant facts are surrounded by distributionally-similar distractor facts ( i.e. , Drops-in-the-Ocean ). In this more complex setting, PERK models again outperform all baselines, including FT-ICR across the GPT-2, Qwen, and Mamba model families. Further analyses reveal PERK ’s strong test-time length extrapolation ability. Models trained with PERK on 1k-token contexts successfully extrapolate performance to unseen 32k-token contexts with only a 42% degradation, far better than FT-ICR’s 52% decline. In addition, unlike FT-ICR’s dramatic drops in performance (up to 90%) when relevant information shifts positions at test time, PERK robustly generalizes across contexts where relevant information is variably distributed along context positions. Finally, our inference complexity analysis shows that PERK ’s memory footprint scales more efficiently than FT-ICR, as does its wall-clock runtime on extremely long contexts (> 32k tokens). 2 Background and Preliminaries Causal Language Models In this work, we focus on causal language models (CLM) f_{\bm{\theta}} with a decoder-only transformer [ 51 ] architecture f and parameters {\bm{\theta}} . The common objective in autoregressive-style training of a CLM is the token-level negative log-likelihood (NLL). For a collection of text sequences {\mathcal{D}}=\{x\} with total length N_{\mathcal{D}} , this is defined as: \mathcal{L}_{\text{NLL}}({\bm{\theta}},{\mathcal{D}})\;=\;-\;\frac{1}{N_{\mathcal{D}}}\sum_{x\in{\mathcal{D}}}\;\;\sum_{t=1}^{\mathcal{T}}\log p_{\bm{\theta}}\bigl(x_{t}\;\big|\;{\bm{x}}_{<t}\bigr). (1) For training, {\bm{\theta}} is optimized iteratively via an optimization algorithm \mathcal{A}lg:(\mathcal{L},{\mathcal{D}},{\bm{\theta}},\bm{h})\mapsto{\bm{\theta}}^{\prime} where \mathcal{L} is the objective function, {\mathcal{D}} is the training corpus, {\bm{\theta}} are the initial parameters, {\bm{\theta}}^{\prime} are the obtained parameter values, and \bm{h} are the hyperparameters, such as optimizer choice and training length. Reasoning with CLMs We denote reasoning problems as tuples r=({\mathcal{K}},{\bm{q}},y) of context {\mathcal{K}} , question {\bm{q}} , and target response y . With the pre-trained large CLMs’s ability to reason over facts in input contexts with reasonable success [ 14 ] , prompt-based in-context reasoning (ICR) has become popular for automating reasoning tasks. In this setting, {\mathcal{K}} and {\bm{q}} are combined via a template into a single prompt {\bm{x}}(r) . The model’s parameters are optimized to predict the tokens of the correct answer: \mathcal{L}_{\text{reason}}({\bm{\theta}},{\mathcal{D}})\;=\;-\;\frac{1}{N_{{\mathcal{D}}_{y}}}\sum_{r=({\mathcal{K}},{\bm{q}},y)\in{\mathcal{D}}}\;\sum_{t=1}^{\mathcal{T}_{y}}\log p_{\bm{\theta}}\bigl(y_{t}\,\big|\,{\bm{x}}(r),y_{<t}\bigr) (2) where {\mathcal{D}} is the training corpus, \mathcal{T}_{y} is the length of an example answer, and N_{{\mathcal{D}}_{y}} is the total length of all answers in the dataset. Reasoning as Test-time Learning In this setting, models simulate reasoning by first encoding the given context {\mathcal{K}} into a CLM’s parameters {\bm{\theta}} , and then using the updated parameters \phi({\bm{\theta}},{\mathcal{K}}) to generate a response to the given question {\bm{q}} [ 13 ] . The context is encoded by adapting the model’s parameters using a CLM objective over the context: \phi({\bm{\theta}},{\mathcal{K}})=\mathcal{A}lg(\mathcal{L}_{\text{NLL}},{\mathcal{K}},{\bm{\theta}},\bm{h}) . We then use f_{\phi} to generate answers for any relevant questions {\bm{q}} . Test-time learning methods for reasoning stem from a key observation: CLMs can store and retrieve diverse information in their parameters from pre-training [ 52 , 5 ] . Yet these same models struggle with long-context problems [ 23 , 32 , 65 , 35 ] , despite the contexts containing far less information than what they’ve seen in pre-training. CLMs may reason better with knowledge in their parameters

Read the complete text