CUB: Benchmarking Context Utilisation Techniques for Language Models
Lovisa Hagström Youna Kim Haeun Yu, Sang-goo Lee Richard Johansson Hyunsoo Cho Isabelle Augenstein · 2025
Read CUB: Benchmarking Context Utilisation Techniques for Language Models by Lovisa Hagström Youna Kim Haeun Yu, Sang-goo Lee Richard Johansson Hyunsoo Cho Isabelle Augenstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16518.txt.
Introduction
Lovisa Hagström and co-authors build a benchmark for testing how well language models use supplied context, in a 2025 paper. Models answering questions from retrieved documents sometimes ignore facts that clash with their training, or get distracted by junk passages. Many tricks have been proposed to fix this, but they had never been compared fairly. Their benchmark, CUB, puts those tricks under varied noisy conditions inside a retrieval setup.
They ran the widest test so far: seven leading methods, three datasets and tasks, and eleven language models. The findings expose clear gaps in current techniques. A shared testbed matters because retrieval systems are now the front door to search and office assistants, and silent failures there mislead users with confident, wrong answers.
Catalog record
| CREATOR | Lovisa Hagström Youna Kim Haeun Yu, Sang-goo Lee Richard Johansson Hyunsoo Cho Isabelle Augenstein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16518
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CUB: Benchmarking Context Utilisation Techniques for Language Models Lovisa Hagström Youna Kim Haeun Yu, Sang-goo Lee Richard Johansson Hyunsoo Cho Isabelle Augenstein Abstract Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language models (LMs) may ignore relevant information that contradicts outdated parametric memory or be distracted by irrelevant contexts. While many context utilisation manipulation techniques (CMTs) have recently been proposed to alleviate these issues, few have seen systematic comparison. In this paper, we develop CUB (Context Utilisation Benchmark) – the first comprehensive benchmark designed to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG). With this benchmark, we conduct the most extensive evaluation to date of seven state-of-the-art methods, representative of the main categories of CMTs, across three diverse datasets and tasks, applied to 11 LMs. Our findings expose critical gaps in current CMT evaluation practices, demonstrating the need for holistic testing. We reveal that most existing CMTs struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. We also find that many CMTs display inflated performance on simple synthesised datasets, compared to more realistic datasets with naturally occurring samples. * * footnotetext: Equal contribution. 2 2 footnotetext: Dept. of AI, Institute for Multiscale Matter and Systems 1 Introduction Figure 1: The Context Utilisation Benchmark. We evaluate a range of LMs under different CMTs on samples from NQ, DRUID and CounterFact for gold, conflicting and irrelevant contexts. Context utilisation is a key component of language models (LMs) used for retrieval-augmented generation (RAG), as the benefits of retrieving external information are only realised if the generative model makes adequate use of the retrieved information. While recent research has identified many benefits of augmenting LMs with retrieved information ( Shuster et al., 2021 ; Hagström et al., 2023 ) , it has also identified weaknesses of LMs used for RAG, of which many are associated with context utilisation. For example, LMs can easily be distracted by irrelevant contexts ( Shi et al., 2023 ) or ignore relevant contexts due to memory-context conflicts ( Xu et al., 2024 ) . The robustness of LMs to irrelevant contexts is important as information retrieval systems used for RAG are not guaranteed to always retrieve relevant information. Moreover, as information may be updated to conflict with the training data of the LM, the model should prioritise the most recently updated information. As a consequence, many different methods for increasing or suppressing LM context utilisation, henceforth referred to as CMTs (Context utilisation Manipulation Techniques), have been proposed. The methods encompass a broad range of approaches, from different decoding methods ( Shi et al., 2024 ; Kim et al., 2024 ) to fine-tuning methods ( Li et al., 2023 ) , prompting ( Liu et al., 2023 ) , multi-agent ( Feng et al., 2024 ; Du et al., 2024 ) , and mechanistic interventions ( Ortu et al., 2024 ; Jin et al., 2024 ; Sun et al., 2025 ) . Many of these CMTs are developed with distinct objectives in mind, often targeting a specific aspect of context utilisation – such as improving robustness to irrelevant contexts or enhancing faithfulness to conflicting information. While each method yields promising results in isolation, their evaluation is often limited to settings that closely match the method’s target objective, making it unclear whether these approaches will generalise or remain effective in more realistic RAG scenarios. To address this pressing evaluation gap, we develop the first comprehensive CMT benchmark to systematically test and compare different CMTs across datasets that represent the diversity of real-world domains and tasks ( Figure 1 ). Our contributions are as follows: • We develop CUB (Context Utilisation Benchmark) – the first comprehensive benchmark designed to enable rigorous evaluation and comparison of CMTs under various noisy context conditions (§ 3 ). 1 1 1 Code and datasets available at https://github.com/copenlu/cmt-benchmark CUB evaluates CMTs across gold, conflicting, and irrelevant contexts that capture key RAG challenges. • We conduct an extensive evaluation of CMTs, assessing seven state-of-the-art methods representative of the main categories of CMTs (§ 4 ) across our benchmark (§ 6 ). The study encompasses approximately 800 experimental data points, spanning 11 LMs, 7 CMTs, 3 datasets, and 3 different context types. We believe this is the most extensive evaluation of CMTs to date. • We provide insights into what CMT works best for different scenarios and identify critical areas for improvement. Our analysis reveals that existing CMTs face fundamental trade-offs – they struggle to optimise performance across all context types. This points to a need for next-generation CMTs that can robustly handle diverse context conditions. 2 Related Work Context-intensive datasets We consider two main categories of context-intensive datasets: 1) datasets representing knowledge-intensive tasks , i.e. tasks for which access to external context is crucial, and 2) datasets designed to diagnose model adaptability to external knowledge. Examples of datasets representative of knowledge-intensive tasks are Natural Questions (NQ), DRUID, the KILT datasets and PubMedQA ( Kwiatkowski et al., 2019 ; Hagström et al., 2025a ; Petroni et al., 2021 ; Jin et al., 2019 ) . Examples of diagnostic datasets representative of the latter category are CounterFact and ConflictQA ( Meng et al., 2022 ; Xie et al., 2024a ) , which contain synthesised queries based on fact triplets. In these datasets, contexts are synthesised to induce knowledge conflicts by promoting answers that conflict with the parametric memory of the studied LM. These diagnostic datasets have found widespread use for work on mechanistic interpretability and the evaluation of context utilisation ( Meng et al., 2022 ; Geva et al., 2023 ; Ortu et al., 2024 ; Sun et al., 2025 ; Islam et al., 2026 ) . Previous work has typically evaluated different CMTs on either of the dataset categories in isolation, creating a fragmented understanding of CMT effectiveness. CUB bridges this critical gap by incorporating datasets representative of both knowledge-intensive tasks and diagnostic datasets, enabling the first comprehensive evaluation of CMTs across truly diverse settings. CMTs Many context utilisation manipulation techniques have recently been proposed. Existing CMTs can be categorised into one of four main groups based on intervention level , i.e. what aspect of the model they manipulate. 1) fine-tuning CMTs update model parameters to modify context utilisation. For example, fine-tuning on distracting contexts was found to yield improved robustness to distracting contexts ( Li et al., 2023 ; Shen et al., 2024 ; Yoran et al., 2024 ) . Fang et al. (2024) specifically focus on different types of retrieval noise likely to be encountered in real-world environments and develop a fine-tuning approach to handle these. 2) prompting techniques modify the input to the LM to improve context utilisation, representing minimally modified settings. 3) mechanistic interventions on the LM modify certain model components at inference time to alter context utilisation. Examples involve attention modification ( Ortu et al., 2024 ; Jin et al., 2024 ) and SpARe interventions ( Zhao et al., 2025 ) . Lastly, 4) decoding methods involve a modified decoding approach, applied to the output logits, to manipulate context utilisation. Examples include context-aware contrastive decoding ( Yuan et al., 2024 ; Kim et al., 2024 ; Shi