A Reality Check on Context Utilisation for Retrieval-Augmented Generation
Lovisa Hagström Sara Vera Marjanović, Haeun Yu Arnav Arora Christina Lioma, Maria Maistro Pepa Atanasova Isabelle Augenstein · 2024
Read A Reality Check on Context Utilisation for Retrieval-Augmented Generation by Lovisa Hagström Sara Vera Marjanović, Haeun Yu Arnav Arora Christina Lioma, Maria Maistro Pepa Atanasova Isabelle Augenstein (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17031.txt.
Introduction
Retrieval-augmented generation gives a language model outside text to answer from, but models may use that text poorly. Lovisa Hagström and co-authors built a dataset in 2024 to test real-world use of retrieved context. Called DRUID, it holds real queries and retrieved passages for claim checking, with human labels for how each passage stands toward the claim.
Against synthetic datasets, real retrieved text proved messier and more varied. Artificial data overstated traits that are rare in real retrieval, which inflated scores for how well models use context. Single traits of a passage explained little; where the passage came from mattered more. The authors urge testing on real-world data before trusting results.
Catalog record
| CREATOR | Lovisa Hagström Sara Vera Marjanović, Haeun Yu Arnav Arora Christina Lioma, Maria Maistro Pepa Atanasova Isabelle Augenstein |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Reality Check on Context Utilisation for Retrieval-Augmented Generation Lovisa Hagström Sara Vera Marjanović, Haeun Yu Arnav Arora Christina Lioma, Maria Maistro Pepa Atanasova Isabelle Augenstein Abstract Retrieval-augmented generation (RAG) helps address the limitations of parametric knowledge embedded within a language model (LM). In real world settings, retrieved information can vary in complexity, yet most investigations of LM utilisation of context has been limited to synthetic text. We introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand contexts) with real-world queries and contexts manually annotated for stance. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. We compare DRUID to synthetic datasets (CounterFact, ConflictQA) and find that artificial datasets often fail to represent the complexity and diversity of realistically retrieved context. We show that synthetic datasets exaggerate context characteristics rare in real retrieved data, which leads to inflated context utilisation results, as measured by our novel ACU score. Moreover, while previous work has mainly focused on singleton context characteristics to explain context utilisation, correlations between singleton context properties and ACU on DRUID are surprisingly small compared to other properties related to context source. Overall, our work underscores the need for real-world aligned context utilisation studies to represent and improve performance in real-world RAG settings. 1 Introduction Figure 1: Datasets for context usage investigations. Retrieval-augmented generation (RAG) can be used to alleviate problems arising from imperfect parametric knowledge of language models (LMs), which may encode limited and potentially outdated information ( Gao et al. 2024 ; Vu et al. 2024 ) . However, the benefits of RAG are only realised if 1) the retrieval module retrieves helpful information and 2) the generative model successfully leverages the retrieved information. As a consequence, there have been many studies looking at the performance and interaction of these two components and how to improve them ( Gao et al. 2024 ) . However, existing research has mainly studied RAG in a disjoint manner, where studies of the quality and relevance of the retrieved information are detached from studies of LM context usage ( Shi et al. 2023 ; Xie et al. 2023 ; Tan et al. 2024 ; Du et al. 2024 ) . Hence, little is understood about 1) the characteristics of retrieved contexts and 2) their impact on LM context usage. Most notably, studies of LM context usage have leveraged controlled datasets using synthesised context to emulate a limited set of context characteristics (see Figure 1 , left). For example, CounterFact and its variants are template-based, lending a controlled albeit very artificial and simplistic setup ( Yu et al. 2023 ; Du et al. 2024 ) . ConflictQA, on the other hand, is based on a mix of generated and retrieved contexts to study context usage in a more realistic setup with coherent and convincing contexts ( Xie et al. 2024 ) . Nevertheless, the scenarios described by these datasets are not representative of real-world RAG scenarios, as the context types do not reflect the diversity and complexity of the ones returned by an actual retriever present in RAG ( Longpre et al. 2021 ; Ravaut et al. 2024 ; Ortu et al. 2024 ) . This work studies context usage for RAG in real-world scenarios with real-world queries and context, as opposed to artificial samples. To this end, we focus on the prototypical information-seeking task of fact verification, where retrieving and utilising real-world information is vital. For the task, an agent is provided with a statement about the world – a claim – and needs to decide whether it is true or false using context retrieved from an external source – evidence ( Guo et al. 2022 ) . We take real fact-checked claims as ‘queries’ and the retrieved evidence as ‘context’ to evaluate RAG in this real-world setting, which naturally facilitates our goal of studying real-world context properties in RAG ( Samarinas et al. 2021 ; Atanasova et al. 2022 ; Chrysidis et al. 2024 ; Glockner et al. 2024 ) . In particular, this work makes three main contributions. First, we introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand context) with real-world (query, context) pairs to facilitate studies of context usage and failures in real-world scenarios (§ 3 ). Second, we introduce a novel context-usage measure, ACU , which rectifies issues in previous measures. Thirdly, we highlight major differences between popular synthetic datasets and real-world data ( DRUID ): both in over-arching characteristics (§ 4 ), as well as how the provided context is used across different popular LMs (§ 5 ). We show that synthetic datasets oversell the impact of certain context characteristics (e.g. knowledge conflicts), which are rare in retrieved data. Furthermore, synthetic data exaggerates the ‘context-repulsion’ seen for LMs, as we rarely see this behaviour in realistic data. Finally, we show that there is no singleton context characteristic (e.g. context length or perplexity) indicating RAG failure in real-world settings. Altogether, our work provides a reality check on LM context usage and points to the need for real-world aligned studies to fully understand and improve context utilisation for RAG. We also provide tools and resources to facilitate such studies. 1 1 1 https://github.com/copenlu/context-utilisation-for-RAG 2 Related Work Claim Verification Datasets Claim verification datasets typically measure an LM’s ability to assess the veracity of a claim based on retrieved context ( evidence ). Importantly, the information requirements of this task can be challenging, resulting in retrieved evidence that is noisy ( Samarinas et al. 2021 ; Atanasova et al. 2022 ; Fajcik et al. 2023 ; Chrysidis et al. 2024 ; Glockner et al. 2022 ; Warren et al. 2025 ) . Furthermore, as LMs fine-tuned for claim verification have been shown to ignore evidence ( Schuster et al. 2019 ; Schuster et al. 2021 ) , it is important to understand the causes of this behaviour. Therefore, with DRUID , we are the first to collect annotations for a range of ‘noisy’ characteristics of retrieved real-world contexts to assess how they affect LMs. Furthermore, unlike concurrent claim verification datasets, which either present artificial samples or a limited or less realistic scenario for context retrieval ( Thorne et al. 2018 ; Augenstein et al. 2019 ; Diggelmann et al. 2020 ; Schlichtkrull et al. 2023 ) , DRUID includes contexts automatically retrieved from the web to assess their impact on RAG, leading to a wide diversity of context properties including insufficient and leaked evidence. No other existing fact-checking datasets fulfil all of these properties (see Table 4 in the Appendix). Datasets for Context Usage Investigations Two popular datasets used for context usage investigations are CounterFact and ConflictQA ( Meng et al. 2022 ; Xie et al. 2024 ) . These datasets contain synthesised queries based on fact triplets from LAMA ( Petroni et al. 2019 ) (e.g. Thomas Ong-citizen of-Singapore) for which some contexts have been synthesised to induce knowledge conflicts by promoting answers in conflict with the parametric memory of the studied LM (e.g. ‘Pakistan’ as opposed to ‘Singapore’). The datasets have found widespread use for work on mechanistic interpretability and the evaluation of context utilisation ( Meng et al. 2022 ; Geva et al. 2023