RAG-E: Quantifying Retriever-Generator Alignment and Failure Modes

Korbinian Randl, Guido Rocchietti, Aron Henriksson, Ziawasch Abedjan, Tony Lindgren, John Pavlopoulos · 2026

Read RAG-E: Quantifying Retriever-Generator Alignment and Failure Modes by Korbinian Randl, Guido Rocchietti, Aron Henriksson, Ziawasch Abedjan, Tony Lindgren, John Pavlopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21803.txt.

Introduction

Korbinian Randl and co-authors build a way to inspect how retrieval systems and language models work together. Retrieval-augmented generation grounds a model's answers in fetched documents. But how the retriever and the generator interact is opaque, which is risky in high-stakes fields.

Their 2026 preprint presents RAG-E, an end-to-end explainability framework. It measures how well the retriever's choices align with what the generator actually uses, through attribution methods grounded in mathematics. And it names the ways the pairing can fail. Deployment teams in medicine, law, and finance need exactly this. A way to tell whether a wrong answer came from bad retrieval, bad generation, or a mismatch between them.

Catalog record

CREATORKorbinian Randl, Guido Rocchietti, Aron Henriksson, Ziawasch Abedjan, Tony Lindgren, John Pavlopoulos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RAG-E: Quantifying Retriever-Generator Alignment and Failure Modes Korbinian Randl, Guido Rocchietti, Aron Henriksson, Ziawasch Abedjan, Tony Lindgren, John Pavlopoulos Abstract Retrieval-Augmented Generation (RAG) systems combine dense retrievers and language models to ground LLM outputs in retrieved documents. However, the opacity of how these components interact creates challenges for deployment in high-stakes domains. We present RAG-E, an end-to-end explainability framework that quantifies retriever-generator alignment through mathematically grounded attribution methods. Our approach adapts Integrated Gradients for retriever analysis, introduces PMC SHAP, a Monte Carlo-stabilized Shapley Value approximation, for generator attribution, and introduces the Weighted Attribution-Relevance Gap (WARG) metric to measure how well a generator’s document usage aligns with a retriever’s ranking. Empirical analysis on TREC CAsT and FoodSafeSum reveals critical misalignments: for 47.4% to 66.7% of queries, generators ignore the retriever’s top-ranked documents, while 48.1% to 65.9% rely on documents ranked as less relevant. These failure modes demonstrate that RAG output quality depends not solely on individual component performance but on their interplay, which can be audited via RAG-E. 1 Introduction Retrieval Augmented Generation ( 18 , RAG) has become a standard in modern Question-Answering tasks, from the everyday use of ChatGPT to applications in critical domains like medicine and law ( 2 ; 6 ) . A standard RAG pipeline operates as a two-stage process: a Retriever ( Ret ) is responsible for ranking the documents by relevance to a user query. A Generator ( Gen ) synthesizes the information contained in the top- k ranked documents and generates an answer in natural language. While RAG reduces the opacity of Large Language Models (LLMs) by grounding its responses in specific sources, its internal reasoning is not inherently transparent. On the one hand, considering all the retrieved documents as sources leaves the uncertainty of which specific documents contain the relevant information. On the other hand, LLM-generated attribution is statistical in nature and not necessarily faithful ( 27 ) . In this work, we address this uncertainty by introducing RAG-E, a mathematically grounded explainability framework for RAG. An example of the explanations provided by our framework is shown in Fig. 1 , where we compute token saliency for Ret input and Gen query input and document attribution. Using RAG-E, we seek to gain a better understanding of RAG: rq1 : What do transformer-based and state-of-the-art Ret and Gen models focus on? Query: Where was Marie Cur ie born ? Legend: Only Ret \rightarrow Only Gen Only Ret \rightarrow Only Gen Document 1: Maria Sk ło d owska , later known as Marie Cur ie , was born on November 7 , 1867 . Document 2: Bor n in Paris on 15 May 1859 , Pierre Cur ie was the son of Eu gène Cur ie , a doctor of French Catholic origin from Als ace . \vdots \vdots Document 5: Maria Sk ło d owska was born in Wars aw , in Congress Poland in the Russian Empire , as the fi ft h and young est child of well - know n teachers Bron i sław a , née Bogu ska , and W ła dy sław Sk ło d owski . Assistant: Based on the provided documents , Marie Cur ie was born in Warsaw , in Congress Poland in the Russian Empire ( Document 5 ). Legend: Doc. 1 (-5%) Doc. 2 (-19%) Doc. 3 (-15%) Doc. 4 (-7%) Doc. 5 (54%) Importance of each token inside the query for both retrieval and generation. Importance of each token inside the documents for retrieval. Attribution of generated tokens to the documents. Percentages in brackets are the overall influence of each document on the output. Figure 1 : Visual example of the generated explanations. We detect important spans influencing both the retrieval and generation steps. This example was generated using Arctic Embed 2 and Llama 3.1. Our results suggest that even neural network-based Ret models select documents based on the existence of keywords also found in the query. Furthermore, we find that Gen statistically prefers documents early in the prompt over documents later on. This is expected as RAG explicitly leverages the ordered nature of the retrieved documents. Even so, the fact that the position of information in the prompt influences its probability of occurring in the output is also known in non-RAG LLMs ( 20 ) and can become problematic if Gen ignores important sources in favour of higher-ranked sources. An empirical analysis, conducted with LLama 3.1 8B and Gemma 3 13B on two datasets spanning open-domain and domain-specific queries, shows that for 60% of queries on domain-specific data, Gen ignores Ret ’s top-ranked document, while in 57% of cases it relies primarily on a document Ret attributed with lower relevancy. This reflects two failure modes, which we term wasted retrieval and noise distraction , revealing that RAG output depends not solely on individual component performance but on their interplay. The lack of a single metric assessing this misalignment motivates our second research question. rq2 : How can the agreement between Ret and Gen be quantified? As RAG systems proliferate in high-stakes domains, addressing misalignment between Ret and Gen becomes increasingly important. In fact, a medical RAG system that ignores the best evidence or a legal system that focuses on marginal precedent might generate unreliable outputs with real consequences. Nonetheless, prior work on RAG systems typically focuses on explaining either Ret or Gen in isolation. For example, 37 ; 12 study retriever explanations, while 26 ; 7 offer source attribution for the generator (more details in § 6 ); however, there is no way to quantify how information flows across both components or how they diverge. Being able to do so can help not only by improving the transparency of RAG systems but also by increasing alignment between the two components and possibly reducing computational cost, e.g.., by retrieving fewer documents if we know some of them will not be used by Gen . To address this gap, we propose an end-to-end RAG auditing framework with three components: Attribution methods tailored to RAG: We introduce pmc SHAP, a Monte Carlo stabilized variant of Kernel-SHAP ( 22 , k SHAP) that achieves significantly more accurate and reproducible approximations of Shapley Values ( 29 , SV) for autoregressive Gen s. This addresses a fundamental limitation of k SHAP: its instability when applied to variable-length generation with dependent features (overlapping documents). Furthermore, we establish a baseline embedding for Integrated Gradients ( 32 , IG) on dense retrievers through systematic empirical analysis, showing that replacing non-special tokens with the [unk] embedding significantly outperforms baselines. Diagnostic metrics for alignment: We propose Weighted Attribution-Relevance Gap (WARG), a novel metric based on Rank-Biased Overlap ( 34 , RBO) that quantifies how well Gen ’s use of documents aligns with Ret ’s ranking. By sweeping a bias parameter p\in(0,1) , it analyses whether misalignment is concentrated at the top (indicative of primacy bias) or diffuse across the ranking. Empirical discovery of failure modes: Our framework shows that structural misalignment is prevalent and model-dependent. Llama 3.1 8B exhibits primacy bias (trusts prompt ranking even when shuffled), while Gemma 3 12B demonstrates a more semantic-driven behaviour and still relies on low-ranked documents in 57% of cases. We provide open-source tooling (RAG-E

Read the complete text