Correctness is not Faithfulness in RAG Attributions

Jonas Wallat ∗,, Maria Heuss ∗,, Maarten de Rijke, Avishek Anand · 2024

Read Correctness is not Faithfulness in RAG Attributions by Jonas Wallat ∗,, Maria Heuss ∗,, Maarten de Rijke, Avishek Anand (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18004.txt.

Introduction

Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand present a computing paper from 2024 on cited answers from language models. Such systems fetch source texts to cut made-up facts, then cite the sources so users can check. Past checks asked only if the cited source backs the claim.

The team splits that check in two: is the citation right, and is it faithful? A faithful citation means the model truly used the source, not that it matched the source to a belief it already held. Their test exposes that fake matching, which they call post-rationalization. Up to fifty-seven percent of citations in their study lacked faithfulness. The authors say trust needs both checks, not correctness alone.

Catalog record

CREATORJonas Wallat ∗,, Maria Heuss ∗,, Maarten de Rijke, Avishek Anand
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18004

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correctness is not Faithfulness in RAG Attributions Jonas Wallat ∗,, Maria Heuss ∗,, Maarten de Rijke, Avishek Anand Abstract Retrieving relevant context is a common approach to reduce hallucinations and enhance answer reliability. Explicitly citing source documents allows users to verify generated responses and increases trust. Prior work largely evaluates citation correctness – whether cited documents support the corresponding statements. But citation correctness alone is insufficient. To establish trust in attributed answers, we must examine both citation correctness and citation faithfulness . In this work, we first disentangle the notions of citation correctness and faithfulness, which have been applied inconsistently in previous studies. Faithfulness ensures that the model’s reliance on cited documents is genuine, reflecting actual reference use rather than superficial alignment with prior beliefs, which we call post-rationalization . We design an experiment that reveals the prevalent issue of post-rationalization, which undermines reliable attribution and may result in misplaced trust. Our findings suggest that current attributed answers often lack citation faithfulness (up to 57 % 57\% of the citations), highlighting the need to evaluate correctness and faithfulness for trustworthy attribution in language models. * * footnotetext: Equal contribution 1 Introduction a) Faithful citation Model Memory Berlin : German capital Retrieved docs Berlin : German capital Bonn: no longer capital Question: What is the capital of Germany? Model Berlin is Germany’s capital [Citation] b) Citing related context Model Memory Berlin : German capital Retrieved docs Berlin : German capital Bonn: no longer capital Question: What is the capital of Germany? Model Berlin is Germany’s capital [Citation] c) Correct but unfaithful Model Memory Berlin : German capital Retrieved docs Berlin : German capital Bonn: no longer capital Question: What is the capital of Germany? Model Berlin is Germany’s capital [Citation] d) Incorrect citation Model Memory Berlin : German capital Retrieved docs Bonn : German capital Bonn: no longer capital Question: What is the capital of Germany? Model Berlin is Germany’s capital [Citation] Figure 1 : Different answer scenarios for the query “What is the capital of Germany?” (a) The ideal case, i.e., a correct citation that is faithful to the answer’s generation process. (c) A correct but unfaithful citation, where the model post-rationalizes a citation to fit its prior. (b) A citation referring to the context that was used during the answer generation but does not contain the statement itself. (d) An incorrect citation. Trustworthiness of IR ( IR ) systems is key to their usage in high-stakes scenarios [ 23 , 20 ] . Hallucinations occur when LLM generate plausible but incorrect or fabricated information, hindering the trustworthiness of IR systems that employ LLMs [ 3 ] . One method to address the challenge posed by hallucinations in text output is to enable generated text that is directly grounded in retrieved source documents and accompanied by citations [ 41 , 6 ] . While citations do not necessarily prevent hallucinations, they increase the verifiability of generated claims [ 31 ] . Grounded text generation [ 16 ] has been studied in NLP ( NLP ) tasks like summarization and question answering, aiming to produce content derived from specific sources – either provided by users or, more recently, retrieved through mechanisms like RAG ( RAG ) [ 28 ] . Attributed text generation, often implemented as RAG systems, focuses on generating text with supporting evidence, ensuring coherence, contextual relevance, and grounding in verifiable sources by generating text alongside supporting evidence [ 6 ] . The current evaluation of attributed text focuses on the overall correctness of the answer and the correctness of citations , which is based on the agreement between attributed statements and the information that can be found in the referenced source documents. Citation correctness, sometimes called answer faithfulness, e.g. Gao et al. [19] , measures the extent to which cited documents support a generated statement. We argue that ensuring correction is not always enough. For tasks like legal information retrieval [ 34 ] or medical question answering [ 27 ] , where the retrieved documents are complex, and the answers are sensitive to model biases, simple fact-checking or correctness evaluation might prove difficult and require a nuanced understanding of the documents themselves. Yet, in those domains, unwarranted trust in generated answers can have detrimental consequences. We need to understand the model’s reasoning process to verify that the model correctly used the cited documents to produce its answer and that it was not answered from its parametric memory through post-rationalization , i.e., where models may cite sources to fit preconceived notions rather than genuine retrieval. We use the term citation faithfulness to describe whether the citation accurately depicts the model’s reasoning process. It is hard to determine citation faithfulness if we only evaluate the correctness of citations. Figure 1 illustrates the differences between faithful and unfaithful behavior as well as correct and incorrect citations. When building trustworthy IR systems that offer self-explanations – in this case, citations – we should strive to convey the system’s decisions accurately. Only if the produced citations are faithful to the underlying processes can we enable justified trust (as opposed to misplaced trust if faithfulness breaks down). Our contributions in this work are threefold. First, we offer coherent notions of attribution and citation in the context of grounded generation and introduce the concept of citation faithfulness. Second, we propose desiderata for citations that go beyond correctness and accuracy and are needed for trustworthy and usable systems. Third, we emphasize the need to evaluate the faithfulness of citations by studying post-rationalization. Our experiments reveal the existence of unfaithful behavior, with up to 57% of citations being post-rationalized. Our work on disentangling citation correctness and faithfulness in grounded text generation using LLMs aims to create more reliable IR systems by ensuring accurate and contextually faithful citations. By focusing on post-rationalization, we enhance accountability, helping IR systems avoid propagating biases or misinformation, thus promoting ethical standards in information dissemination * * * Code available at https://github.com/jwallat/RAG-attributions . 2 Related Work We summarize relevant background and position our work w.r.t. the evaluation of attributed generation, faithfulness in interpretability, and faithfulness of self-explanations. The area of knowledge conflicts [ 57 ] aims to understand information flow and whether answers originate from parametric memory or contextual information [ 38 , 53 ] . Its goal of understanding models is similar to ours but it has a different focus (full answers vs. citations) and is therefore out of scope. 2.1 LLMs and Attributions GTR (Post-hoc Attr.) Question DB LLM Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer find support RTG (Post-hoc Attr.) Question DB DB LLM Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer find support RTG (direct Attr.) Question DB LLM Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed Answer Attributed

Read the complete text