CiteEval: Principle-Driven Citation Evaluation for Source Attribution
Yumo Xu Peng Qi Jifan Chen Kunlun Liu Rujun Han, Lan Liu Bonan Min Vittorio Castelli Arshit Gupta Zhiguo Wang · 2025
Read CiteEval: Principle-Driven Citation Evaluation for Source Attribution by Yumo Xu Peng Qi Jifan Chen Kunlun Liu Rujun Han, Lan Liu Bonan Min Vittorio Castelli Arshit Gupta Zhiguo Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01829.txt.
Introduction
Yumo Xu and co-authors build a principled way to grade citations in AI answers in a 2025 paper. Search systems that cite sources earn trust only when the cites truly back the claims. Common grading leans on entailment checks that miss context, the team argues.
Their frame, CiteEval, judges cites against the query, the full retrieved set, and the answer text together. A benchmark with careful human grades, CiteBench, spans many fields. Auto graders built on the frame track human grades well, beating entailment-based rivals. Better cite scores, they say, can drive better cite habits in models.
Catalog record
| CREATOR | Yumo Xu Peng Qi Jifan Chen Kunlun Liu Rujun Han, Lan Liu Bonan Min Vittorio Castelli Arshit Gupta Zhiguo Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CiteEval: Principle-Driven Citation Evaluation for Source Attribution Yumo Xu Peng Qi Jifan Chen Kunlun Liu Rujun Han, Lan Liu Bonan Min Vittorio Castelli Arshit Gupta Zhiguo Wang Abstract Citation quality is crucial in information-seeking systems, directly influencing trust and the effectiveness of information access. Current evaluation frameworks, both human and automatic, mainly rely on Natural Language Inference (NLI) to assess binary or ternary supportiveness from cited sources, which we argue is a suboptimal proxy for citation evaluation. In this work we introduce CiteEval, a citation evaluation framework driven by principles focusing on fine-grained citation assessment within a broad context, encompassing not only the cited sources but the full retrieval context, user query, and generated text. Guided by the proposed framework, we construct CiteBench, a multi-domain benchmark with high-quality human annotations on citation quality. To enable efficient evaluation, we further develop CiteEval-Auto , a suite of model-based metrics that exhibit strong correlation with human judgments. Experiments across diverse systems demonstrate CiteEval-Auto ’s superior ability to capture the multifaceted nature of citations compared to existing metrics, offering a principled and scalable approach to evaluate model-generated citations. 1 1 1 Our code and datasets can be found at https://github.com/amazon-science/CiteEval 1 1 footnotetext: Equal contribution. ⋄ Work done at AWS AI Labs. 1 Introduction Figure 1: CiteEval considers all contexts used in the generation phase (with dashed lines) for fine-grained citation assessment. In contrast, AIS ( Rashkin et al. 2023 ; bottom left) judges citation quality with NLI, solely based on cited sources (e.g., Passage 2 in the shown example), a subset of retrieval sources . Statements in the response are in the same color as the contexts to which they should be attributed. Information-seeking systems, such as retrieval-augmented generation (RAG; Lewis et al. 2020 ) for question answering, play a vital role in how we access and understand knowledge. A key aspect of these systems is their ability to provide accurate source attribution, typically in the form of citations Gao et al. 2023b . Accurate citations establish user trust and enable verification of generated content Liu et al. 2023 ; Malaviya et al. 2024 . Evaluating the quality of citations, however, remains a significant and under-addressed challenge. Pioneered by Attributable to Identified Sources (AIS; Rashkin et al. 2023 ), existing work has largely focused on measuring the degree of supportiveness , using frameworks based on Natural Language Inference (NLI; Williams et al. 2018 ), for both human Gao et al. 2023b ; Yue et al. 2023 and automatic evaluation Zhang et al. 2024c ; Fierro et al. 2024 . In this work we critically examine the received wisdom and rethink whether NLI is truly the optimal lens through which to evaluate citation quality. As shown in Figure 1 , NLI-based metrics determine the quality of citations solely based on the cited passages. On the other hand, RAG systems often consume a wide range of contexts, and content in a model-generated response often come from paraphrasing the user query Katsis et al. 2025 , fusing parametric knowledge from pre-training Jiang et al. 2023 , or reasoning over preceding statements in the response Trivedi et al. 2023 . Not all can and should be attributable to the retrieved passages, and doing so, as a result of context insufficiency in evaluation, leads to inaccurate estimation of citation quality. Additionally, equating citation quality to binary or ternary supportiveness often falls short in capturing the nuances of citation utility Malaviya et al. 2024 . For instance, in Figure 1 , the original citation that supports the “Concept of Inertia” is from a sci-fi movie script, which can be supportive but less informative and credible compared to a physics book or Wikipedia page, as shown in the “Thinking” and “Editing” sections. To address these limitations, we introduce CiteEval, a citation evaluation framework driven by a set of design principles, including: 1. Evaluating citations against full retrieval sources (Section 2.2 ), 2. Evaluating citations beyond the retrieval context (Section 2.3 ), and 3. Evaluating citations with fine-grained criteria and scenarios (Section 2.4 ). Particularly, CiteEval mitigates the context insufficiency in NLI-based approaches with Principles 1 and 2 jointly, and moves beyond supportiveness and entailment in NLI by considering what constitutes good citations to human with Principle 3. Guided by the proposed framework, we develop CiteBench, a high-quality citation evaluation benchmark to support research in this field. CiteBench includes statement-level human judgments on RAG responses and citations, constructed with a multi-stage annotation process comprising context attribution, critical editing, and citation rating. To enable efficient assessment of citation quality, we further propose CiteEval-Auto , a suite of model-based metrics that demonstrate a strong correlation with human judgment. We benchmark the citation quality of a wide array of existing systems, providing insights on the impact of critical RAG components, and the potential of CiteEval in citation improvement. 2 Citation Evaluation Principles 2.1 Background Problem Formulation We first briefly introduce the task formulation for RAG. Given a corpus consisting of documents \mathcal{D} , a retriever fetches relevant sources \mathcal{S} (e.g., fixed-length passages) for user query Q : \mathcal{S}=\text{Retriever}_{\psi}(\mathcal{D},Q). A generator then reads the retrieved sources \mathcal{S} to produce a response. Following Gao et al. 2023b , we employ an LLM-based generator which generates a text response R , jointly with fine-grained citations \mathcal{C} : R,\mathcal{C}=\text{Generator}_{\chi}(\mathcal{S},Q) . 2 2 2 Approaches to generate citations are many and varied (see Section 6 for details). We opt for joint generation considering its effectiveness and conceptual simplicity, but nothing prohibits this work from being applied to other approaches. As shown in Figure 1 , when applicable, citations are provided at the end of each statement : \mathcal{C}=\{\mathcal{C}_{i}\}_{i=1}^{|R|} where |R| denotes the number of statements in the response, and statement-level citations \mathcal{C}_{i}=\{c_{ij}\} is a subset of retrieved passages: \mathcal{C}_{i}\subseteq\mathcal{S} . Attributable to Identified Sources (AIS) For the i th sentence in the response R_{i} , AIS ( Rashkin et al. 2023 ) evaluates its citations \mathcal{C}_{i} as: \displaystyle r_{i}\overset{\mathrm{def}}{=\joinrel=}\text{NLI}_{\phi}(\text{concat }(\mathcal{C}_{i}),R_{i}) (1) where the rating r_{i}\in\{0,1\} denotes whether statement R_{i} (i.e., the hypothesis) can be inferred from the concatenation of cited sources \mathcal{C}_{i} (i.e., the premise). As the rating approximates whether all required sources are cited, it has been adopted as citation recall in AIS and its automatic version, Auto-AIS Gao et al. 2023a based on an NLI model \phi . Auto-AIS further extends the notion to citation precision , by applying \phi to assess the relevance of each individual citation c_{ij}\in\mathcal{C}_{i} Liu et al. 2023 ; Gao et al. 2023b . CiteEval: A Principle-Driven Framework Departing from existing approaches based on NLI, such as Auto-AIS, we propose CiteEval which formulates the problem of citation evaluation as: \displaystyle r_{i}\overset{\mathrm{def}}{=\joinrel=}f_{\theta}(\mathcal{C}_{i};\mathcal{S},R,Q). (2) Particularly, f_{\theta} directly estimates fine-grained ratings r_{i} based on the full retrieval sources \mathcal{S} and beyond (e.g., R and Q ). We then derive the response-level rating based on \{r_{i}\}_{i=1}^{|R|} . We next state the principles mentioned in Section 1 that we follow to drive the modeling of f_{\theta} . 2.2 Evaluating Citations against Full Sources User Context \bullet Affirmation: Sometimes, when falling asleep, individuals may experience a heightened perception of sounds, which can feel extremely loud when Q asks about the reason behind sound sensitivity when falling asleep. Response