Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin · 2025
Read Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges by Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15205.txt.
Introduction
Human and model judges for answer support are compared by Nandan Thakur and co-authors in a 2025 arXiv paper. Search-aided answer systems cite sources to cut made-up claims. Judges must check if the cited text truly backs the answer. The team ran a big study on a shared test track with forty-five entries on thirty-six topics.
A GPT-4o judge faced human judges in two modes, from scratch and with model drafts to edit. Full match came in just over half of from-scratch checks, rising to near three quarters with edits. A fresh human judge matched the model more than the first humans did. Model judges look fit for this task.
Catalog record
| CREATOR | Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin Abstract Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing “ground truth”, thereby reducing system hallucinations. A crucial factor in RAG evaluation is “support”—whether the information in the cited documents supports the answer. To this end, we conducted a large-scale comparative study of 45 participant submissions on 36 topics to the TREC 2024 RAG Track, comparing an automatic LLM judge (GPT-4o) against human judges for support assessment. We considered two conditions: (1) fully manual assessments from scratch and (2) manual assessments with post-editing of LLM predictions. Our results indicate that for 56% of the manual from-scratch assessments, human and GPT-4o predictions match perfectly (on a three-level scale), increasing to 72% in the manual with post-editing condition. Furthermore, by carefully analyzing the disagreements in an unbiased study, we found that an independent human judge correlates better with GPT-4o than a human judge, suggesting that LLM judges can be a reliable alternative for support assessment. To conclude, we provide a qualitative analysis of human and GPT-4o errors to help guide future iterations of support assessment. 1 Introduction Retrieval-Augmented Generation (RAG) has recently gained popularity in both academic and industrial settings (e.g., Bing Search [ 19 ] and popular frameworks like LangChain [ 4 ] ). In RAG, large language models (LLMs) generate answers to user queries that include citations to source documents as necessary [ 11 , 16 , 12 , 3 ] . RAG systems improve factuality and verifiability, reducing hallucinations observed in “closed-book” LLM generation [ 14 , 16 , 10 , 18 ] . Document-level citations for supporting facts in LLM-generated answers are integral to any deployed RAG system. Therefore, support evaluation assesses whether a RAG answer factually supports the information present in the cited documents, which is crucial for evaluating the quality of a RAG system. Prior work on support evaluation in the RAG literature [ 17 , 5 , 26 , 9 , 29 , 31 , 25 ] relies on an automatic judge, i.e., an LLM as a proxy judge. However, it is unknown whether an LLM judge can potentially replace a human judge for support evaluation. This paper examines results from the TREC 2024 RAG Track, assessing 45 participant systems on 36 information-based queries. A sample query and answer is shown in Table 1 . We conducted a large-scale comparative study between human and LLM judges using resources provided by the National Institute of Standards and Technology (NIST) to better understand whether support assessment can be automated. Unique to the TREC setup, we contrast our automatic judgment process using a strong LLM judge (like GPT-4o) against a manual process under two conditions: (1) manual from scratch , where human annotators perform assessments from scratch and (2) manual with post-editing , where human annotators are shown GPT-4o predictions during the evaluation process. In this paper, we focus exclusively on support, i.e., whether the information in an answer sentence is supported by the cited documents, which we consider as the “ground truth”. This is, of course, only one aspect of RAG evaluation. For in-depth analyses of other aspects of the TREC 2024 RAG Track, we refer readers to Upadhyay et al. 2024 for relevance assessment and Pradeep et al. 2024b for nugget evaluation. Our experimental results indicate that GPT-4o and human judgments perfectly match 56% of the time in the manual from-scratch condition, increasing to 72% in the manual with post-editing condition. These results show promise in using LLM judges for support assessment in both conditions. We measured support of a system’s overall answer in terms of two metrics: weighted precision and weighted recall, where precision penalizes overcitation, and recall penalizes undercitation. We observe a high correlation at the run level (above 0.79 Kendall \tau ) between GPT-4o and human judges, providing evidence that LLMs can potentially replace human judges for support evaluation. In addition, to better understand the discrepancies between GPT-4o and human judges, we conducted an unbiased disagreement study with an independent human judge who carefully re-assessed 537 randomly sampled pairs, including both assessment conditions. Our results surprisingly show that the independent judge agrees more with GPT-4o than the human judge (e.g., Cohen’s \kappa of 0.27 vs. 0.07). Finally, we discuss annotation errors to help improve future iterations of support evaluation. Topic: how taylor swift’s age affects her relationships Answer : Taylor Swift’s age has significantly influenced her relationships, often contributing to both the dynamics and outcomes of her romances [ ] . Swift’s relationships have frequently involved notable age gaps, which have sometimes led to complications [ ] . For instance, her relationship with John Mayer, who was 11 years her senior, reportedly strained due to the age difference and Mayer’s reputation as a playboy, leading to Swift feeling taken advantage of [0, 3, 4] . This relationship inspired her song "Dear John," reflecting her emotional turmoil [3, 4] . Similarly, Swift’s relationship with Jake Gyllenhaal, who was nine years older, also highlighted the impact of age differences. Gyllenhaal reportedly felt the age gap, and Swift’s inexperience made it difficult for her to navigate the complexities of the relationship, leading to heartbreak and inspiring several songs on her "Red" album [8, 1, 2] . […] In summary, Taylor Swift’s age has played a crucial role in shaping her relationships, influencing both the dynamics and outcomes [8, 1] . Age differences have often led to complications, while relationships with peers have been relatively simpler. Over time, her maturity has allowed her to navigate romantic complexities with greater understanding and resilience. [ ] Passage [3]: A Timeline Of Taylor Swift’s Age-Inappropriate Romances | Business Insider […] Taylor Swift, 21, & John Mayer, 32. The inappropriateness of Swift’s dating habits peaked when she started seeing 11-years her senior and notorious playboy, John Mayer. Neither party confirmed the romance, but they reportedly dated through February 2010 […] Passage [8]: What Really Went Wrong With Jake Gyllenhaal And Taylor Swift? […] Taylor Swift and Jake Gyllenhaal dated from October to December 2010. […] What went wrong with these two “ twin fire signs? ” It turns out, a lot. Maybe that’s why it’s such a touchy subject for him these days! The age gap, when they dated, Gyllenhaal was 29 to Swift’s 20, and by the time they split, he was 30 to her 21. “ He said he could feel the age difference ,” a source told Us Weekly […] Table 1 : A sample RAG answer segmented into multiple sentences with citations to passages retrieved from the MS MARCO V2.1 segment collection for a query about Taylor Swift. We highlight the cited passages, which are provided as a list (that can be empty, i.e., zero citations). 2 Background and Related Work What we refer to as support evaluation represents an attempt to determine whether information in a RAG answer is factually supported by its cited documents. In our work, we evaluate support at the sentence level in the answer. We assume an answer r segmented into n sentences, r=\{a_{1},\cdots,a_{n}\} , where each answer sentence