LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Sometime; Somewhere ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Evaluation of retrieval results
Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, Nick Craswell · 2025
Read LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Sometime; Somewhere ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Evaluation of retrieval results by Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, Nick Craswell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19076.txt.
Introduction
Laura Dietz and co-authors examine the risks of letting large language models grade the quality of search systems. Human judges once supplied those grades, and model grades often agree with them, tempting labs to drop humans entirely. The authors warn that the agreement can mislead, especially once model signals feed back into the systems being graded.
They describe repeated story patterns, or tropes, in current debate about model evaluation. In the risky pattern, the same model family helps build a system and then certifies its success, a loop that can lock in shared bias. The paper maps where validity breaks down. It urges care before human judgment is retired from evaluation.
Catalog record
| CREATOR | Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, Nick Craswell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Sometime; Somewhere ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Evaluation of retrieval results Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, Nick Craswell Abstract . Large Language Models (LLMs) are increasingly used to evaluate information retrieval (IR) systems, generating relevance judgments traditionally made by human assessors. Recent empirical studies suggest that LLM-based evaluations often align with human judgments, leading some to suggest that human judges may no longer be necessary, while others highlight concerns about judgment reliability, validity, and long-term impact. As IR systems begin incorporating LLM-generated signals, evaluation outcomes risk becoming self-reinforcing, potentially leading to misleading conclusions. This paper examines scenarios where LLM-evaluators may falsely indicate success, particularly when LLM-based judgments influence both system development and evaluation. We highlight key risks, including bias reinforcement, reproducibility challenges, and inconsistencies in assessment methodologies. To address these concerns, we propose tests to quantify adverse effects, guardrails, and a collaborative framework for constructing reusable test collections that integrate LLM judgments responsibly. By providing perspectives from academia and industry, this work aims to establish best practices for the principled use of LLMs in IR evaluation. 1. Introduction Large Language Models (LLMs) are increasingly used in the evaluation of information retrieval (IR) systems, generating relevance judgments that were traditionally the domain of human assessors. Given an information need (or topic) and a set of documents, assessors determine the relevance of each document to the topic. A process that forms the foundation of retrieval evaluation. However, due to the vast number of topic-document pairs, traditional assessment relies on pooling methods to identify a subset of documents for judgment. LLMs offer an alternative approach, with the potential to scale relevance assessments far beyond the limits of human annotation. However, Soboroff 2025 writes: “Letting the LLM write your truth data handicaps the evaluation by setting that LLM as a ceiling on performance.” In this paper we aim for a middle ground by discussing 13 ways in which LLMs can negatively impact the evaluation and how avoid this adverse effect. Pro. One of the most impactful advantages of LLM-based evaluation is speed. Unlike human assessors, who require coordination, training, and extensive annotation time, LLMs can generate relevance labels almost instantly. This dramatically lowers the cost of evaluation, making it possible to assess larger datasets, cover a wider range of retrieval tasks, and conduct evaluations more frequently. These benefits have led to the rapid adoption of LLMs in large-scale evaluation pipelines. Microsoft, for example, now uses OpenAI’s GPT models for relevance assessment in Bing ( Thomas et al. 2024b ) . More recently, Upadhyay et al. 2024c introduced Umbrela , an open-source toolkit based on a similar prompt that uses proprietary LLMs to label unjudged documents. Its application in a recent TREC task, further reinforces the notion that human assessors could be replaced ( Upadhyay et al. 2024a ) . Beyond labeling, LLMs have been proposed for fully synthetic test collections, where they replace human users in both query generation and relevance judgment ( Rahmani et al. 2024 ) . Con Although empirical studies demonstrate the effectiveness of LLM-based judgments, concerns remain regarding their reliability, validity, and long-term implications for IR evaluation ( Faggioli et al. 2023 ) . The increasing reliance on LLMs for test collection creation raises fundamental questions about reproducibility. At the same time traditional pooled judgment methodologies are becoming impractical for assessing generative and multi-modal systems. Furthermore, there is no consensus on how to mitigate the risks associated with these models, including biases, inconsistencies, and potential vulnerabilities such as susceptibility to query stuffing ( Alaofi et al. 2024b ) . Without a shared principled framework for responsible adoption, the unchecked use of LLMs in evaluation studies may lead to misleading conclusions, where prior work is cited without appropriate consideration of its limitations. Below we outline our contributions by specifying questions that are addressed in this paper. 1.1. Can I Use LLM-judgments in My Next Research Paper? Contributions This paper explores validity challenges posed by using LLMs as a evaluator, aims to codify the best practices for ensuring that LLM-based evaluation remains a valid experimentation approach for IR research. Maintaining scientific rigor requires identifying and recognizing the risks, implementing guardrails, and continuously confirming their validity with human judgments. Based on many discussions and reviewing the literature, we establish a taxonomy of reoccurring issues with LLM evaluations, which we refer to as LLM Evaluation Tropes (Figure 1 ). To this end, we outline guardrails that should be applied when using LLM-based evaluation for research publications. 1 1 1 We hope our recommendations offer a balanced compromise that satisfies both authors and reviewers. We underscore the truthfulness and relevance of these tropes by case studies from academia and industry. This paper is co-authored by a team of established international researchers from who hold a spectrum of viewpoints. 1.2. What is a Valid Experiment? We take valid LLM evaluators to mean evaluators whose measurements align with human intuition and quantify the utility for real people. We follow the guidance from Spärck Jones and van Rijsbergen 1975 : “It is apparent in particular that it is most important that the ideal collection(s) should be a means of relating valid abstract studies of information retrieval and those of operational systems and user behaviour.” In industry, the effectiveness of systems is typically validated through A/B testing and manual assessments. In academia, evaluation helps to determine which approaches constitute research advances and should be submitted to conferences and shared tasks. In both industry and academia, it is essential to obtain quantitative quality measures that credibly reflect relevance while mitigating unintentional biases, such as test data leakage, which may compromise the validity of drawn conclusions. In this paper, we discuss conditions under which the use of LLM evaluators may (inadvertently) threaten the validity of the experiment. Figure 1. LLM-evaluation tropes that can lead to invalid conclusions about evaluation, systems, and the efficacy of human judges that oversee the process. Overarching patterns are circularity \circlearrowright , Goodhart’s law \heartsuit , and loss of variety \cong . Evaluation Tropes: \circlearrowright #1 Circularity : Leaking the evaluation signal into the IR system. \circlearrowright #2 LLM-Eval as a Ranker : Using the same approach in the system and the evaluation. \heartsuit #3 LLM Narcissism : LLMs prefer text from their own model. \cong #4 Loss of Variety of Opinion : When all judges think alike. Meta-Evaluation Tropes: \heartsuit #5 Ignored Label Correlation : When human and LLM judges disagree on relevance labels. \cong #6 Old Systems : Evaluators need to identify the best systems of the future. #7 LLM Evolution : LLMs are not static; they can improve or degrade over time. System Tropes: \heartsuit #8 Test Set Leak : LLMs trained on test collections create the illusion of quality. \circlearrowright #9 Self-Training Collapse : Concept drift from training LLMs on LLM output. \heartsuit #10 Goodhart’s Overfitting : IR systems that are “trained for the test”. \heartsuit #11 Adversarial Threats : Bad actors want to manipulate the systems and evaluation. Judge Tropes: \cong #12 Rubber-Stamp Effect