Criteria-Based LLM Relevance Judgments

Naghmeh Farzi, Laura Dietz · 2025

Read Criteria-Based LLM Relevance Judgments by Naghmeh Farzi, Laura Dietz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09488.txt.

Introduction

Naghmeh Farzi and Laura Dietz improve machine-made relevance labels in a 2025 paper. Judging which search results are relevant is central to testing search engines, but human labels are slow and pricey. Large language models can label at scale, yet free-form prompting yields wrong and hard-to-read judgments.

Their framework splits relevance into named criteria such as exactness, coverage, topic fit, and context fit, then grades each one. Tests on three standard data sets, including two TREC tracks, show the split judgments rank systems better than direct grading. The authors also map where the method helps and where it lags, to guide future automatic evaluation.

Catalog record

CREATORNaghmeh Farzi, Laura Dietz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09488

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Criteria-Based LLM Relevance Judgments Naghmeh Farzi, Laura Dietz Abstract . Relevance judgments are crucial for evaluating information retrieval systems, but traditional human-annotated labels are time-consuming and expensive. As a result, many researchers turn to automatic alternatives to accelerate method development. Among these, Large Language Models (LLMs) provide a scalable solution by generating relevance labels directly through prompting. However, prompting an LLM for a relevance label without constraints often results in not only incorrect predictions but also outputs that are difficult for humans to interpret. We propose the Multi-Criteria framework for LLM-based relevance judgments, decomposing the notion of relevance into multiple criteria---such as exactness, coverage, topicality, and contextual fit---to improve the robustness and interpretability of retrieval evaluations compared to direct grading methods. We validate this approach on three datasets: the TREC Deep Learning tracks from 2019 and 2020, as well as LLMJudge (based on TREC DL 2023). Our results demonstrate that Multi-Criteria judgments enhance the system ranking/leaderboard performance. Moreover, we highlight the strengths and limitations of this approach relative to direct grading approaches, offering insights that can guide the development of future automatic evaluation frameworks in information retrieval. 1 1 1 Online Appendix at https://github.com/TREMA-UNH/appendix-criteria-based-llm-relevance-judgments/ 1. Introduction The evaluation of information retrieval (IR) systems relies on relevance judgments to assess the quality of retrieved passages. By determining which pieces of information are relevant to specific information needs (queries), we derive evaluation scores for different IR systems. These scores allow us to compare systems and select the most effective approaches. For a benchmark of queries and a corpus, the evaluation process depends on having relevance labels for each query and document (or passage) pair in the corpus. While human-assigned relevance labels remain the gold standard, their acquisition at scale presents significant challenges in terms of cost, time, and consistency. Ultimately, only human judges can be the arbitrator of relevance ( Faggioli2023PerspectivesOL ; soboroff2024dontusellmsmake ; clarke_llm-based_2024 ) , but we aim to provide methods to support the rapid development of new information retrieval approaches. We study the task of automatically predicting passage-level relevance labels that can be used with standard IR evaluation tools. Task Statement. Given a query and a passage, predict the relevance on a graded scale. The objective is to produce relevance labels that, when used to evaluate a set of retrieval systems, yield a system ranking (i.e., leaderboard) similar to the one derived from human-annotated judgments. Example Domain. While our formulation is task-agnostic, all experiments are conducted on the TREC Deep Learning (DL) benchmark. The TREC DL uses a 0–3 relevance scale, where 3 indicates perfect relevance and 0 denotes no relevance. Retrieval systems submitted to this benchmark are ranked using gold-standard labels. Our goal is to predict relevance labels such that the resulting leaderboard closely mirrors that of the human judgments. Figure 1. Overview the two phases of Multi-Criteria: grading criteria individually, then aggregating grades (see Section 3 ). Two-phase diagram showing criteria-specific grading followed by grade aggregation. Individual criteria like exactness are evaluated first, then combined using prompts or summation. Issues with the current LLM Judge approaches. Recent advances in large language models (LLMs) have shown promising potential for automated relevance assessment ( Sun2023IsCG ; MacAvaney2023OneShotLF ; Thomas2023LargeLM ; Faggioli2023PerspectivesOL ) . These approaches are based on the careful design of one LLM prompt, which for a given query-document pair can obtain a relevance label. Moreover, they have demonstrated strong empirical results in terms of leaderboard correlation, regularly obtaining Kendall’s tau above 0.9, and an inter-annotator agreement Cohen’s kappa of 0.40. Grading with a single prompt, where the model is merely asked, “Is this document relevant?”, comes with notable challenges. First, LLMs can sometimes produce unpredictable results, especially if the prompt design is not carefully controlled. Subtle variations in how a prompt is phrased may yield drastically different relevance predictions. What is worse is that this behavior changes across LLM families and versions. Second, the single-label output (“relevant” or “not relevant”) obscures the rationale behind the decision, limiting the transparency of the evaluation process. In IR and many other fields, interpretability is often vital: understanding why a decision was made can help diagnose systematic biases, guide system improvements, and build confidence among stakeholders. Finally, while LLMs may have been exposed to enormous corpora, their notions of relevance could deviate from established IR conventions or human judgments if they rely on obscure patterns in their training data. Thus, while single-prompt approaches facilitate labeling large datasets, they are met with skepticism regarding their reliability, primarily due to a lack of transparency in the underlying reasoning. Contribution. To enable a more structured assessment, we propose a framework that evaluates relevance by decomposing it into multiple pre-defined criteria. Our approach assesses each criterion independently and then combines these evaluations to derive a final relevance label for the documents. While inspired by Chain-of-Thought (CoT) prompting ( Wei2022ChainOT ) , our framework takes a distinctive approach by evaluating each criterion independently and in parallel, rather than following a sequential process. This approach allows us to investigate whether decomposing relevance into criteria can both improve evaluation quality and maintain competitiveness with existing LLM-based methods, while also making the grading process more transparent to human evaluators. We compare our method with recent frameworks, such as UMBRELA ( upadhyay2024umbrelaumbrelaopensourcereproduction ) , which demonstrates better performance with larger LLMs ( farzi2025umbrela ) . Our results show that, even with smaller LLMs, our approach provides equally effective evaluations. 2. Related Work 2.1. LLM Based Relevance Judgment Recent work has explored using LLMs for automated relevance assessment. Many approaches rely on a single, hand-crafted prompt that the LLM uses to generate a relevance label. Instances of this approach are as follows. Thomas et al. ( Thomas2023LargeLM ) demonstrate that LLMs can effectively generate relevance judgments through direct prompting, establishing a baseline for automated evaluation methods, like the UMBRELA framework ( upadhyay2024umbrelaumbrelaopensourcereproduction ) . Farzi and Dietz ( farzi2025umbrela ) show that achieving optimal results with UMBRELA requires large LLMs, such as GPT-4. Arabzadeh and Clarke ( arXiv:2504.12558 ) compare LLM-based methods (binary, graded, pairwise, and nugget-based) on TREC DL ( dl19 ; dl20 ) and ANTIQUE ( 10.1007/978-3-030-45442-5_21 ) datasets, finding pairwise preferences best align with human labels while binary and graded (UMBRELA) methods achieve higher system ranking correlations. Sun et al. ( Sun2023IsCG ) show zero-shot capabilities for relevance assessments. MacAvaney et al. ( MacAvaney2023OneShotLF ) suggest one-shot learning for binary preference judgments. While Faggioli et al. ( Faggioli2023PerspectivesOL ) show that LLM-based relevance assessment can be effective, they also warn of several pitfalls, including a potential mismatch between predicted labels and human-perceived relevance. Beyond prompting, Zhang et al. ( zhang2024largelanguagemodelsgood ) introduce a utility-based framework for QA datasets, outperforming relevance judgments. Meng et al. ( meng2024qppgenre ) introduced QPP-GenRE based on fine-tuning LLMs to generate relevance judgments for query performance prediction on TREC DL datasets. However, Alaofi et al. ( Alaofi ) show LLMs misclassify irrelevant passages due to lexical overlap, and Abbasiantaeb et al. ( abbasiantaeb2024uselargelanguagemodels ) find that LLM-generated relevance judgments in conversational search tasks distort rankings when mixed with partial human annotations. These biases and inconsistencies show that single, coarse-grained metrics often fail to capture nuanced judgments. 2.2. Grading Rubrics and Relevance Nuggets Recent advancements

Read the complete text