Fine Grained Evaluation of LLMs-as-Judges DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Relevance assessment

Sourav Saha, Mandar Mitra · 2026

Read Fine Grained Evaluation of LLMs-as-Judges DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Relevance assessment by Sourav Saha, Mandar Mitra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08919.txt.

Introduction

Sourav Saha and Mandar Mitra grade chat models that act as judges of text and image work. Labs now use such judges in place of paid human marks. A judge score is cheap and fast. But a coarse score can hide where a judge is weak, such as on close calls or on one class of task.

The pair call for fine grade checks. A judge should be tested by task type, by gap size, and by type of flaw in the work it marks. Fine checks show if a judge can tell a near miss from a true hit. A reader who uses AI judges gets a stricter scale. It can stop a lab from trust in a judge that fails where it counts.

Catalog record

CREATORSourav Saha, Mandar Mitra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine Grained Evaluation of LLMs-as-Judges DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Relevance assessment Sourav Saha, Mandar Mitra Abstract . A good deal of recent research has focused on how Large Language Models (LLMs) may be used as ‘judges’ in place of humans to evaluate the quality of the output produced by various text / image processing systems. Within this broader context, a number of studies have investigated the specific question of how effectively LLMs can be used as relevance assessors for the standard ad hoc task in Information Retrieval (IR). We extend these studies by looking at additional questions. Most importantly, we use a Wikipedia based test collection created by the INEX initiative, and prompt LLMs to not only judge whether documents are relevant / non-relevant, but to highlight relevant passages in documents that it regards as useful. The human relevance assessors involved in creating this collection were given analogous instructions, i.e., they were asked to highlight all passages within a document that respond to the information need expressed in a query. This enables us to evaluate the quality of LLMs as judges not only at the document level, but to also quantify how often these ‘judges’ are right for the right reasons . Our findings suggest that LLMs-as-judges work best under human supervision. \setupctable botcap, captionskip=2pt, mincapwidth=2in, framebg=1 1 1, framefg=0 0 0, framerule=0pt, framesep=0pt, 1. Introduction Large Language Models (LLMs) have brought us closer than ever before to the objective of retrieving useful information (as opposed to documents or web-pages) in response to a natural language statement of a user’s information need (the prompt ). The “astoundingly human-like flowing text output” ( 25 ) produced as answers to natural language queries effectively creates the illusion that the user is communicating with an intelligent, very well-informed (and superhumanly efficient) person. This has led to a good deal of ongoing research on whether and how LLMs may replace human assessors or annotators ( 1 ; 11 ; 19 ; 21 ) when creating ‘ground truth’ or ‘gold standard’ data that is commonly used to evaluate information access systems. In this study, we focus on relevance assessments for the canonical ad hoc search task ( ( 28 ) , Section 3.1; ( 25 ) , Section 1) in Information Retrieval (IR). Recent work, e.g., ( 27 ; 28 ) , has hitherto focused on using LLMs to provide relevance judgments on a graded scale (typically comprising four levels: non-relevant, related, highly relevant, and perfectly relevant) for query-document pairs. These judgments are evaluated both directly, in terms of agreement / overlap with human-assigned labels, and indirectly, in terms of whether IR systems are ranked differently when evaluated using judgments provided by LLMs or humans. To the best of our knowledge, our study is the first to look beyond document-level relevance labels. We seek to extend earlier studies by investigating why an LLM regards a document as relevant to a query. A natural answer to this question may be obtained by asking the LLM to also label or highlight the parts of a document that it thinks are relevant for the information need contained in the query. Such highlighted passages may be regarded as rationales , a term that is commonly used in the literature on explainability ( 3 , esp. Section 9) ). For our experiments, we leverage the INEX 2009 and 2010 collections ( 14 ; 5 ) , which provide human-annotated ground-truth data on a Wikipedia dump for this task. We use a ‘few-shot’ setting by providing (via the prompt) a handful of examples illustrating what we want. Our experiments enable us to quantify how well LLMs agree with humans about which parts of a relevant document are actually useful, and thereby, to (partially) address the issue of whether LLMs are “right for the right reasons” ( 3 ; 4 ) . Interestingly, even though our experience as general users suggests that LLMs are very good at providing precise, focused answers to many of our information needs, our experimental results show that LLMs are not as impressive when it comes to finding (micro) needles in small haystacks. In other words, when only a small passage within a sizeable document is relevant for a certain query, LLMs are often unable to pinpoint these passages, and end up highlighting large portions of the document. Other results on this Wikipedia corpus are largely in agreement with those reported in recent studies on conventional TREC corpora ( 28 ; 26 ) : LLMs are promising, and may be used to subtantially reduce the amount of human involvement required to generate high-quality benchmark datasets, but they cannot replace humans as assessors. In the next section, we discuss the relation between recent studies and our work in greater detail. Section 3 provides some background about the INEX task and data that forms the basis of our experiments. It also describes the structure of our prompt. Section 4 provides more details about the experimental setup, and the LLMs used. Results are discussed in Section 5 . Finally, in Section 6 , we list a number of additional questions that we intend to explore in the near future. For reproducibility purposes, we release our codebase at https://anonymous.4open.science/r/llm-eval-2396/ . 2. Relationship between our work and recent research Early work in this area was initiated by discussions in 6 and 12 . One recent study ( 27 ) presented encouraging results by showing that LLM-based relevance assessors correlate strongly with trained human assessors, and generally do better than crowdsourcing. However, several researchers ( 28 ; 25 ) have raised both philosophical and empirical concerns that LLMs may be “overconfident” when predicting the relevance label of a document. Similarly, 2 showed that an LLM can be misled into predicting a document as “perfectly relevant,” even when it is not actually relevant to the given query. Therefore, relying heavily on automated relevance assessment frameworks remains risky. To address this issue, 26 propose to use LLMs more carefully. They present a method for calibrating LLM scores using a regression model trained on a few graded relevance assessments. Based on this calibration, we may estimate how confident an LLM is about a prediction. The eventual objective is to eliminate the need for human assessors for decisions that the LLM is confident about, while using human annotations in cases where the LLM exhibits low confidence. However, their work also considers relevance at the level of full documents only. All these studies work with TREC collections ( 29 ) for their experiments. While most of the documents in these collection are general-interest news articles, there are also some government documents (e.g., from the Federal Register). The search topics at TREC are also created by trained information analysts. The Wikipedia corpus along with the INEX queries may be more representative of general users. Another valid concern involves biases that may be inherent in LLMs used as judges. 22 propose to reduce biases from closed source models by using an aggregation framework to combine relevance judgments from different LLMs into a single one, but this issue needs more

Read the complete text