LLM-Driven Usefulness Judgment for Web Search Evaluation

Mouly Dewan, Jiqun Liu, Aditya Gautam, Chirag Shah · 2025

Read LLM-Driven Usefulness Judgment for Web Search Evaluation by Mouly Dewan, Jiqun Liu, Aditya Gautam, Chirag Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14401.txt.

Introduction

Mouly Dewan, Jiqun Liu, Aditya Gautam, and Chirag Shah use large language models to judge usefulness in web search evaluation, in a research paper from 2025. Standard evaluation leans on relevance labels, which say whether a page matches a query. Relevance alone misses whether the page helped the user reach their goal, so usefulness is a needed second gauge.

Their method builds rubrics and reasons over full search sessions, with personal and context signals. Models can produce moderate usefulness labels from such history. Fine-tuned models do better with structured session context. Tests also probe whether models can split relevance from usefulness where the two diverge. Key metrics for label quality are identified.

Catalog record

CREATORMouly Dewan, Jiqun Liu, Aditya Gautam, Chirag Shah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Driven Usefulness Judgment for Web Search Evaluation Mouly Dewan, Jiqun Liu, Aditya Gautam, Chirag Shah Abstract . Evaluation is fundamental to optimizing search experiences and supporting diverse user intents in Information Retrieval (IR). Traditional search evaluation methods primarily rely on relevance labels, which assess how well retrieved documents match a user’s query. However, relevance alone fails to capture a search system’s effectiveness in helping users achieve their goals, making usefulness a critical evaluation criterion. Recent LLM-enabled evaluation works have mostly focused on relevance label generation. In this paper, we explore an alternative approach: LLM-generated usefulness labels that incorporate implicit and explicit user behavior signals along with relevance. We introduce Task-aware Rubric-based Usefulness Evaluation (TRUE), a reproducible rubric-driven framework that leverages iterative sampling and Chain-of-Thought reasoning to model complex search behavior patterns. Our study shows that: (i) pre-trained LLMs can generate moderate usefulness labels with rich session-level context; and (ii) LLMs with TRUE outperform state-of-the-art methods and our systematically constructed baseline. We further analyze the relationship between usefulness and user satisfaction by comparing label generation with and without satisfaction signals, quantifying their correlation and impact on model behavior. Additionally, we conduct an ablation study to identify key features for accurate usefulness label generation, enabling cost-effective evaluation. Overall, this work advances LLM-based evaluation beyond relevance by proposing a reproducible and scalable framework for usefulness judgment, addressing key reproducibility challenges. Conference: Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 25, 2026; Melbourne, VIC, Australia Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (ICTIR ’26), July 25, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805713.3820429 ISBN: 979-8-4007-2600-2/2026/07 CCS: Information systems Language models † † cc-license: by 1. Introduction In IR evaluation scenarios, assessing relevance alone is not enough, as it often fails to capture the extent to which a document is useful for an underlying task ( Cole et al., 2009 ; Belkin et al., 2008 ) . While prior work argues that relevance is insufficient for evaluating search systems ( Cole et al., 2009 ; Liu, 2022 ) , mainstream IR has largely focused on objective relevance assessment. Traditional Cranfield paradigms ( Cormack et al., 1998 ) evaluate documents individually, ignoring broader task goals ( Belkin et al., 2008 ) and limiting applicability to interactive information retrieval. Relevance annotations alone cannot fully represent document usefulness or user satisfaction ( Belkin, 2015 ; Belkin et al., 2009 ; Mao et al., 2016b ) , and third-party annotators often lack users’ situational context, creating a situational disconnect . Prior studies show weak correlations between relevance metrics and user satisfaction ( Mao et al., 2016a ) . Usefulness judgment is deeply context-dependent, carrying richer contextual information than relevance but is also more challenging to capture in real time or reuse in evaluation experiments ( Liu and Shah, 2022 ; Jiang et al., 2026 ) . Therefore, our work incorporates usefulness as an evaluation metric to provide a more holistic assessment, capturing not only document relevance but also the user’s search intent and satisfaction throughout the search session ( Mitsui et al., 2017 ; Liu et al., 2020 ; Liu and Yu, 2021 ) . Moreover, traditional IR evaluation has relied on human assessors and crowd workers, which is time-consuming and costly ( Thomas et al., 2024 ) . Human evaluations are also difficult to reproduce and often not directly comparable across rounds due to inconsistent annotator pools ( Jiang et al., 2026 ) . To address these challenges, researchers have explored automated evaluation using large language models (LLMs) ( Dietz et al., 2025 ) . LLM-based evaluation (LLM4eval) ( Rahmani et al., 2024a ) has gained traction due to its strong reasoning and task-solving capabilities ( Wei et al., 2022b ; Wei et al., 2022a ) . However, most LLM-enabled IR evaluation efforts focus on TREC-style document relevance ( Rahmani et al., 2024b ; Chen et al., 2026 ) , while user-centric usefulness remains underexplored. Therefore, this paper evaluates LLM-generated usefulness labels that incorporate search session, task, query, and document-level user signals to better capture user perceptions of system quality. However, usefulness judgment poses challenges beyond those encountered in relevance evaluation. First, it is unclear how to effectively represent users’ search journeys to LLMs. Second, the choice of features and criteria that should guide usefulness prediction remains under-specified. Finally, LLM-based evaluation itself raises reproducibility concerns, as model outputs can vary across prompt formulations, inference settings, and model versions. Addressing these challenges requires the design of an evaluation framework specifically tailored to usefulness prediction. We address these challenges by incorporating both implicit and explicit user behavior signals into LLM-based usefulness evaluation. Implicit signals include Clicks, Click-Through Rate (CTR), Query Dwell Time, Task Dwell Time, and URL Dwell Time capture engagement, while explicit signals include task satisfaction, query satisfaction, task relevance, and query relevance. We also incorporate document-level features such as URL, title, summary from the search engine results page (SERP), ranking position, associated queries, and query position. These signals enable richer representations of users’ search journeys and provide concrete criteria for usefulness assessment. While satisfaction may not be available in real-world deployment and can be considered downstream of usefulness, we treat it as a post-hoc signal in an offline evaluation setting and explicitly analyze its impact on usefulness estimation. Most existing LLM-based judgment methods rely on prompt-based procedures that are highly sensitive to prompt design, often limiting reproducibility ( Farzi and Dietz, 2025 ) . To address this, we introduce TRUE (Task-aware Rubric-based Usefulness Evaluation), a rubric-driven framework that structures LLM reasoning over search context and behavioral signals. We compare TRUE with state-of-the-art LLM methods, traditional machine learning models, our own curated baseline, and session-aware methods using Chain-of-Thought (CoT) ( Wei et al., 2022b ) reasoning. This work is guided by the following RQs: • RQ1: Can LLMs generate reliable usefulness judgments when provided with representations of search sessions? • RQ2: Does a rubric-driven evaluation framework improve the performance of LLM-based usefulness assessment? • RQ3: What is the effect of user satisfaction signals on LLM-based usefulness judgments? • RQ4: What are the most effective features in improving LLMs’ document usefulness judgments? Motivated by these research questions, this work examines whether LLMs can assess usefulness beyond traditional relevance-based evaluation addressing key limitations in current IR methods. The novelty lies in introducing TRUE (Task-aware Rubric-based Usefulness Evaluation) , a rubric-driven reproducible method that integrates implicit and explicit user signals to evaluate document usefulness. The main contributions of our work include: • We propose TRUE , a structured and reproducible rubric-based framework for usefulness evaluation using LLMs. • We introduce a method to represent user search sessions by structuring implicit and explicit behavioral signals into LLM-compatible inputs. • We identify the most effective implicit and explicit user signals through an ablation study . • We analyze the impact of user satisfaction on usefulness judgments, evaluating robustness with and without them. 2. Background and Related Works Evaluation of search systems has long been central to IR research, with the goal of supporting and understanding users’ information needs. Traditional evaluation campaigns such as TREC, NTCIR, CLEF, and FIRE ( Carmel and Yom-Tov, 2010 ; Hauff,

Read the complete text