LLM-Driven Usefulness Labeling for IR Evaluation DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Information retrieval Evaluation of retrieval results Retrieval efficiency

Mouly Dewan, Jiqun Liu, Chirag Shah · 2025

Read LLM-Driven Usefulness Labeling for IR Evaluation DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Information retrieval Evaluation of retrieval results Retrieval efficiency by Mouly Dewan, Jiqun Liu, Chirag Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08965.txt.

Introduction

Mouly Dewan, Jiqun Liu, and Chirag Shah test smart labels for search results in a 2025 study. Judging search tools takes labels on how well each page fits a user's need. Crowd workers once made those labels by hand, a slow and costly path. Plain fit labels also miss the user's true task and aim.

The team asked large language models to grade how useful each page is for its task. Their tests fed the models task, query, page, and click clues. Pre-trained models gave fair grades on their own. The work points to cheap, task-aware labels for search tests.

Catalog record

CREATORMouly Dewan, Jiqun Liu, Chirag Shah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Driven Usefulness Labeling for IR Evaluation DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Information retrieval Evaluation of retrieval results Retrieval efficiency Mouly Dewan, Jiqun Liu, Chirag Shah Abstract . In the information retrieval (IR) domain, evaluation plays a crucial role in optimizing search experiences and supporting diverse user intents. In the recent LLM era, research has been conducted to automate document relevance labels, as these labels have traditionally been assigned by crowd-sourced workers - a process that is both time and consuming and costly. This study focuses on LLM-generated usefulness labels, a crucial evaluation metric that considers the user’s search intents and task objectives, an aspect where relevance falls short. Our experiment utilizes task-level, query-level, and document-level features along with user search behavior signals, which are essential in defining the usefulness of a document. Our research finds that (i) pre-trained LLMs can generate moderate usefulness labels by understanding the comprehensive search task session, (ii) pre-trained LLMs perform better judgement in short search sessions when provided with search session contexts. Additionally, we investigated whether LLMs can capture the unique divergence between relevance and usefulness, along with conducting an ablation study to identify the most critical metrics for accurate usefulness label generation. In conclusion, this work explores LLM-generated usefulness labels by evaluating critical metrics and optimizing for practicality in real-world settings. 1. Introduction Assessing relevance is not enough for many IR scenarios, as in many cases it cannot fully capture the extent to which a document is actually useful for addressing the motivating tasks. Although many scholars have advocated for this over decades ( Cole et al. 2009 ; Liu 2022 ) , mainstream IR has remained focused on the seemingly objective assessment of relevance, following the traditional Cranfield evaluation paradigm ( Cormack et al. 1998 ) . Relevance is often used as a proxy for other constructs due to its strong correlation with user-side evaluation labels like usefulness and satisfaction ( Mao et al. 2016a ) . However, usefulness is tied to both task context and users’ context-dependent preferences, making it a richer but more challenging construct to capture in real time. For Web search evaluations, relevance-based metrics alone cannot capture the user’s search goal and satisfaction, especially during complex task-driven sessions ( Belkin et al. 2008 ; Liu 2021 ) . Therefore, incorporating usefulness as an evaluation metric provides a more holistic assessment, capturing not only document relevance, but also user search intent and satisfaction throughout sessions ( Mitsui et al. 2017 ; Liu et al. 2020 ) . To mitigate time and costly 1 1 1 As a concrete example, for the 50 topics in the TREC-8 Ad Hoc track ( Voorhees and Harman 2000 ) , 129 participating systems led to more than 86,000 pooled documents to judge, requiring more than 700 assessor hours at a cost of about USD 15,000 ( Thomas et al. 2024 ) . constraints of traditional evaluation challenges ( Thomas et al. 2024 ) , IR researchers have been exploring the possibility and techniques of automated search evaluation with LLMs ( Faggioli et al. 2023 ; Chen et al. 2024 ; Alaofi et al. 2024 ) . LLMs for automated evaluation (LLM4eval) have recently attracted growing attention in IR community because of their flexibility and increasing task-solving capabilities ( Rahmani et al. 2024a ) . In LLM-automated evaluations, researchers have primarily focused on relevance labels, while user-centric usefulness metrics remain largely unexplored. As usefulness is a measure that directly reflects a document’s contribution to achieving user goals , in this paper we particularly focus on usefulness as the key metric for evaluating Web search along with relevance. To achieve this, we generate usefulness labels using LLMs and employ search session, task-level, query-level and document-level user metrics, aiming to better capture how users perceive the quality of search systems. We also seek to explore whether LLMs can capture the diverging relationship between relevance and usefulness. In some cases, a retrieved document may be relevant to the query but not necessarily useful to the user, or it may be irrelevant to the query yet highly valuable for completing the task. Being able to identify and address such situations can open up new possibilities for enhancing IR applications, especially in complex, open-ended tasks. Existing ML-based approaches for predicting usefulness often overlook the broader task and user contexts. This is the place where LLMs can step in with their are pre-trained existing knowledge to better understand user, task, and query context through appropriate prompting. The purpose of generating the usefulness labels is to gauge how well and accurately these LLMs can perform in automated usefulness judgment when given all relevant features (both online and offline) ( Chen et al. 2017 ; Zhang et al. 2018 ; Liu et al. 2018 ) . Specifically, our research seeks to answer the following three research questions (RQs): RQ1 : To what extent can large language models (LLMs) accurately label document usefulness in search sessions? RQ2 : To what extent LLMs understand the comprehensive search goal of a user and navigate the divergence between relevance and usefulness of the retrieved documents? RQ3 : What are the most effective search metrics in improving LLMs’ document usefulness judgments? 2. Related Works In LLM automated search engine evaluation, most of the research has solely focused on automating relevance labels ( Abbasiantaeb et al. 2024 ; Faggioli et al. 2023 ; Rahmani et al. 2024b ; Upadhyay et al. 2024a ) , where else user-centric usefulness metrics have not been explored yet. Faggioli et al. ( Faggioli et al. 2023 ) were among the first to examine how LLMs can be used for relevance judgment, offering unique insights into human-LLM collaboration. Their work explored direct prompting strategies, highlighting both the potential and limitations of using LLMs to generate relevance labels. Another prominent work in LLM-automated relevance assessment by Thomas et al. ( Thomas et al. 2024 ) asserted that LLMs can accurately predict searcher’s preferences and perform relevance labeling as accurately as human labelers. LLMs have also been used to fill relevance judgment gaps ( Abbasiantaeb et al. 2024 ) and, to some extent, studied for utility judgment using retrieval-augmented generation (RAG) ( Zhang et al. 2024 ; Salemi and Zamani 2024 ) . To address the limitations of Thomas et al. ( Thomas et al. 2024 ) , Upadhyay et al. ( Upadhyay et al. 2024a ) conducted large-scale relevance assessment on DL TREC tracks using UMBRELLA ( Upadhyay et al. 2024b ) , an LLM-based relevance assessor for academic TREC-style evaluation. Most LLM-automated IR evaluations have focused solely on relevance labeling, but relevance alone is insufficient, as usefulness directly reflects user goal achievement ( Belkin et al. 2008 ; Cole et al. 2009 ) . In this work, we explore usefulness as a key metric for web search evaluation alongside relevance. We generate usefulness labels using LLMs, incorporating implicit user search behavior signals like CTR and dwell time ( Chuklin et al. 2013 ; Chapelle et al. 2009 ; Jiang et al. 2015 ) . Additionally, we examine whether LLMs can capture contextual relationships in search sessions by integrating relevance and satisfaction labels to assess search system quality while exploring the contrast between

Read the complete text