Improving the Reusability of Conversational Search Test Collections

Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, Mohammad Aliannejadi · 2025

Read Improving the Reusability of Conversational Search Test Collections by Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, Mohammad Aliannejadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09899.txt.

Introduction

Zahra Abbasiantaeb and co-authors tackle stale test collections for conversational search in this 2025 paper. Search benchmarks judge systems against pre-judged document pools. New systems surface unjudged documents, called holes, and get scored unfairly. Chat-style search makes bigger holes, since needs shift each turn and answers vary widely. The team fills holes with language model judgments, guided by existing human ones.

Experiments on two TREC collections show reuse drops in deeper conversation turns. A fine-tuned open model agreed closely with human judges. A prompted commercial model agreed poorly, and filling with it warped rankings. Filling with the tuned model restored fairer comparisons. The authors offer the method as upkeep for living search benchmarks.

Catalog record

CREATORZahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, Mohammad Aliannejadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-09899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving the Reusability of Conversational Search Test Collections Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, Mohammad Aliannejadi Abstract Incomplete relevance judgments limit the reusability of test collections. When new systems are compared to previous systems that contributed to the pool, they often face a disadvantage. This is due to pockets of unjudged documents (called “holes”) in the test collection that the new systems return. The very nature of Conversational Search (CS) means that these holes are potentially larger and more problematic when evaluating systems. In this paper, we aim to extend CS test collections by employing Large Language Models (LLMs) to fill holes by leveraging existing judgments. We explore this problem using TREC iKAT 23 and TREC CAsT 22 collections, where information needs are highly dynamic and the responses are much more varied – leaving bigger holes to fill. Our experiments reveal that CS collections show a trend towards less reusability in deeper turns. Also, fine-tuning the Llama-3.1 model leads to high agreement with human assessors, while few-shot prompting the GPT-3.5 results in low agreement with humans. Consequently, filling the holes of a new system using GPT-3.5 leads to higher change in the location of the new system. While regenerating the assessment pool with few-shot prompting the GPT-3.5 model and using it for evaluation achieves a high rank correlation with human-assessed pools. We show that filling the holes using few-shot training the Llama-3.1 model enables a fairer comparison between the new system and the systems contributed to the pool. Our hole-filling model based on few-shot training of the Llama-3.1 model can improve the reusability of test collections. 1 Introduction and Background Building reusable test collections in a cost-efficient manner to evaluate current and future systems has been a long-standing challenge in the field of Information Retrieval (IR) [ 13 , 43 ] . The predominant strategy for creating such collections has been through the use of pooling [ 43 , 36 ] – where the top-k documents from each contributing system are assessed for relevance. This approach is considered an acceptable compromise, moving away from the “ ideal test collection ” with complete relevance judgments, which is neither feasible nor practical. While the pooling strategy is fairly robust [ 14 , 9 , 34 , 21 ] , it leads to various evaluation biases (e.g., [ 7 , 9 ] ) where systems that did not contribute to the pool, can be significantly disadvantaged. This is because documents that haven’t been judged are considered irrelevant. Therefore, the fewer judged documents returned in a ranking, the lower the retrieval performance ceiling [ 8 ] . However, the fewer the judgments required to compare systems the cheaper the test collection. Researchers have tried to address these trade-offs by proposing new metrics and methodologies for compensating for the unjudged documents (e.g., [ 9 , 5 , 8 ] ), and/or developing new pooling and judgment strategies (e.g., [ 10 , 20 , 28 ] ). Either way, “holes” in the pools still remain [ 42 , 22 ] . Various studies have proven that the level of missing judgment does not affect the final ranking significantly [ 15 , 41 ] . However, with new advanced ranking models, nowadays, it is more likely that the majority of the top-ranked passages are unjudged. However, with the advances in the development of powerful LLM and other neural-based models, new opportunities have emerged for creating scalable, robust, and reusable test collections at a lower cost. . LLM provide the possibility to: (1) assess large volumes of documents reasonably cheaply, especially compared to human judgments [ 35 ] , (2) consistent yet potentially biased evaluations [ 3 ] , (3) collect judgments under the same conditions at different times, if the LLM and prompts are fixed and shared. (4) typically achieve higher quality than “typical” crowd workers [ 2 ] . Indeed, recent studies [ 25 , 19 , 37 , 18 , 33 , 39 , 40 ] have shown the effectiveness of using LLM to generate relevance judgments in the scenario of ad-hoc search automatically. These works demonstrate that LLM-based judgments exhibit a high correlation with human judgments. Existing works [ 37 , 18 ] prompted commercial LLM (e.g., GPT-3.5/4) to generate relevance judgments. However, commercial LLM come with limitations like non-reproducibility, non-deterministic outputs, and potential data leakage between training and evaluation data, which impend their utility in scientific research [ 31 ] . MacAvaney and Soldaini [22] and Khramtsova et al. [19] prompted an OS ( OS ) LLM , Flan-T5 [ 11 ] , for generating relevance judgments. While OS LLM are less effective, they do offer the potential for the development of reproducible and reusable test collections at scale. This led to efforts by Meng et al. [25] , who fine-tuned the Llama [ 38 ] model to better condition the LLM for performing the task of assigning relevance judgments. They found that fine-tuned Llama-3 leads to better agreement with human annotators than GPT-3.5 [ 18 ] . Further suggesting that more complete test collections could be produced, with high quality, for lower cost. Upadhyay et al. [40] introduced UMbrela, an OS toolkit designed for relevance assessment leveraging OpenAI models. UMbrela is designed based on the research by Thomas et al. [37] and is employed for a large-scale relevance evaluation in the TREC RAG competition [ 39 ] , demonstrating a strong correlation with human assessors. While this prospect is very appealing, it is fraught with new, unexplored challenges. For example, Alaofi et al. [3] demonstrated that randomly inserting query words into passages can mislead LLM , causing them to incorrectly assess irrelevant passages as relevant. Of interest, in this work is the notion of grounding. Training systems on the judgments of LLM , and then evaluating those systems on subsequent test collections, based on judgments from LLM , creates a potentially dangerous cycle that may amplify and re-enforce existing biases inherent in LLM [ 12 ] . Grounding the LLM -based judgments given the human judgments provides a mechanism to condition the LLM to be more aligned with human annotators – serving as a means to avoid the hypothesized AI doom loop [ 30 ] . To this end, MacAvaney and Soldaini [22] focus on a setting where the LLM is provided with one relevant example in the prompt to help ground the subsequent judgments. In this paper, we draw upon this direction in the context of CS ( CS ) where we aim to build and augment test collections with grounded LLM -based judgments. CS is defined as responding to the user’s information needs in the context of the conversation [ 32 , 6 , 24 , 23 , 26 , 27 ] . In CS the user’s information need depends on the query, the context of the conversation, and the user’s personal preferences [ 4 ] . This results in highly dynamic, non-linear conversational trajectories, where a user’s information need can be addressed differently depending on the system’s interpretation, as those needs evolve and change in response to the presented information. Therefore, to address such dynamic information needs, systems can follow many different paths. As a result, they retrieve a

Read the complete text