Evaluating Contrastive Feedback for Effective User Simulations Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Develo

Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer · 2025

Read Evaluating Contrastive Feedback for Effective User Simulations Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Develo by Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02560.txt.

Introduction

Andreas Konstantin Kruff, Timo Breuer, and Philipp Schaer wrote a 2025 paper on simulating search users with large language models. Such simulations help researchers test search systems without running costly user studies. Yet how well the model users behave, and what shapes their behavior, is still debated.

The authors borrow an idea from contrastive training, where a model learns from good and bad examples side by side. In their setup, the simulated user gets summaries of documents already judged relevant, not relevant, or both. That extra context is meant to mimic the knowledge a real user builds up while searching. They compare these user setups to see which context mix gives the most realistic simulations. The findings guide how to brief model users in future studies.

Catalog record

CREATORAndreas Konstantin Kruff, Timo Breuer, Philipp Schaer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02560

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Contrastive Feedback for Effective User Simulations Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730189 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Users and interactive retrieval Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer Abstract . The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their application and capabilities remain highly debated and understudied. This study explores whether the underlying principles of contrastive training techniques, which have been effective for fine-tuning LLMs, can also be applied beneficially in the area of prompt engineering for user simulations. Previous research has shown that LLMs possess comprehensive world knowledge, which can be leveraged to provide accurate estimates of relevant documents. This study attempts to simulate a knowledge state by enhancing the model with additional implicit contextual information gained during the simulation. This approach enables the model to refine the scope of desired documents further. The primary objective of this study is to analyze how different modalities of contextual information influence the effectiveness of user simulations. Various user configurations were tested, where models are provided with summaries of already judged relevant, irrelevant, or both types of documents in a contrastive manner. The focus of this study is the assessment of the impact of the prompting techniques on the simulated user agent performance. We hereby lay the foundations for leveraging LLMs as part of more realistic simulated users. † † cc-license: by 1. Introduction The aim of this work is to evaluate the search effectiveness of LLM-based user agents by comparing different prompting strategies to simulate a human-like knowledge state. We hypothesize that, as demonstrated in other fine-tuning contexts, contrastive learning may enable the LLM to internalize task-specific distinctions more effectively, resulting in improved interaction capabilities compared to other prompting strategies presented here. LLM have shown a remarkable influence in current IR ( IR ) research. While some of their applications, like the ability of LLM regarding relevance judgments, are highly debated and controversial, other use cases, like LLM-based query reformulations, have shown their potential. Query reformulations can be used to increase retrieval performance ( Breuer 2024 ) , and the effectiveness of LLM-based query reformulations has been shown, especially when the LLM was allowed to rely on itself or when provided with additional TREC topic descriptions ( Engelmann et al. 2024 ) . Query reformulations are also a crucial step in modern user simulations ( Balog and Zhai 2024 ) and in related frameworks like SimIIR 3 ( Azzopardi et al. 2024 ) . Therefore, while still understudied, LLM show a high potential for user simulations. The knowledge state can be modeled by incorporating the information need that is typically described in a structured format given by the topic title, description, narrative, and furthermore, the relevance labels (qrels) of on IR test collection. However, we argue that in a real-world scenario, the users are uncertain about their explicit information needs ( Belkin 1980 ) , and the realism of the user simulation when proving the LLM with the explicitly formulated topic descriptions can be questioned. Instead, this study tries to iteratively update the simulated user’s knowledge state by summaries of already seen and judged documents. The study evaluates whether supplying summaries of relevant documents, irrelevant documents, or a combination of both aid the model in decision-making and query formulation. 2. Related Work Early attempts of user simulations in IIR ( IIR ) were characterized by the use of rule-based user agents. This includes, for example, the Common Interaction Model (CIM), which handles the different user interactions like query reformulation, relevance judgment or stopping behaviour by deterministc rules like stopping after a predefined number of irrelevant document ( Pääkkönen et al. 2017 ) . The same applies for the Complex Searcher Model (CSM) which was used in the first version of the SimIIR framework ( Maxwell and Azzopardi 2016 ; Maxwell et al. 2015 ) . While these approaches are transparent and allow easy interpretation, their effectiveness is limited, especially when trying to simulate real user behavior. Recent studies have shown the potential capabilities of LLM regarding their use in language understanding and interactive decision making tasks, which combine reasoning, acting, and planning. It has been shown that these capabilities can be further improved by applying approaches like chain-of-thought prompting or action plan generation ( Yao et al. 2023 ; Zhou et al. 2024 ) . These led to the recent development of extended frameworks like SimIIR 3 ( Azzopardi et al. 2024 ) and USimAgent ( Zhang et al. 2024b ) that allow the use LLM for simulating user interactions. The effectiveness of LLM-based compared to advanced rule-based interactions remains understudied. Recent studies have shown how contrastive learning approaches can improve the performance of LLM in various ways. These include, among others, fewer hallucinations in conversational usecases ( Sun et al. 2022 ) , improvements in few-shot text classification ( Zhang et al. 2024a ) or few-shot natural language understanding tasks ( Gunel et al. 2021 ) . These versatile improvements in fine-tuning LLM by contrastive learning led to the hypothesis that these findings are also applicable and transferable for improvements in prompting strategies for user simulations. Using LLM for certain one-shot-learning tasks is a recent and highly debated topic in the community. The usage of LLM for creating relevance judgments is of great importance to the IR community due to its crucial role in the whole evaluation process ( Soboroff 2025 ; Alaofi et al. 2024 ; Takehi et al. 2024 ) . However, this discussion is still ongoing and there are recent venues like the LLM4Eval SIGIR workshop that push the discussion forward with contributions like LLMJudge ( Rahmani et al. 2024 ) . We need to synthesize the relevance decision in our simulated IIR in some way or the other. Therefore, we take advantage of the new approaches but would like to point to the ongoing discussion. 3. Methodology Figure 1. A flowchart of the analyzed user configuration described in Section 3.1 . The color codes correspond to those in Figure 2 that show the experimental results in the simulated search sessions. In this study, different prompting strategies are evaluated regarding their potential to improve the user agent’s search effectiveness in simulated search sessions. In particular, we evaluate how the effectiveness of the LLM-based user agent changes if context information based on the already seen documents is added to pre-existing context information that represents the user’s knowledge state. In our simulations, the context information is obtained from the topic files of test collections. TREC test collections provide information needs in topic files alongside the document collection and the corresponding relevance judgments. These topic files are formatted in a de-facto standardized format and usually cover a title, a more detailed description, and an additional narrative to provide annotators with context information when making relevance judgments. Our simulated users are instantiated with different combinations of the topic file’s information sections by including them in

Read the complete text