Can LLM Annotations Replace User Clicks for Learning to Rank? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 457 CCS: Information systems Retrieval models and rankin

Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng · 2025

Read Can LLM Annotations Replace User Clicks for Learning to Rank? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 457 CCS: Information systems Retrieval models and rankin by Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06635.txt.

Introduction

Lulu Yu and colleagues ask whether labels written by language models can replace real user clicks when training search ranking. Click logs are cheap but noisy, human labels are dear, and model-written relevance judgments sit between the two. A head-to-head comparison across many dimensions had been missing.

On a public dataset and an industrial one, click-trained models won on frequent queries while model-label training won on medium and rare queries. Clicks captured document quality signals better, while model labels captured semantic matching better. Two combined training schemes, data scheduling and frequency-aware multi-objective learning, lifted ranking at every query frequency, with the frequency-aware version stronger.

Catalog record

CREATORLulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06635

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLM Annotations Replace User Clicks for Learning to Rank? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 457 CCS: Information systems Retrieval models and ranking Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng Abstract . Large-scale supervised data is essential for training modern ranking models, but obtaining high-quality human annotations is costly. Click data has been widely used as a low-cost alternative, and with recent advances in large language models (LLMs), LLM-based relevance annotation has emerged as another promising annotation. This paper investigates whether LLM annotations can replace click data for learning to rank (LTR) by conducting a comprehensive comparison across multiple dimensions. Experiments on both a public dataset, TianGong-ST, and an industrial dataset, Baidu-Click, show that click-supervised models perform better on high-frequency queries, while LLM annotation-supervised models are more effective on medium- and low-frequency queries. Further analysis shows that click-supervised models are better at capturing document-level signals such as authority or quality, while LLM annotation-supervised models are more effective at modeling semantic matching between queries and documents and at distinguishing relevant from non-relevant documents. Motivated by these observations, we explore two training strategies—data scheduling and frequency-aware multi-objective learning—that integrate both supervision signals. Both approaches enhance ranking performance across queries at all frequency levels, with the latter being more effective. Our code is available at https://github.com/Trustworthy-Information-Access/LLMAnn_Click . 1. Introduction Learning to rank (LTR) plays a crucial role in real-world scenarios, especially in search engines. The goal of LTR is to train a ranking model that can sort documents based on their relevance to a given query. Early LTR models are typically tree-based models such as LambdaMART ( 9 ) , which rely on hand-crafted features extracted from query-document pairs, including traditional LTR features like BM25 ( 35 ) . With the rapid progress of deep learning—particularly the emergence of Transformer architectures ( 44 ) —modern ranking models are often implemented as cross-encoders ( 6 ; 47 ) , which jointly encode the query and document. However, these models involve a large number of parameters, making them data-hungry during training. Since obtaining large-scale human annotations is expensive, alternative sources of low-cost supervision have been explored. Click data, readily available in real-world search logs, has emerged as a practical source of relevance annotations. Search logs contain abundant user interaction signals, among which click signals are the most straightforward and widely used. Clicks serve as implicit indicators of relevance between queries and documents, and they can be collected at scale with low cost. However, click signals are inherently biased ( 18 ) due to various factors such as position bias ( 5 ; 24 ; 46 ) , selection bias ( 2 ; 43 ; 42 ) , and trust bias ( 29 ; 45 ; 54 ) . This has led to extensive research in Unbiased learning to rank (ULTR), which aims to mitigate these biases and obtain unbiased ranking models. Due to the lack of publicly available real-world click datasets, most ULTR methods are evaluated on simulated click data, where clicks are generated according to assumed user browsing behavior ( 34 ; 41 ) . In contrast, real-world clicks result from more complex and nuanced user browsing behavior ( 19 ; 48 ; 56 ) . Therefore, developing effective methods to debias real-world click data and obtain unbiased ranking models remains a significant challenge. With the rapid development of large language models (LLMs), using LLMs for relevance annotation has become increasingly promising ( 16 ; 33 ; 39 ; 52 ) . Like click data, LLM annotations also offer a low-cost alternative for relevance annotation compared to costly human annotations. Considering that LLM annotations tend to have lower noise than click data, it is natural to ask whether LLM annotations could potentially replace click data for LTR. To explore this question, we first conduct a comparative analysis of their annotation characteristics, as illustrated in Fig. 1 . The most significant differences lie in two aspects: 1) LLMs provide greater flexibility in generating annotations; and 2) the criteria for relevance judgment differ. Click data can capture not only the semantic matching between queries and documents, but also document-level signals such as authority and quality. In contrast, LLM annotations are highly effective at capturing semantic relevance between query-document pairs, but are less sensitive to document-level signals. Figure 1 . Comparison of annotation characteristics between click data and LLM annotations We design three annotation strategies using LLMs: generating multi-level annotations based on given annotation guidelines, directly ranking results, and having LLMs directly select relevant results. Experimental results show that, under the given annotation guidelines, using listwise input of query-document pairs for annotation leads to the best model performance. Considering that search engines perform better on high-frequency queries ( 56 ) , returning more relevant results, it is crucial to further distinguish relevance using document-level signals. Therefore, we conduct a comparative analysis of ranking models trained on click data and LLM annotations across queries of different frequencies, and further explore their capabilities by introducing LTR features from different levels (query-document level and document-level) ( 4 ; 10 ; 35 ; 50 ) , shedding light on the complementarity between LTR features and ranking models trained with either click data or LLM annotations. The experiments demonstrate that models trained on click data perform better on high-frequency queries, while models trained on LLM annotations have an advantage on medium- and low-frequency queries. The experiments involving the introduction of different levels of LTR features further confirm that models trained on click data could capture document-level signals but are less effective in capturing semantic matching between queries and documents compared to models trained on LLM annotations. This suggests that query-document semantic features help address the limitations of click-trained models, while document-level signals can enhance the performance of models trained with LLM annotations. Moreover, given the high proportion of relevant results in the training set, it is unclear whether the ranking models truly possess the ability to distinguish between relevant and irrelevant results. To address this, we introduce true negative samples to investigate the relevance discrimination capabilities of ranking models trained on these two types of annotations. The experiments indicate that models trained on LLM annotations have a stronger ability to distinguish between relevant and irrelevant results compared to those trained on click data. Additionally, introducing true negative samples into the training set improves the ability of models trained on click data to distinguish between relevant and irrelevant results, and further enhances their capability to differentiate the degree of relevance among relevant results. Based on the comparative analysis above, click data and LLM annotations each have their advantages for queries of different frequencies. To leverage the strengths of both, we propose two methods to combine click data and LLM annotations: 1) Data Scheduling: since LLM annotations are of higher quality and have a stronger ability to capture semantic matching between queries and documents—which is fundamental for relevance judgment—we use the ranking model trained on LLM annotations for warm start. To prevent forgetting during the training process, we further perform interleaved training using click data from high-frequency queries and

Read the complete text