Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search

Jayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy · 2025

Read Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search by Jayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15990.txt.

Introduction

Jayant Sachdev and co-authors automate relevance labels for e-commerce search in a 2025 paper. Ranking products well needs large sets of judged query-product pairs. Human labeling is slow, costly, and error-prone at that scale.

The team tested several public and private language models as labelers. Prompt methods included step-by-step reasoning, worked examples, and retrieval with a diversity filter. On open and in-house data sets, model labels neared human accuracy at a slice of the time and cost. Labels made this way now feed live search evaluation at their company. The authors see wide use across search and recommendation work.

Catalog record

CREATORJayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15990

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search Jayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy Abstract . Accurate query-product relevance labeling is indispensable to generate ground truth dataset for search ranking in e-commerce. Traditional approaches for annotating query-product pairs rely on human-based labeling services, which is expensive, time-consuming and prone to errors. In this work, we explore the application of Large Language Models (LLMs) to automate query-product relevance labeling for large-scale e-commerce search. We use several publicly available and proprietary LLMs for this task, and conducted experiments on two open-source datasets and an in-house e-commerce search dataset. Using prompt engineering techniques such as Chain-of-Thought (CoT) prompting, In-context Learning (ICL), and Retrieval Augmented Generation (RAG) with Maximum Marginal Relevance (MMR), we show that LLM’s performance has the potential to approach human-level accuracy on this task in a fraction of the time and cost required by human-labelers, thereby suggesting that our approach is more efficient than the conventional methods. We have generated query-product relevance labels using LLMs at scale, and are using them for evaluating improvements to our search algorithms. Our work demonstrates the potential of LLMs to improve query-product relevance thus enhancing e-commerce search user experience. More importantly, this scalable alternative to human-annotation has significant implications for information retrieval domains including search and recommendation systems, where relevance scoring is crucial for optimizing the ranking of products and content to improve customer engagement and other conversion metrics. CCS: Computing methodologies Information extraction CCS: Information systems Relevance assessment CCS: Information systems Search engine indexing CCS: Information systems Novelty in information retrieval CCS: Computing methodologies Natural language generation CCS: Computing methodologies Language resources 1. Introduction When a customer performs a search on an e-commerce platform like Walmart, it is essential that the search results show the exact products or relevant products they are looking for, saving the customer time and effort while providing a positive shopping experience. The product search experience on e-commerce platforms is powered by a collection of machine learning models and algorithms across the stages of pre-retrieval, retrieval, and post-retrieval ranking and filtering ( Qiu et al., 2022 ) . Many of these models rely on accurate query-product (Q-P) relevance label data for training and testing. Specifically, these labels indicate how relevant a given product is to a customer search query. For example, the product “ Ergonomic PU leather high back office chair with flip-up armrest ” is highly relevant to the search query “ leather chair ”, while the product “ hard floor beveled edge chair mat ” is not relevant to the same query. (a) Previous approaches of acquiring Q-P relevance (b) Our approach to annotating Q-P relevance Figure 1 . Comparison of pipeline architectures for Q-P relevance annotation Figure 2 . System Architecture for Automated Q-P relevance with LLM and RAG We have worked with an external data labeling service which employs human labelers. The labelers were given Q-P pairs with detailed instructions and examples to guide labelers. They were not given any information on customer engagement data for Q-P pairs. Typically, three to five independent assessments are gathered per Q-P pair. Despite the guidelines, human labelers do not always reach consensus on an annotation, hence labeled data is often noisy. Furthermore, external human labeling is expensive and time-consuming, and hence only Q-P pairs corresponding to a small fraction of all queries can be labeled. Despite these challenges, external human labelers are considered as the most reliable method of acquiring data annotations, and is widely used across e-commerce and internet industries. Previously, Q-P relevance labels were obtained from clickstream data and human annotation as shown in Figure 1 (a). Higher engagement of a product given a search query would be considered a positive signal for relevance. However, cold-start products (such as newly introduced products and tail products that are sold infrequently) may lack sufficient impressions, click-through-rate (CTR), or add-to-cart (ATC) rate information due to their limited exposure to the customers. Furthermore, some products might get a lot of impressions and clicks even if they are not very relevant to the search query. For example, the product “ fresh lime ” gets a relatively high number of impressions and clicks for the query “ lemon ”, but it is not a very relevant product. Identifying the strong need for an automated framework for Q-P relevance labeling, in this work, we demonstrate that Large Language Models (LLMs) can be used successfully to obtain accurate relevance labels in a fully automated manner and at scale. A schematic of our approach is shown in Figure 1 (b), which we will detail in the subsequent sections. We have experimented with several LLMs for this task. We have explored different e-commerce search datasets, as well as ablation studies on the effectiveness of prompt engineering techniques including Chain-of-Thought (CoT) prompting, In-context Learning (ICL), and Retrieval Augmented Generation (RAG) with Maximum Marginal Relevance (MMR). Figure 2 shows Retrieval-Augmented Generation (RAG) architecture, designed and used in our work, which contains a retrieval component that retrieves relevant labeled query-product pairs for a given query-product pair. These relevant query-product pairs, providing the contextual information, are then used to create prompts which are provided as input to the LLMs. 2. Background This work is an application of data-centric Artificial Intelligence (AI), which is systematic engineering of the data used to build an AI system, including but not limited to automated data labeling, collection, augmentation, and cleaning ( Zha et al., 2023 ) . One approach to automated data labeling is called programmatic labeling which includes using human-defined labeling functions (LFs) that capture labeling logic to auto-generate large training sets ( Ratner et al., 2017 ) . Programmatic labeling significantly increases efficiency over manual labeling through automating the labeling process. However, labeling decisions still require human domain expertise on what heuristics should be applied ( Guan et al., 2023 ) . 2.1. Large Language Models Large Language Models can generate human-like text and demonstrate state-of-the-art logical reasoning ability across a broad range of reasoning tasks and benchmarks ( et al., 2023 ) . With advancements in the field, open-source models have also proven to be an effective alternative to GPT models in multiple reasoning tasks ( Vergho et al., 2024 ; Llama Team, 2024 ; Jiang et al., 2023 ; Jiang et al., 2024 ) . Prior work has shown that LLMs can be used to label data at par with human labeling on a variety of NLP tasks, with up to 96% saving in costs ( Wang et al., 2021 ) . LLMs are proven to be a viable substitute to crowd-sourced annotators and can match or surpass crowd-sourced annotators ( He et al., 2023 ) . Recent work has also explored the idea of using LLMs to annotate shopping relevance ( Soviero et al., 2024 ; Faggioli et al., 2023 ) . 2.2. Mathematical Formulation Let (Q,P) denote the set of all possible Q-P pairs. We define the task of relevance labeling as a classification problem with the candidate classes set \mathcal{Y}=\{y_{1},y_{2},\ldots,y_{n}\} . We want to find a map \Phi_{h}:(Q,P)\rightarrow\mathbb{R}^{\mathcal{Y}} that is able to ideally score the relevance for the Q-P pairs against the candidate classes. We assume Q and

Read the complete text