Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models

Hongwei Shang, Nguyen Vo, Nitin Yadav, Tian Zhang, Ajit Puthenputhussery, Xunfan Cai, Shuyi Chen, Prijith Chandran, Changsung Kang · 2025

Read Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models by Hongwei Shang, Nguyen Vo, Nitin Yadav, Tian Zhang, Ajit Puthenputhussery, Xunfan Cai, Shuyi Chen, Prijith Chandran, Changsung Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07105.txt.

Introduction

Hongwei Shang and eight co-authors distill a large language model for Walmart search in two thousand twenty-five. Search results must match what shoppers mean by their queries. Big models rank well but reply too slowly for live use. The team moves that skill into a fast student model.

The teacher is first trained as a classifier with soft targets. The student then learns the margin between product pairs per query. Fresh unlabeled data, tagged by the teacher, grows the student set a lot. Student skill kept rising as that data grew. With enough of it, the student beat the teacher. It now runs live on the store site.

Catalog record

CREATORHongwei Shang, Nguyen Vo, Nitin Yadav, Tian Zhang, Ajit Puthenputhussery, Xunfan Cai, Shuyi Chen, Prijith Chandran, Changsung Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07105

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models Hongwei Shang, Nguyen Vo, Nitin Yadav, Tian Zhang, Ajit Puthenputhussery, Xunfan Cai, Shuyi Chen, Prijith Chandran, Changsung Kang Abstract . Ensuring the products displayed in e-commerce search results are relevant to users’ queries is crucial for improving the user experience. With their advanced semantic understanding, deep learning models have been widely used for relevance matching in search tasks. While large language models (LLMs) offer superior ranking capabilities, it is challenging to deploy LLMs in real-time systems due to the high-latency requirements. To leverage the ranking power of LLMs while meeting the low-latency demands of production systems, we propose a novel framework that distills a high-performing LLM into a more efficient, low-latency student model. To help the student model learn more effectively from the teacher model, we first train the teacher LLM as a classification model with soft targets. Then, we train the student model to capture the relevance margin between pairs of products for a given query using mean squared error loss. Instead of using the same training data as the teacher model, we significantly expand the student model’s dataset by generating unlabeled data and labeling it with the teacher model’s predictions. Experimental results show that the student model’s performance continues to improve as the size of the augmented training data increases. In fact, with enough augmented data, the student model can outperform the teacher model. The student model has been successfully deployed in production at Walmart.com with significantly positive metrics. Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 2025, Sydney, NSW, Australia DOI: 10.1145/3701716.3715242 ISBN: 979-8-4007-1331-6/2025/04 CCS: Information systems Learning to rank 1. Introduction In recent years, there has been exceptionally rapid growth in e-commerce platforms. On e-commerce platforms such as Walmart, Amazon and Taobao, customers enter search queries, and the platform must deliver the most relevant items from millions of products in the product catalog. As the product catalog continues to grow, continuously improving search ranking systems has become essential for showing customers highly relevant items on e-commerce platforms. Product search ranking engine typically comprises two main stages: the retrieval stage where a set of relevant products candidates is retrieved from the product catalog to form the recall set, the rerank stage where the candidates obtained from the previous stage are re-ranked to form a ranked product list to be shown to customers. The embedding-based two tower model has become an important approach in the retrieval stage of e-commerce search engines, especially the bi-encoder which uses transformer-encoded representations. Bi-encoder models learn embedding representations for search queries and products from training data. At retrieval time, semantically similar items are retrieved based on simple similarity metrics, such as the cosine similarity between the query and the item. Since the query and document embeddings are learned through separate models, this approach allows the embeddings to be precomputed offline with minimal computational overhead and low latency ( Yates et al., 2021 ; Bai et al., 2022 ) . In contrast, the cross-encoder model takes the query and item text as input and directly outputs a prediction. Although bi-encoder approaches are typically much faster, they are generally less effective than cross-encoder methods ( Vo et al., 2024 ; Wang et al., 2024 ) . By concatenating query and item as input, the cross-encoder model can benefit from the attention mechanism across all tokens in the inputs, thus capturing interactions between query and item text. As a result, the trade-off with bi-encoders involves sacrificing the model’s effectiveness for large latency gain. The retrieval stage has strict latency requirements as it needs to search through millions of products in the product catalog. However, the downstream re-rank stage only ranks the products retrieved during the retrieval stage, allowing it to afford more relaxed latency requirements compared to the retrieval stage. Our previous production model is the first ranking model with a cross-encoder (XE) feature that can better understand the relevance of items to search queries. In this work, we present our our enhanced BERT-Base ( BERT_{\text{Base}} ) model with cross-encoder architecture, optimized through knowledge distillation from LLMs on an augmented large-scale training dataset. In this work, our goal is to improve effectiveness of cross-encoder models used in production for the tail segment. To achieve this goal, we propose a novel knowledge distillation framework that distills the teacher LLM model (i.e. 7 billion parameters) into a student model (i.e. BERT_{\text{Base}} ( Kenton and Toutanova, 2019 ) ) with significantly augmented training data. This framework improves the student model’s effectiveness while preserving its efficiency. First, we train the LLM teacher model using soft-label targets converted from human editorial labels. In addition to item’s title, product type, brand, color and gender, we further incorporate item’s description into the model input to improve our teacher LLM model’s performance. After that, the student model is trained to mimic the margin between two documents predicted by the teacher model. Rather than distilling the knowledge on the labeled training data, we generate large-scale unlabeled data from user’s log data, and label it with the teacher model’s predictions such that the student model can learn from the teacher at a much larger dataset. The effectiveness of our cross-encoder based student model is demonstrated through both offline and online experiments, showing its ability to improve both the relevance and engagement metrics of the system. Our cross-encoder based student model has been deployed in Oct 2024 as the most dominant feature in the current production system, and is powering tail queries in Walmart.com’s e-commerce engine. Our main contributions are summarized as follows: • We propose a knowledge distillation framework in which the teacher model is trained with soft-labeling, enabling more effective distillation through the teacher’s single output. The distillation process uses a extended Margin-MSE loss on an augmented, large-scale unlabeled dataset. Building on the work of Hofstätter et al. (2020) , where both the student and teacher models share the same training data and require true labels for distillation, we significantly expand the dataset used to train the student model. Specifically, we generate a large-scale unlabeled dataset and use the teacher model’s predictions to label the data. The Margin-MSE loss is then computed on all item pairs for a given query, eliminating the need for true labels. • We conduct an ablation study that reveals the following findings: (1) using Margin-MSE loss to align the margin with the teacher model significantly improves knowledge distillation compared to using pointwise cross-entropy (CE) loss. (2) distilling knowledge through a large-scale dataset labeled with the teacher’s predictions leads to a substantial performance boost, enabling the student model to achieve results comparable to the teacher model. (3) the student model performs on par with the teacher model, even without the ‘item description’ field, which has been shown to be beneficial. These results demonstrate that increasing the model’s capacity is not necessary to effectively learn from the teacher model. • Our proposed model has been successfully deployed in production at Walmart.com with significantly positive metrics. The rest of this paper is organized as follows: in Section 2 ,

Read the complete text