ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Dev
Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo · 2025
Read ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Dev by Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02555.txt.
Introduction
Zheng Fang and seven co-authors work on search relevance for online stores, where wording often misleads. Classic term matching, such as BM25, misses meaning when shoppers and sellers use different words. Neural models read meaning better, but they need hard training examples that are scarce within each product domain.
Their system, ADORE, is an autonomous relevance engine built for e-commerce search. It aims to find or build the domain specific hard cases that neural models lack. Sharper relevance means shoppers see products that match intent, and sellers reach buyers who truly want their goods. Each hard case the engine finds makes the next round of search a bit less blind than the last.
Catalog record
| CREATOR | Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02555
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3731944 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Language models CCS: Information systems Sponsored search advertising CCS: Computing methodologies Natural language processing CCS: Information systems Relevance Modeling CCS: Information systems Novelty in information retrieval CCS: Information systems Online shopping Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo Abstract . Relevance modeling in e-commerce search remains challenged by semantic gaps in term-matching methods (e.g., BM25) and neural models’ reliance on the scarcity of domain-specific hard samples. We propose ADORE, a self-sustaining framework that synergizes three innovations: (1) A Rule-aware Relevance Discrimination module, where a Chain-of-Thought LLM generates intent-aligned training data, refined via Kahneman-Tversky Optimization (KTO) to align with user behavior; (2) An Error-type-aware Data Synthesis module that auto-generates adversarial examples to harden robustness; and (3) A Key-attribute-enhanced Knowledge Distillation module that injects domain-specific attribute hierarchies into a deployable student model. ADORE automates annotation, adversarial generation, and distillation, overcoming data scarcity while enhancing reasoning. Large-scale experiments and online A/B testing verify the effectiveness of ADORE. The framework establishes a new paradigm for resource-efficient, cognitively aligned relevance modeling in industrial applications. 1. Introduction E-commerce applications, such as Amazon, Taobao, and JD, have changed people’s shopping habits significantly. The relevance module is crucial for the user experience, as displaying irrelevant products can harm trust and revenue for e-commerce systems. Therefore, relevance modeling, which evaluates the match between search queries and products, is a basic part of e-commerce engines. Traditional relevance models, such as term matching and vector matching methods, have been widely used in online e-commerce systems to ensure fundamental search experience. Term matching based methods apply TF-IDF, BM25 ( 14 ) and statistical features to compute the matching degree between query and product, but fail to capture semantic relevance and contextual understanding. Vector matching based approaches ( 11 ; 19 ; 25 ; 10 ) leverage BERT ( 3 ) and its variants ( 23 ; 24 ; 15 ; 1 ; 7 ) to understand the semantic relationships between queries and products and achieve remarkable improvement in relevance modeling. Despite the progress made, vector matching based methods still face two challenges when tackling domain-specific relevance modeling tasks: (1) The first challenge is the lack of domain-specific hard samples. Most existing methods can achieve satisfactory results by leveraging a moderate amount of manually annotated data. However, further performance improvements often require a substantial number of hard samples, which are not only difficult to obtain but also challenging to annotate accurately. (2) The second challenge is the models’ insufficient reasoning and representation capabilities. Constrained by their parameter size, online lightweight models often struggle with complex reasoning and relevance discrimination. To enhance reasoning and comprehension capabilities, large language models (LLMs) have been increasingly applied to relevance modeling tasks ( 2 ; 8 ; 18 ; 20 ; 22 ; 17 ; 12 ) . However, most prior LLM approaches primarily utilize the representation of the last token for relevance classification and employ soft labels from the classification layer to distill knowledge into smaller online models. This discriminative paradigm is unable to intuitively analyze which features impact relevance decisions, nor can it enhance the reasoning capabilities of the online student models. To address the above issues, we propose the Autonomous Domain-Oriented Relevance Engine (ADORE), which consists of three parts: a domain rule-aware relevance discrimination (RD) module, an error-type-aware data synthesis (DS) module, and a key-attribute-enhanced knowledge distillation (KD) module. To automate the construction of domain-specific hard samples, we propose a Chain-of-Thought (CoT) ( 21 ) based reasoning LLM model within the RD module. To ensure that LLM predictions align with domain rules and user behavior, we use Kahneman-Tversky Optimization (KTO) ( 5 ) to align LLM. Furthermore, to endow ADORE with autonomous error correction capabilities, we design an error-type-aware generation model in the DS component. Finally, to effectively transfer the reasoning capabilities of the LLM to the online model, we incorporate the key attributes derived from the CoT analysis into the model input in the KD module, thereby achieving explicit knowledge enhancement. In summary, the main contributions of this paper are as follows. • We propose the A utonomous D omain- O riented R elevance E ngine (ADORE), which integrates a domain rule-aware relevance discrimination (RD) module, an error-type-aware data synthesis (DS) module, and a key-attribute-enhanced knowledge distillation (KD) module. • We introduce a CoT-based reasoning LLM in the RD module, which automatically constructs domain-specific hard samples by relevance reasoning on online exposure products. Additionally, the DS module incorporates an error-type-aware generation model to autonomously produce various challenging samples, further enhancing the system’s robustness. Key attribute features from CoT analysis are explicitly integrated into the KD module, enabling explicit knowledge enhancement and improving the reasoning and representation capabilities of the online student model. • We conduct extensive experiments to evaluate the performance of ADORE, including large-scale offline dataset and online A/B testing. The results demonstrate that ADORE significantly outperforms existing state-of-the-art methods, delivering substantial improvements in both offline and online metrics. This highlights the practical applicability and effectiveness of ADORE in handling complex relevance modeling tasks in e-commerce. 2. METHODOLOGY In our work, we formulate search relevance prediction as a binary classification problem. Given a query Q and a product P , relevance modeling in search engines aims to predict the relevance degree between them. Specifically, the model is designed to classify query-product pairs into one of two distinct relevance categories: Relevant and Irrelevant. Figure 1 illustrates the components of the Autonomous Domain-Oriented Relevance Engine (ADORE). Figure 1. The architecture of the Autonomous Domain-Oriented Relevance Engine. 2.1. Rule-aware Relevance Discrimination 2.1.1. CoT Tuning To identify hard samples that can enhance the performance of our online model, we have developed a relevance discrimination model based on Large Language Models. Previous research ( 21 ; 9 ) has demonstrated that Chain-of-Thought (CoT) prompting significantly enhances the reasoning accuracy of LLMs. Building on this insight, we fine-tuned our LLM using a CoT-based approach. We initiate the process by utilizing GPT-4o to create CoT labeled data. This is achieved by simulating the human annotation process, where GPT-4o is provided with domain-specific annotation guidelines and a few exemplary CoT instances. As a result, we generate a dataset enriched with detailed reasoning chains, mirroring the depth and nuance of human-like annotation. To further enhance the accuracy of the labeled data, we empirically construct only query-product data that can be validated using human-annotated labels and exclude any CoT data inconsistent with the human labels, forming our final training set. Subsequently, we employ these samples to further fine-tune the LLaMA-3 ( 4 ) model through optimization of a standard Language Model loss: (1) \mathcal{L}_{\text{LM}}=-\frac{1}{N}\sum_{i=1}^{N}\log P(w_{i}|w_{<i}) The CoT output of the fine-tuned model is illustrated in the figure 1 . In this output, the initial step involves extracting key attributes from the query and item descriptions, including product names, brands, model specifications, target audiences, and more. Subsequently, the