Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808481 CCS: Information systems Learning to rank CCS: Computing methodologies Neural networks CCS: Computing methodologies Multi-task learning CCS: Information systems Relevance assessment CCS: Applied computing Online shopping
Luming Chen, Jiaqi Xi, Raghav Saboo, Kenny Chi, Martin Wang, Sudeep Das, Danny Nightingale, Aditya Dodda, Elyse Winer, Akshad Viswanathan · 2026
Read Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808481 CCS: Information systems Learning to rank CCS: Computing methodologies Neural networks CCS: Computing methodologies Multi-task learning CCS: Information systems Relevance assessment CCS: Applied computing Online shopping by Luming Chen, Jiaqi Xi, Raghav Saboo, Kenny Chi, Martin Wang, Sudeep Das, Danny Nightingale, Aditya Dodda, Elyse Winer, Akshad Viswanathan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27704.txt.
Introduction
Luming Chen and colleagues build a live multi-task ranking system that balances relevance with engagement for e-commerce search. An ordinal relevance head predicts running probabilities over thresholds, merged with engagement heads in one value model for tunable trade-offs. Lightweight language models supply three-level relevance labels for over 100 million query-item pairs, improving meaning match without losing engagement.
Catalog record
| CREATOR | Luming Chen, Jiaqi Xi, Raghav Saboo, Kenny Chi, Martin Wang, Sudeep Das, Danny Nightingale, Aditya Dodda, Elyse Winer, Akshad Viswanathan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20803 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808481 CCS: Information systems Learning to rank CCS: Computing methodologies Neural networks CCS: Computing methodologies Multi-task learning CCS: Information systems Relevance assessment CCS: Applied computing Online shopping Luming Chen, Jiaqi Xi, Raghav Saboo, Kenny Chi, Martin Wang, Sudeep Das, Danny Nightingale, Aditya Dodda, Elyse Winer, Akshad Viswanathan Abstract . Optimizing industrial search ranking models solely for user engagement signals often introduces systematic biases, prioritizing popular or price-anchored items that may not satisfy semantic intent. We present a production-scale multi-task ranking system that integrates semantic relevance as a primary optimization objective, enabling explicit and controllable relevance–engagement trade-offs. Our architecture employs an ordinal relevance head that predicts cumulative probabilities over relevance thresholds, preserving the inherent ordering of labels. These outputs are integrated with engagement heads through a unified value model scoring function, enabling systematic balancing of semantic quality and short-term behavioral signals. To provide high-quality supervision for this multi-task framework, we utilize fine-tuned lightweight Large Language Models (LLMs) to generate three-level ordinal relevance labels: irrelevant, moderately relevant, and highly relevant. We address challenges regarding label distribution sensitivity and ensure high alignment with human annotations to enable efficient labeling for over 100 million query–item pairs. Evaluation across offline metrics, including NDCG@10, and online A/B experiments demonstrates that our approach significantly improves semantic alignment while preserving core engagement objectives. † † cc-license: by 1. Introduction Search is a primary driver of discovery and revenue in large-scale e-commerce platforms. In production systems, ranking models are typically optimized against user engagement signals such as click-through rate (CTR), add-to-cart rate (ATCR), and conversion rate (CVR). These signals are abundant and directly aligned with business objectives ( 2 ; 7 ) . However, optimizing solely for engagement can introduce systematic biases: items that attract interaction due to popularity, price anchoring, or lexical overlap may be up-ranked even when they do not semantically satisfy user intent ( 3 ; 4 ) . Over time, such behavior can degrade perceived search quality, increase query reformulation, and erode user trust. A central challenge in addressing this issue is the scarcity of high-quality semantic relevance supervision at production scale. Human annotation is expensive, slow, and difficult to maintain for dynamic catalogs containing millions of items and long-tail queries. Behavioral signals are abundant but are affected by position bias, exposure bias, and feedback loops ( 3 ; 4 ) . As a result, many industrial ranking systems lack reliable fine-grained semantic supervision beyond implicit engagement feedback. Recent advances in large language models (LLMs) offer a promising alternative. Multiple studies show that prompted or fine-tuned LLMs can generate high-quality relevance judgments that closely align with human annotations in product search and web-scale settings ( 5 ; 6 ; 14 ; 17 ; 13 ) . However, integrating LLM supervision into production search systems presents practical challenges. First, LLM inference is too expensive and latency-sensitive for real-time serving at high queries-per-second rates. Second, fine-tuned LLM classifiers can be sensitive to label distribution and may systematically over- or under-predict certain relevance levels. Third, raw LLM scores must be calibrated and structured appropriately to interact with existing engagement-driven ranking objectives ( 11 ) . In this work, we present a deployed multi-task ranking system that incorporates efficient LLM-based relevance supervision as a first-class optimization objective. We fine-tune GPT models on hundreds of thousands of human-annotated query–item pairs and generate large-scale three-level ordinal relevance labels offline. We observe that fine-tuning performance is highly sensitive to the underlying label distribution, and we adopt sampling strategies that mimic production relevance proportions to ensure stable classification behavior. The resulting model achieves strong agreement with human annotations, including on long-tail and multi-word queries. We integrate the LLM-generated relevance signal into a shared multi-task neural ranking architecture that jointly predicts CTR, ATCR, CVR, and relevance. A unified value function enables explicit control over relevance and engagement trade-offs at serving time. This design allows us to improve semantic alignment while maintaining core business performance, without invoking LLMs during online inference. Our contributions are threefold: • We design a scalable pipeline for LLM-based relevance supervision that generates high-quality ordinal labels aligned with production query distributions. • We integrate ordinal semantic relevance directly into a multi-task ranking objective, enabling controllable trade-offs between relevance and engagement. • We demonstrate offline relevance improvements and positive online impact in a large-scale production search system. 2. Related Work Multi-task learning (MTL) is widely adopted in industrial product search ranking to jointly optimize engagement-driven objectives such as CTR, ATCR, and CVR using shared representations with task-specific heads. Recent systems integrate heterogeneous signals, including tabular user features and semantic text embeddings, to model complex query–item interactions, consistently outperforming single-task and tree-based approaches ( 15 ; 2 ; 7 ) . However, relevance is often absent from the primary optimization objectives or treated as an auxiliary task, leaving such models susceptible to bias and exposure effects. LLMs have recently demonstrated strong capability in generating high-quality relevance supervision at scale. Prior work demonstrates that fine-tuned LLMs can produce relevance judgments with strong agreement with human annotations, enabling large-scale offline evaluation and label generation for training and distillation ( 6 ; 17 ; 14 ; 5 ; 13 ; 1 ) . LLM-predicted semantic relevance scores have been incorporated into learning-to-rank objectives to complement engagement-based signals through calibrated label formulations that balance content relevance and behavioral outcomes ( 5 ) . In addition, several studies highlight the importance of fine-grained relevance supervision, showing that graded relevance labels improve ranking reliability compared to binary judgments ( 17 ; 14 ; 13 ) . Separately, LLMs have also been explored as reranking agents, showing strong effectiveness but high inference cost ( 12 ; 18 ) . In contrast to prior work that treats LLMs primarily as external relevance teachers, evaluation tools, or standalone rerankers, our approach integrates efficient, fine-grained LLM supervision directly into a multi-task ranking objective, enabling explicit control of relevance–engagement trade-offs in production-scale search systems without incurring online LLM inference overhead. 3. Methodology Given a user query and a set of candidate items, our goal is to rank items by jointly optimizing relevance and user engagement outcomes. Engagement is modeled through multiple behavioral signals, including CTR, ATCR, and CVR, while relevance reflects the semantic match between query intent and item content. 3.1. Multi-task Model Architecture We adopt a multi-task deep neural network (DNN) architecture that jointly predicts engagement and relevance signals, as illustrated in Figure 1 . The model incorporates multiple feature groups, including query features (e.g., query embeddings, inferred categories, and brand), item features (e.g., item embeddings, price, category, brand attributes, and historical engagement statistics), and consumer features representing past user interactions. In addition, we incorporate interaction features between queries and items, such as BM25 scores for multiple text fields, historical query–item engagement rates, and string similarity measures between the query and item text attributes. We also include user–item and user–taxonomy