SORT: A Systematically Optimized Ranking Transformer for Industrial-scale Recommenders
Chunqi Wang ∗ Bingchao Wu Taotian Pang Jiahao Wang Jie Yang Jia Liu Hao Zhang Hai Zhu Lei Shen Shizhun Wang Bing Wang ∗ Xiaoyi Zeng · 2026
Read SORT: A Systematically Optimized Ranking Transformer for Industrial-scale Recommenders by Chunqi Wang ∗ Bingchao Wu Taotian Pang Jiahao Wang Jie Yang Jia Liu Hao Zhang Hai Zhu Lei Shen Shizhun Wang Bing Wang ∗ Xiaoyi Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03988.txt.
Introduction
Chunqi Wang and others propose SORT, a systematically optimized ranking Transformer for industrial-scale recommenders. SORT tackles those challenges with request-centric sampling, local focus, query pruning, making pre-training, and refinements to tokenization, focus, and feed-forward modules. Training system tuning elevates model FLOPs utilization to 45%. Online A/B testing confirms significant gains, including orders up 7.47%, while cutting latency and boosting throughput nearly sevenfold. SORT now runs in production, serving every AliExpress user.
Catalog record
| CREATOR | Chunqi Wang ∗ Bingchao Wu Taotian Pang Jiahao Wang Jie Yang Jia Liu Hao Zhang Hai Zhu Lei Shen Shizhun Wang Bing Wang ∗ Xiaoyi Zeng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44255 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SORT: A Systematically Optimized Ranking Transformer for Industrial-scale Recommenders DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 Thanks: Correspondence: {shiyuan.wcq,lingfeng.wb}@alibaba-inc.com CCS: Information systems Recommender systems Chunqi Wang ∗ Bingchao Wu Taotian Pang Jiahao Wang Jie Yang Jia Liu Hao Zhang Hai Zhu Lei Shen Shizhun Wang Bing Wang ∗ Xiaoyi Zeng Abstract . While Transformers have achieved remarkable success in LLMs through superior scalability, their application in industrial-scale ranking models remains nascent, hindered by the challenges of high feature sparsity and low label density. In this paper, we propose SORT ( S ystematically O ptimized R anking T ransformer), a scalable model designed to bridge the gap between Transformers and industrial-scale ranking models. We address the high feature sparsity and low label density challenges through a series of optimizations, including request-centric sample organization, local attention, query pruning and generative pre-training. Furthermore, we introduce a suite of refinements to the tokenization, multi-head attention (MHA), and feed-forward network (FFN) modules, which collectively stabilize the training process and enlarge the model capacity. To maximize hardware efficiency, we optimize our training system to elevate the model FLOPs utilization (MFU) to 22%. Extensive experiments demonstrate that SORT outperforms strong baselines and exhibits excellent scalability across data size, model size and sequence length, while remaining flexible at integrating diverse features. Finally, online A/B testing in large-scale e-commerce scenarios confirms that SORT achieves significant gains in key business metrics—including orders (+6.35%), buyers (+5.97%) and GMV (+5.47%)—while simultaneously halving latency (-44.67%) and doubling throughput (+121.33%). 1. Introduction The Transformer architecture ( 19 ) has established dominance in large language models (LLMs) ( 18 ; 10 ; 24 ) , owing to its excellent scalability. However, its application to ranking models in industrial-scale recommenders remains in a nascent stage. For a long time, the landscape of ranking models has been dominated by specialized networks that focus on explicit feature interaction. For instance, DIN ( 30 ) is designed to extract patterns from sequential features, while DeepFM ( 6 ) and DCN ( 21 ) specialize in heterogeneous feature interaction. These networks are primarily designed to capture feature interaction by low-compute operations including vector-wise operations and low-dimensional matrix multiplication, which struggle to leverage the massive parallel computational capacity of modern accelerators and are also unable to capture high-dynamic data patterns. Given the critical role of ranking models in industrial recommenders and the huge success of LLMs, we propose rebuilding ranking models based on the Transformer architecture, paving the way for highly unified and scalable ranking models. However, directly applying the Transformer architecture to ranking models is sub-optimal due to the inherent differences between recommendation and language modeling. A primary challenge lies in the synergy between high feature sparsity and low label density , which hinders the model’s ability to learn robust representations from discrete, massive feature spaces with infrequent supervisory signals. In language modeling, feature sparsity is relatively low due to the adoption of subword tokenization techniques such as BPE ( 14 ) and wordpiece ( 16 ) , which constrain the vocabulary size to a manageable range (e.g., within hundreds of thousands). In the meanwhile, its label density is inherently high, as every position within the text sequence possesses a supervisory signal (i.e., next-token prediction) drawn from the entire vocabulary-sized search space. Conversely, ranking models must handle item vocabularies at the billion scale, necessitating enormous embedding tables and resulting in extremely high feature sparsity. The challenge is further defined by low label density, stemming from two key factors. First, the historical sequence makes up the majority of the input but lacks direct supervision, which is restricted solely to the target items. Second, these signals are binary and sparse by nature, where the vast majority of samples are negative (zero-valued), such as instances of non-clicks. This information asymmetry, where a massive parameter space is regularized by sparse binary labels, not only induces overfitting, but also leads to substantial computational waste on long historical sequences. To address these issues, we introduce request-centric sample organization , local attention , and query pruning to mitigate the computational inefficiency caused by the long historical sequences. By reallocating the Transformer’s computation to label-relevant interactions, these techniques significantly reduce computational overhead. In addition, we incorporate generative pre-training to resolve the issues introduced by the information asymmetry of high feature sparsity and low label density. Since the sparse binary signals from target items are insufficient to regularize a billion-scale parameter space, this approach augments label density through generative training (i.e., next-item prediction), thereby effectively alleviating overfitting and stabilizing the training of Transformer-based ranking models. Complementing these optimizations, we further refine the Transformer architecture by enhancing the tokenization, multi-head attention layer (MHA) and feed-forward network (FFN) modules. For the tokenization module, we introduce special tokens as feature boundaries. Within the MHA layer, we introduce QKNorm and attention gate to stabilize the training process. For the FFN module, we replace the standard architecture with a mixture-of-experts (MoE) layer, thereby increasing model capacity without incurring additional computational costs. We refer to the proposed approach as SORT, standing for S ystematically O ptimized R anking T ransformer. Extensive experiments show that SORT achieves significant performance gains compared with strong baselines. We also evaluate SORT’s scalability across multiple aspects: data size, model size and sequence length, demonstrating that SORT exhibits excellent scalability. Feature engineering is another critical aspect of ranking models. We demonstrate that SORT is flexible to integrate diverse handcrafted features, including multi-modal features, statistical features and other model-derived features. Finally, extensive online A/B testing across multiple scenarios on AliExpress 1 1 1 http://www.aliexpress.com demonstrates that SORT not only yields substantial business growth (+6.35% in orders, +5.97% in buyers, and +5.47% in GMV) but also drastically enhances serving efficiency, halving latency (-44.67%) and more than doubling throughput (+121.33%). 2. Related Work Conventional deep learning recommendation models (DLRMs) for recommendation typically rely on specialized architectures to handle sequence features (e.g., DIN ( 30 ) ) and extensive handcrafted heterogeneous features (e.g., DeepFM ( 6 ) , DCN ( 21 ) ). Unlike Transformers, these architectures lack scalability, resulting in limited model capacity. Consequently, patterns are primarily captured through the rote memorization of sparse embeddings (e.g., item embeddings). Previous studies have demonstrated that such massive embedding tables often impair generalization and lead to overfitting ( 28 ; 20 ) . Motivated by LLMs, HSTU ( 25 ) introduces scalable architectures into RMs. Specifically, HSTU treats ranking as a next-token prediction task and relies solely on a unified sequence as input. This modification amortizes the training cost since it allows processing multiple candidates within a single sample. Subsequent models, such as MTGR ( 7 ) , GenRank ( 9 ) and this work, have also adopted a similar paradigm. Wukong ( 27 ) also introduces scalable architectures into ranking tasks. However, in contrast to HSTU, Wukong focuses on handling heterogeneous feature interaction. Rankmixer ( 31 ) also falls into this category. Recently, OneTrans ( 29 ) introduced a Transformer variant that leverages independent parameters to model heterogeneous features and user sequence. The independent parameters can be regarded as rule-based routing MoE. Different from that, SORT avoids rule-based routing MoE instead using a data-driven routing one. To better position our work, one can envision a spectrum where traditional DLRMs—reliant