RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference
Jiarui Wang * , Huichao Chai * , Yuanhang Zhang, Zongjin Zhou, Wei Guo, Xingkun Yang, Qiang Tang, Bo Pan, Jiawei Zhu, Ke Cheng † , Yuting Yan † , Shulan Wang † , Yingjie Zhu † , Zhengfan Yuan, Jiaqi Huang, Yuhan Zhang, Xiaosong Sun, Zhinan Zhang, Hong Zhu, Yongsheng Zhang, Tiantian Dong, Zhong Xiao, · 2026
Read RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference by Jiarui Wang * , Huichao Chai * , Yuanhang Zhang, Zongjin Zhou, Wei Guo, Xingkun Yang, Qiang Tang, Bo Pan, Jiawei Zhu, Ke Cheng † , Yuting Yan † , Shulan Wang † , Yingjie Zhu † , Zhengfan Yuan, Jiaqi Huang, Yuhan Zhang, Xiaosong Sun, Zhinan Zhang, Hong Zhu, Yongsheng Zhang, Tiantian Dong, Zhong Xiao, (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01712.txt.
Introduction
Online recommender systems must rank items within tens of milliseconds, which caps how much of a user's history a generative ranking model can read. Jiarui Wang and forty co-authors present RelayGR, a production system. It works around the cap in a simple way. Most of the user's behavior sequence is computed ahead of time. The result is then reused at ranking.
RelayGR pre-computes long-term behavior prefixes for selected requests. It keeps their cached data in fast memory across pipeline stages. It also routes each request so the cache and the ranking meet on the same machine. A memory-aware expander also reuses short-term patterns through local storage. In tests with real queries on Huawei Ascend hardware, the system supported sequences up to one and a half times longer and raised throughput several-fold under the same latency limit.
Catalog record
| CREATOR | Jiarui Wang * , Huichao Chai * , Yuanhang Zhang, Zongjin Zhou, Wei Guo, Xingkun Yang, Qiang Tang, Bo Pan, Jiawei Zhu, Ke Cheng † , Yuting Yan † , Shulan Wang † , Yingjie Zhu † , Zhengfan Yuan, Jiaqi Huang, Yuhan Zhang, Xiaosong Sun, Zhinan Zhang, Hong Zhu, Yongsheng Zhang, Tiantian Dong, Zhong Xiao, |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01712
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference Thanks: * Equal contribution. Thanks: † Work done during their internships at Huawei. Thanks: 🖂 Corresponding authors: {jinyibo1, fengyongxiang1, pengfei.zuo}@huawei.com. Jiarui Wang * , Huichao Chai * , Yuanhang Zhang, Zongjin Zhou, Wei Guo, Xingkun Yang, Qiang Tang, Bo Pan, Jiawei Zhu, Ke Cheng † , Yuting Yan † , Shulan Wang † , Yingjie Zhu † , Zhengfan Yuan, Jiaqi Huang, Yuhan Zhang, Xiaosong Sun, Zhinan Zhang, Hong Zhu, Yongsheng Zhang, Tiantian Dong, Zhong Xiao, Deliang Liu, Chengzhou Lu, Yuan Sun, Zhiyuan Chen, Xinming Han, Zaizhu Liu, Yaoyuan Wang, Ziyang Zhang, Yong Liu, Jinxin Xu, Yajing Sun, Zhoujun Yu, Wenting Zhou, Qidong Zhang, Zhengyong Zhang, Zhonghai Gu, Yibo Jin 🖂 , Yongxiang Feng 🖂 , Pengfei Zuo 🖂 Huawei Technologies Co., Ltd. Abstract . Abstract Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for ranking. Generative recommendation (GR) models can improve quality by consuming long user-behavior sequences, but in production their online sequence length is tightly capped by the ranking-stage P99 budget. We observe that the majority of GR tokens encode user behaviors that are independent of the item candidates, suggesting an opportunity to pre-infer a user-behavior prefix once and reuse it during ranking rather than recomputing it on the critical path. Realizing this idea at industrial scale is non-trivial: the prefix cache must survive across multiple pipeline stages before the final ranking instance is determined, the user population implies cache footprints far beyond a single device, and indiscriminate pre-inference would overload shared CPU/PCIe/NPU resources under high QPS. We present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR selectively pre-infers long-term user prefixes, keeps their per-layer KV caches resident in HBM over the request lifecycle, and ensures the subsequent ranking stage can consume them without remote fetches. RelayGR combines three techniques: (1) a sequence-aware trigger that admits only at-risk requests under a bounded cache footprint and pre-inference load, (2) an affinity-aware router that co-locates cache production and consumption by routing both the auxiliary pre-infer signal and the ranking request to the same instance, and (3) a memory-aware expander that uses server-local DRAM to capture short-term cross-request reuse while avoiding redundant reloads. We implement RelayGR on Huawei Ascend NPUs and evaluate it with real queries in a production-mirror environment. Under a fixed P99 SLO, RelayGR supports up to 1.5 × \times longer sequences and improves SLO-compliant throughput by up to 3.6 × \times . 1. Introduction Modern industrial recommender systems ( 1 ; 22 ; 14 ) serve tens of billions of requests per day through a multi-stage cascade—retrieval, pre-processing (also known as coarse ranking), and fine-grained ranking. Because user engagement is highly sensitive to latency, the entire pipeline must complete within a few hundred milliseconds; otherwise, timeouts translate directly into revenue loss. The fine-grained ranking stage is the bottleneck: it typically has only tens of milliseconds at the 99th percentile (P99) to score hundreds of candidate items with a high-capacity model. Generative recommendation (GR) models ( 29 ; 3 ; 45 ; 2 ; 10 ; 41 ; 40 ; 16 ; 6 ; 48 ; 21 ; 46 ) are increasingly adopted as the next generation of ranking models. Compared with traditional deep learning recommenders (DLRMs) ( 34 ; 50 ; 49 ; 15 ; 5 ; 35 ; 51 ) , GR models better capture long sequential behaviors and often exhibit favorable scaling with longer sequences and larger capacity. Offline training and evaluation routinely use thousands of behavior tokens per user; in online serving, however, increasing sequence length (or feature dimension) quickly inflates inference latency and violates the ranking-stage P99 SLO. Prior efforts either pursue end-to-end GR ( 6 ; 48 ; 21 ) that unifies retrieval and ranking but may need to weaken or remove the candidate-specific cross features ( 10 ) , or deploy GR within retrieval/ranking ( 28 ; 10 ; 46 ) and treat cross features as compensation. In all cases, production deployments remain bounded by the same first-order barrier: ranking-stage P99 caps the usable sequence length and thus limits GR’s online scaling benefits (Figure 1 ). Our key observation is that GR inputs have a key structure property, i..e, most tokens encode user behaviors ( 11 ; 45 ; 43 ) rather than candidate items. Long-term behaviors (e.g., weeks/months of clicks and views) naturally form a prefix and evolve more slowly than short-term signals and candidate-dependent cross features injected later. This suggests a simple opportunity: pre-infer the long-term user prefix, cache its intermediate states (per-layer KV), and reuse them when scoring candidate items in fine-grained ranking. This resembles prefix caching in LLM serving ( 8 ; 27 ) , but the recommender setting is fundamentally harder: prefixes are user-specific (not shared across queries), and the cache must survive across multiple pipeline stages before ranking executes. Turning the above idea into a production optimization requires solving three systems challenges. (1) Bridging caches across the pipeline under late binding. A prefix cache may be produced during retrieval, but can only be consumed during ranking after pre-processing completes. At cache creation time, the eventual ranking instance may not yet be determined; if production and consumption land on different instances, the cache must be fetched remotely—often too expensive for a tens-of-milliseconds ranking. Even within a server, overflowing device memory forces a flush/reload that competes with the ranking SLO. (2) The user population makes “cache everything” infeasible. Industrial recommenders serve billions of users ( 7 ; 33 ; 10 ) . Even modest GR backbones ( 45 ; 10 ) can generate megabytes of KV states per user when consuming thousands of tokens, yielding petabyte-scale aggregates (megabytes multiplying all users). A naïve distributed cache pool would frequently trigger remote fetches whose latency can rival or exceed the ranking budget, undermining the entire approach. (3) High QPS makes unconditional pre-inference unsafe. At production scale, each accelerator instance may serve hundreds of queries per second ( 4 ; 31 ) . After feature processing and embedding lookup, each query can carry tens of megabytes of embeddings ( 19 ; 42 ) . Under concurrency, host-to-device transfers and accelerator execution contend for shared CPU/PCIe/NPU resources ( 52 ) . If we pre-infer prefixes for every request, we risk overloading the very resources needed to keep ranking within P99. Therefore, pre-inference must be selective and load-aware. (a) Restricted sequence length (b) Restricted throughput Figure 1 . Ranking-stage P99 restricts performance. To address these challenges, we present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR pre-infers long-term user prefixes only when beneficial, keeps the resulting per-layer KV cache resident in device memory over the request lifecycle, and ensures the subsequent ranking stage consumes the cache locally without remote fetches. More broadly, RelayGR addresses a distinct systems problem: lifecycle caching under late-binding placement , where cache production and consumption are separated by multiple pipeline stages and the consumer is determined only after intermediate filtering. RelayGR is built around three techniques that co-design when to pre-infer, where to place the cache, and how to extend reuse safely, for the real-time recommender: Sequence-aware Trigger. During retrieval, RelayGR