FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets
Kairui Fu Tao Zhang Shuwen Xiao Ziyang Wang, Xinming Zhang Chenchi Zhang Yuliang Yan Junjun Zheng, Yu Li Zhihong Chen Jian Wu Xiangheng Kong, Shengyu Zhang Kun Kuang Yuning Jiang Bo Zheng · 2025
Read FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets by Kairui Fu Tao Zhang Shuwen Xiao Ziyang Wang, Xinming Zhang Chenchi Zhang Yuliang Yan Junjun Zheng, Yu Li Zhihong Chen Jian Wu Xiangheng Kong, Shengyu Zhang Kun Kuang Yuning Jiang Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20904.txt.
Introduction
Recommender systems increasingly generate items from learned semantic codes instead of searching a fixed list. Kairui Fu and eleven co-authors benchmark how those codes should be built in FORGE, a 2025 paper accepted for KDD 2026. The setting is industrial recommendation data.
Semantic identifiers assign each item a meaningful code that a generative model can produce token by token. Earlier work rarely compared construction strategies, and judging codes usually meant expensive model training. FORGE offers a taxonomy of the construction process and tests each choice offline across many settings. The findings were applied in production, where the authors report a measurable lift in transactions. The benchmark gives industry teams a cheaper way to design identifiers before committing to full training.
Catalog record
| CREATOR | Kairui Fu Tao Zhang Shuwen Xiao Ziyang Wang, Xinming Zhang Chenchi Zhang Yuliang Yan Junjun Zheng, Yu Li Zhihong Chen Jian Wu Xiangheng Kong, Shengyu Zhang Kun Kuang Yuning Jiang Bo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20904
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets Kairui Fu Tao Zhang Shuwen Xiao Ziyang Wang, Xinming Zhang Chenchi Zhang Yuliang Yan Junjun Zheng, Yu Li Zhihong Chen Jian Wu Xiangheng Kong, Shengyu Zhang Kun Kuang Yuning Jiang Bo Zheng Abstract Semantic identifiers (SIDs) have gained increasing attention in generative retrieval (GR) due to their meaningful semantic discriminability. However, current research on SIDs faces three main challenges: (1) the absence of large-scale public datasets with multimodal features, (2) limited investigation into optimization strategies for SID generation, which typically rely on costly GR training for evaluation, and (3) slow online convergence in industrial deployment. To address these challenges, we propose FORGE , a comprehensive benchmark for FO rming semantic identifie R in G enerative r E trieval with industrial datasets. Specifically, FORGE is equipped with a dataset comprising 14 billion user interactions and multimodal features of 250 million items sampled from Taobao, one of the biggest e-commerce platforms in China. Leveraging this dataset, FORGE explores several optimizations to enhance the SID construction and validates their effectiveness via offline experiments across different settings and tasks. Further online analysis conducted on the "Guess You Like" section of Taobao’s homepage shows a 0.35% increase in transaction count, highlighting the practical impact of our method. Regarding the expensive SID validation accompanied by the full training of GRs, we propose two novel metrics of SID that correlate positively with recommendation performance, enabling convenient evaluations without any GR training. For real-world applications, FORGE introduces an offline pretraining schema that reduces online convergence by half. The code and data are available at https://github.com/selous123/al_sid . 1 Introduction With the ability to predict the next item in an end-to-end manner, generative retrieval (GR) has recently emerged as a promising approach in recommender systems ( rajput2023recommender ; liang2025tbgrecall ) . Typically, this framework encodes user behaviors into a sequence of identifiers , employs larger models to capture item dependencies, and generates identifiers of candidates as results. Therefore, the tokenization, which determines identifiers for each item, plays a fundamental role in advancing accurate recommendation ( qu2025tokenrec ) . Recent advancements in item tokenization for recommendation can be broadly categorized into single-token identifiers and semantic identifiers (SIDs) ( lin2025order ; liu2025generative ; zheng2025universal ) . i) In Figure 1 (a), the former randomly assigns a unique identifier to each item (e.g., 1892392 \rightarrow ) and updates the corresponding latent token independently during training. This schema would lead to the management of massive vocabularies with billions of items in real-world recommendations ( barkan2016item2vec ; petrov2025efficient ) . Numerous identifiers prevent it from including all items, but only several negative ones are sampled for loss calculation to reduce computational burdens ( zhai2024actions ; ma2024negative ) . Such an approximation, however, might result in inconsistency between Figure 1 : Diagram of (a) single token identifier and (b) semantic identifier. (c) The generation of SIDs. offline training and online serving, where the latter requires measuring all items. ii) The semantic identifiers shown in Figure 1 (b) represent items with multi-level codewords (e.g., < C_{1}1248 >< C_{2}6475 >< C_{3}6968 > \rightarrow ), and retrieval models would generate the top-K item identifiers as candidates with continuous beam search ( li2025bbqrec ; zheng2025universal ) . The construction of codewords within SID is based on multimodal data, enabling knowledge sharing among similar items. Furthermore, with hierarchical codewords, SIDs could compactly encode an exponentially large item space (e.g., a 3-level SID with 8192 tokens per level could represent 8192^{3}\gg 10^{10} items using 24576 tokens). The small number of tokens also makes it computationally feasible to include all of them in the calculations during training ( rajput2023recommender ) . As depicted in Figure 1 (c), the tokenization process for SIDs typically follows an encoder-quantization framework: an encoder for extracting semantic features from raw multimodal data and another quantization module ( esser2021taming ; lee2022autoregressive ) for SID encoding, separately ( zheng2024adapting ) . These compact yet semantically rich identifiers are subsequently used by downstream retrieval models during task-specific training. Despite their promising results on real-world datasets ( wang2024enhanced ; liu2025generative ; ju2025generative ) , they still have the following challenges that hinder the development of this field: 1. Limited behavior and features within datasets . The majority of publicly available datasets in recommendation are limited in size ( \leq 10 million) ( harper2015movielens ; ni2019justifying ) , resulting in findings that may not generalize well when applied to industrial scenarios ( \geq 1 billion) ( li2019multi ) . In addition, their construction is primarily based on single-token identifiers in the absence of abundant multimodal features, thus not accounting for SIDs ( zhu2021open ; zhu2022bars ; yuan2022tenrec ) . The dataset limitations impede the ability of later researchers to generate practical insights. 2. Neglect of the optimization for SID generation . Even if SID serves as the foundation for the retrieval task ( yang2024unifying ; ju2025generative ) , useful insights around the design choices for it are typically not well-discussed in previous literature, making it complicated for researchers to select an optimal configuration. Furthermore, recent works assess the quality of SIDs through time-consuming training of GR ( ju2025generative ) , remaining a critical need for direct metrics with convenient SID evaluations. 3. Inefficiency in industrial deployment . Once a better SID configuration is confirmed via offline experiments, it typically requires deployment in the production environment and further training with the retrieval model for several days to reach convergence. The lengthy process hinders engineers from leveraging online observations to validate the SID optimizations adequately. Unfortunately, recent studies ( deng2025onerec ; zheng2025ega ) consistently overlook the practical aspects for efficient deployment. This absence limits their broader impact and applicability in real industrial settings. To mitigate this gap, we propose FORGE , a comprehensive benchmark for FO rming semantic identifie R in G enerative r E trieval with industrial datasets. ( Challenge 1 ) In contrast to current public datasets that suffer from small scale and insufficient multimodal richness, FORGE is collected from one of the biggest e-commerce platforms in China, providing more than 14 billion user behaviors in 10 days, along with multimodal features of 250 million items. We partition the dataset into three consecutive phases for continuous training and robustly separated evaluation, facilitating convenient usage for researchers. ( Challenge 2 ) With the proposed dataset, FORGE introduces several optimization strategies for SID generation, encompassing diverse data modalities and ID collision mitigation (i.e., cases where multiple similar items correspond to the same SID). We thoroughly validate the effectiveness of these strategies across a range of configurations, including different downstream GR models, SID structures, and search tasks. Following improvements observed in the 7-day online evaluation further support the validity of the proposed optimizations. In addition, FORGE introduces two novel metrics (i.e., embedding hitrate and Gini coefficient) for directly assessing SID quality. Experiments show that these metrics exhibit strong correlation with the performance of GRs, enabling SID measurements without the need for costly GR training and thus providing an effective and efficient evaluation process. ( Challenge 3 ) Serving as an industrial benchmark, FORGE also explores several warm-up techniques aimed at accelerating the convergence of new SIDs deployed in online scenarios. Continuous experiments spanning 10 days indicate