How Reliable Are Semantic-ID Tokenizer Comparisons in Generative Recommendation? CCS: Information systems Recommender systems
Qian Zhang, Lech Szymanski, Haibo Zhang, Jeremiah D. Deng · 2026
Read How Reliable Are Semantic-ID Tokenizer Comparisons in Generative Recommendation? CCS: Information systems Recommender systems by Qian Zhang, Lech Szymanski, Haibo Zhang, Jeremiah D. Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25330.txt.
Introduction
Qian Zhang and colleagues show standard scores for generative recommenders can mislead when item codes collide. Semantic-ID systems give each item a token sequence, and evaluation matches generated sequences to target sequences. That scoring stays honest only when each sequence names one item, yet collisions reach about 31 percent across tokenizers and datasets. The team introduces collision-corrected metrics and a reassignment method that rebuilds collision-free codes cheaply. Under honest item-level scoring, inflation grows with collision rate and can flip the winning tokenizer, casting doubt on past rankings.
Catalog record
| CREATOR | Qian Zhang, Lech Szymanski, Haibo Zhang, Jeremiah D. Deng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53976 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Reliable Are Semantic-ID Tokenizer Comparisons in Generative Recommendation? CCS: Information systems Recommender systems Qian Zhang, Lech Szymanski, Haibo Zhang, Jeremiah D. Deng Abstract . In Semantic-ID (SID) based generative recommendation, each item is represented as a sequence of discrete codes, and an autoregressive model is trained to generate the SID sequence of the next item; top- K K performance is then measured by checking whether the SID sequence of the target item appears among the generated sequences. This evaluation protocol equates SID-level matching with item-level recommendation, an equivalence that holds only when every SID sequence maps to a single item. We show this assumption breaks down in practice: because tokenizers compress item features into a code space, semantically similar but collaboratively distinct items are frequently assigned the same SID sequence. Across four datasets and five representative tokenizers, the fraction of items involved in such collisions reaches 30.5%, so matching a shared SID sequence identifies only a collision group rather than the target item. Consequently, SID-level metrics overestimate item-level performance (Hit@10 is inflated by up to 103.36%), and the inflation grows with the collision rate. To support faithful comparison, we develop collision-aware item-level metrics computed directly from generated SID sequences, together with a post-tokenizer procedure that reassigns last-level SIDs at minimum cost to obtain a collision-free assignment for any existing tokenizer. Our results indicate that SID-level rankings in prior work should be interpreted with caution, and that reliable tokenizer evaluation requires either item-level correction or collision-free SID assignments. 1. Introduction Generative recommendation has emerged as a paradigm that represents items as Semantic ID (SID) sequences and trains an autoregressive model to generate the SID sequence of the next item from a user’s interaction history ( 31 ; 3 ; 23 ; 1 ) . In this paradigm, item tokenization plays a central role by mapping items to discrete code sequences that serve as the generator’s training targets ( 38 ; 39 ) . Consequently, recent studies have increasingly focused on quantization-based tokenizers and recommendation-aware tokenizer objectives to improve the quality of SID representations ( 21 ; 11 ; 24 ; 37 ; 25 ) . However, the evaluation protocols used to compare these tokenizers implicitly assume that each generated SID sequence uniquely identifies one item. Common evaluation protocols in SID-based generative recommendation compute top- K metrics by checking whether the SID sequence of the target item appears in the ranked list of generated SID sequences ( 31 ; 38 ; 15 ) , as illustrated in Figure 1 (a). However, recommendation accuracy is fundamentally item-level: a generated SID sequence should count as correct only if it uniquely maps to the target item. SID collision , where multiple items share the same SID sequence, breaks this unique mapping ( 11 ) . In this case, the generated sequence identifies only a collision group rather than the target item itself, causing SID-level metrics to overestimate item-level performance and bias tokenizer comparisons. Figure 1 (b) contrasts zero collision with SID collision: the generated SID sequence maps to a single item under zero collision, but to a group of items under SID collision. For example, when two similar 3D printers ( Mini 3D Printer and Standard 3D Printer ) share one SID sequence, generating that SID cannot distinguish which is the target item. Thus, what appears to be a correct SID prediction may only be a group-level match, motivating faithful item-level evaluation of generated SID lists. Otherwise, tokenizers with higher collision rates can be favored by inflated SID-level metrics. Table 1 further shows that such collisions are not rare and occur across datasets and tokenizers. Collision rates range from 8.52% (Beauty, RQ-VAE) to 30.52% (Scientific, RK-Means), with RK-Means consistently producing higher rates than RQ-VAE on the same dataset. Because these collisions occur in practice, matching the target SID sequence does not always mean identifying the target item. Therefore, faithful evaluation should assign full item-level credit only when a generated SID sequence identifies the target item, and adjust the credit when SID collision occurs. Figure 1. (a) Overview of SID-based generative recommendation. (b) Under zero collision, each SID sequence identifies one item; under SID collision, the same SID sequence maps to multiple items, inflating estimates of item-level performance. Table 1. SID collision in two representative SID tokenizers across four datasets. Coll.% denotes the fraction of items whose SID sequence is shared with at least one other item, and G_{\max} denotes the size of the largest collision group. Dataset Tokenizer Coll.% G_{\max} Scientific RK-Means 30.52 26 Scientific RQ-VAE 20.42 22 Cell RK-Means 28.52 56 Cell RQ-VAE 14.04 34 Beauty RK-Means 21.58 18 Beauty RQ-VAE 8.52 12 Yelp RK-Means 15.73 65 Yelp RQ-VAE 9.05 65 For faithful item-level evaluation under SID collision, we introduce Collision-Corrected Evaluation (CCE) and use ItemHit@ K and ItemNDCG@ K as the main metrics, distinguishing them from SID-level Hit@ K and NDCG@ K . When a matched SID sequence is shared across multiple items, these metrics divide the item-level credit by the collision-group size because the SID sequence no longer uniquely identifies a single item. We also report conventional Hit@ K and NDCG@ K as SID-level reference metrics for comparability with previous work. CCE provides faithful item-level scores even when SID collisions remain. However, CCE still evaluates each tokenizer on its native SID outputs, whose collision rates range from 8.52% to 30.52% in Table 1 . Thus, tokenizers are still compared under different degrees of SID collision. To compare tokenizers without SID collisions affecting the comparison, we need a zero-collision SID assignment that stays close to each tokenizer’s original assignment; otherwise, performance differences may be caused by excessive changes to the SID assignment rather than by removing collisions. To this end, we introduce Zero-Collision Reassignment (ZCR), a post-tokenizer method that constructs a zero-collision SID assignment with minimum reassignment cost. In summary, our contributions are as follows: • We identify collision-induced metric inflation , where SID-level metrics overestimate item-level recommendation performance under SID collision. • We introduce CCE, using ItemHit@ K and ItemNDCG@ K to measure faithful item-level performance from the generated SID sequences. • We introduce ZCR, a post-tokenizer method that constructs zero-collision SID assignments with minimum reassignment cost. • We conduct experiments across four datasets and five representative SID tokenizers, demonstrating that higher collision rates result in larger metric inflation and that this inflation is large enough to flip pairwise tokenizer comparisons under item-level re-evaluation. For reproducibility, our implementation and the pre-built SID indexes for all tokenizers and datasets will be made publicly available. 2. Task Formulation 2.1. SIDs-based Generative Recommendation Let \mathcal{U} and \mathcal{I} denote the user set and item set, respectively, where |\mathcal{I}|=N is the number of items. Each user has a chronological interaction history (i_{1},i_{2},\ldots,i_{t}) , where i_{t}\in\mathcal{I} denotes the item interacted with at time step t , and the task is to predict the next item i_{t+1}\in\mathcal{I} (target item). SID-based generative recommendation comprises two core components: item tokenization and SID-based generation. (i) Item Tokenization. Each item i\in\mathcal{I} is mapped by a tokenizer to a fixed length- L discrete code sequence (i.e., SID sequence): (1) \mathbf{s}_{i}=[s_{i}^{(1)},\ldots,s_{i}^{(L)}]\in\{1,\ldots,V\}^{L},\quad i\in\mathcal{I}, where V is the codebook size and L is the length of SID sequence. At level l , let \mathcal{E}^{(l)}=\{\mathbf{e}_{1}^{(l)},\ldots,\mathbf{e}_{V}^{(l)}\} denote the