LLaDA-Rec: Discrete Diffusion for Parallel Semantic ID Generation in Generative Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information system

Teng Shi, Chenglei Shen, Weijie Yu, Shen Nie, Chongxuan Li, Xiao Zhang, Ming He, Yan Han, Jun Xu · 2025

Read LLaDA-Rec: Discrete Diffusion for Parallel Semantic ID Generation in Generative Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information system by Teng Shi, Chenglei Shen, Weijie Yu, Shen Nie, Chongxuan Li, Xiao Zhang, Ming He, Yan Han, Jun Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06254.txt.

Introduction

Teng Shi and colleagues rebuild generative recommendation around parallel generation instead of one-token-at-a-time decoding. In generative recommendation, each item is a short sequence of discrete tokens called a semantic ID. Standard models generate those tokens left to right, which blocks attention to later tokens and lets early mistakes poison the rest.

Their system, LLaDA-Rec, uses discrete diffusion to produce the tokens in parallel with bidirectional attention and a flexible generation order. A parallel tokenization scheme, two masking designs, and an adapted beam search make the approach workable. On three real-world datasets it beat both classic ID-based recommenders and leading generative ones.

Catalog record

CREATORTeng Shi, Chenglei Shen, Weijie Yu, Shen Nie, Chongxuan Li, Xiao Zhang, Ming He, Yan Han, Jun Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLaDA-Rec: Discrete Diffusion for Parallel Semantic ID Generation in Generative Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems Teng Shi, Chenglei Shen, Weijie Yu, Shen Nie, Chongxuan Li, Xiao Zhang, Ming He, Yan Han, Jun Xu Abstract . Generative recommendation represents each item as a semantic ID, i.e., a sequence of discrete tokens, and generates the next item through autoregressive decoding. While effective, existing autoregressive models face two intrinsic limitations: (1) unidirectional constraints, where causal attention restricts each token to attend only to its predecessors, hindering global semantic modeling; and (2) error accumulation, where the fixed left-to-right generation order causes prediction errors in early tokens to propagate to the predictions of subsequent token. To address these issues, we propose LLaDA-Rec , a discrete diffusion framework that reformulates recommendation as parallel semantic ID generation. By combining bidirectional attention with the adaptive generation order, the approach models inter-item and intra-item dependencies more effectively and alleviates error accumulation. Specifically, our approach comprises three key designs: (1) a parallel tokenization scheme that produces semantic IDs for bidirectional modeling, addressing the mismatch between residual quantization and bidirectional architectures; (2) two masking mechanisms at the user-history and next-item levels to capture both inter-item sequential dependencies and intra-item semantic relationships; and (3) an adapted beam search strategy for adaptive-order discrete diffusion decoding, resolving the incompatibility of standard beam search with diffusion-based generation. Experiments on three real-world datasets show that LLaDA-Rec consistently outperforms both ID-based and state-of-the-art generative recommenders, establishing discrete diffusion as a new paradigm for generative recommendation. 1. Introduction In recent years, generative language models ( Achiam et al. 2023 ; Zhao et al. 2023 ; Yang et al. 2025a ; Liu et al. 2024 ) have exhibited remarkable capabilities, motivating a growing number of studies to explore the application of the generative paradigm to recommendation ( Rajput et al. 2023 ; Wang et al. 2024a ; Zheng et al. 2024 ; Hou et al. 2025 ; Zhou et al. 2025 ; Deng et al. 2025 ) . In contrast to traditional discriminative recommendation models ( Kang and McAuley 2018 ; Sun et al. 2019 ; Zhou et al. 2022 ) , generative recommendation reformulates the recommendation task as a sequence-to-sequence generation problem, where the next target item is generated in an autoregressive manner. Generative recommendation typically comprises two key components: item tokenization and autoregressive generation. In item tokenization, each item is assigned a Semantic ID (SID), represented as a sequence of discrete tokens. Common techniques include hierarchical clustering ( Si et al. 2024 ) (e.g., K-means) and vector quantization (e.g., RQ-VAE ( Rajput et al. 2023 ; Zheng et al. 2024 ) ). The user history encoded as discrete tokens is subsequently fed into the generative model, which autoregressively produces the tokens of the next item in a step-by-step manner. Despite their effectiveness, existing methods still exhibit several limitations: (1) Unidirectional Constraint . Most approaches rely on autoregressive architectures that predict the tokens of each item in a left-to-right manner. This left-to-right dependency restricts the model’s ability to capture global relationships among all tokens that jointly define an item ( Gu et al. 2022 ) . As a result, the generated items often exhibit limited semantic coherence and expressiveness. (2) Error Accumulation . During inference, models generate each token conditioned on previously sampled ones, in contrast to the training phase where teacher forcing ( Raffel et al. 2020 ) is employed and the ground-truth token is provided at each step. As a result, errors made on earlier tokens during inference cannot be corrected and are likely to propagate to subsequent tokens, thereby amplifying their adverse effects ( Lin et al. 2025 ) . Figure 1 provides a further illustration. Figure 1. An illustration of the advantages of discrete diffusion over autoregressive generation. In autoregressive models, an error in the first token will propagate to subsequent tokens. In contrast, a discrete diffusion model predicts all masked positions in parallel at each step, and re-masks and re-predicts low-confidence error tokens, ultimately producing more accurate results. To address these limitations, we explore a new formulation of generative recommendation based on discrete diffusion ( Li et al. 2025a ; Nie et al. 2025 ; Zhu et al. 2025 ; Yang et al. 2025b ) . During training, the model employ a forward token masking noise process followed by a reverse denoising reconstruction process, with a bidirectional Transformer parameterizing the prediction of masked tokens. This design offers two key advantages: (1) Bidirectional Understanding . The bidirectional architecture enhances contextual comprehension and enables the modeling of global relationships among all tokens. (2) Adaptive Generation Order . During inference, all tokens are predicted in parallel, while low - confidence tokens are iteratively re - masked and re - predicted. This yields a dynamic rather than fixed generation order, prioritizing tokens with higher model confidence (i.e., easier tokens) and mitigating error accumulation. Overall, this discrete diffusion framework offers a compelling solution to the limitations identified in prior generative recommendation approaches. Although diffusion models inherently operate under a fixed - length generation constraint ( Nie et al. 2025 ) , this property naturally aligns with recommendation tasks, where each item is represented by a fixed number of semantic IDs—making diffusion particularly well - suited for generative recommendation. However, applying discrete diffusion to generative recommendation still presents several challenges: (1) Mismatch between Residual Quantization (RQ) and Discrete Diffusion. Since discrete diffusion relies on a bidirectional Transformer where all tokens are equally important, it requires parallel tokenization schemes beyond hierarchical ones like RQ-VAE. (2) Beam Search is Not Directly Applicable to Discrete Diffusion. Unlike standard discrete diffusion models that rely on probabilistic sampling, recommendation tasks require generating the top- k outputs. Beam search, which is widely used in autoregressive top- k generation ( Rajput et al. 2023 ; Zheng et al. 2024 ) , is primarily designed for fixed left-to-right decoding and thus cannot be directly applied to discrete diffusion with an adaptive generation order, thereby requiring specialized decoding strategies tailored to the discrete diffusion setting. (3) Differences between Language Modeling and Recommendation. Original discrete diffusion models are primarily developed for language modeling. When applied to recommendation tasks, they need to be adapted to capture intra - item semantics and inter - item sequential relations within user histories. To address the above challenges, we propose LLaDA-Rec , a bidirectional generative recommendation method based on discrete diffusion. Our approach consists of three main modules: (1) Parallel Tokenization . We design Multi-Head VQ-VAE that encode the item embedding and split it into several sub - vectors, each queried against a separate codebook, ultimately producing parallel semantic IDs. (2) Discrete Diffusion Training . We employ two masking mechanisms: a User - History level masking to model inter - item sequential relationships, and a Next - Item level mask, with the history fixed, to capture intra - item semantics, enabling the model to better understand the semantic relationships among different tokens within an item. (3) Discrete Diffusion Inference . We adapt beam search for diffusion model. At each step, we first select the position with the highest model confidence

Read the complete text