Diffusion Generative Recommendation with Continuous Tokens
Haohao Qu, Shanru Lin, Yujuan Ding, Yiqi Wang, Wenqi Fan · 2025
Read Diffusion Generative Recommendation with Continuous Tokens by Haohao Qu, Shanru Lin, Yujuan Ding, Yiqi Wang, Wenqi Fan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12007.txt.
Introduction
Haohao Qu and co-authors propose in 2025 ContRec, a recommendation frame that feeds continuous tokens to a language model. Past model-based recommenders chop users and items into discrete codes, which loses detail. Quantized codes also block clean gradient flow during training.
A variational tokenizer encodes users and items as smooth continuous tokens. Three tricks guard the code space against collapse during training. A diffusion module then grows user taste step by step, guided by past tokens. A spread loss sharpens the taste forms the model makes. The blend aims for finer picks than code-based rivals.
Catalog record
| CREATOR | Haohao Qu, Shanru Lin, Yujuan Ding, Yiqi Wang, Wenqi Fan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12007
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Generative Recommendation with Continuous Tokens Haohao Qu, Shanru Lin, Yujuan Ding, Yiqi Wang, Wenqi Fan Abstract . Recent advances in generative artificial intelligence, particularly large language models (LLMs), have opened new opportunities for enhancing recommender systems (RecSys). Most existing LLM-based RecSys approaches operate in a discrete space, using vector-quantized tokenizers to align with the inherent discrete nature of language models. However, these quantization methods often result in lossy tokenization and suboptimal learning, primarily due to inaccurate gradient propagation caused by the non-differentiable argmin operation in standard vector quantization. Inspired by the emerging trend of embracing continuous tokens in language models, we propose ContRec , a novel framework that seamlessly integrates continuous tokens into LLM-based RecSys. Specifically, ContRec consists of two key modules: a σ \sigma -VAE Tokenizer, which encodes users/items with continuous tokens; and a Dispersive Diffusion module, which captures implicit user preference. The tokenizer is trained with a continuous Variational Auto-Encoder (VAE) objective, where three effective techniques are adopted to avoid representation collapse. By conditioning on the previously generated tokens of the LLM backbone during user modeling, the Dispersive Diffusion module performs a conditional diffusion process with a novel Dispersive Loss, enabling high-quality user preference generation through next-token diffusion. Finally, ContRec leverages both the textual reasoning output from the LLM and the latent representations produced by the diffusion model for Top-K item retrieval, thereby delivering comprehensive recommendation results. Extensive experiments on four datasets demonstrate that ContRec consistently outperforms both traditional and state-of-the-art LLM-based recommender systems. Our results highlight the potential of continuous tokenization and generative modeling for advancing the next generation of recommender systems. Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 10.1145/3774904.3792428 ISBN: 979-8-4007-2307-0/2026/04 CCS: Information systems Web mining † † cc-license: by Resource Availability: The source code of this paper has been made publicly available at https://doi.org/10.5281/zenodo.17285302 . Figure 1. Comparison of five representative deep generative models, namely VQ-VAE ( van den Oord et al., 2017 ) , RQ-VAE ( Rajput et al., 2023 ) , MQ-VAE ( Qu et al., 2025 ) , VAE ( Kingma and Welling, 2014 ) , and Diffusion ( Ho et al., 2020 ) , in reconstructing item embeddings on the Beauty dataset. The standard diffusion model with continuous embeddings demonstrates superior reconstruction performance and loss convergence compared to the vanilla VAE and its discrete counterparts. 1. Introduction Web mining is a pivotal research area that focuses on extracting valuable knowledge and patterns from vast and heterogeneous web data. Within this domain, recommender systems (RecSys) serve as crucial tools to tackle information overload and enrich user experiences on various web applications such as e-commerce, job search, and social media platforms ( Wang et al., 2020 ; Fan et al., 2019 ; Fan et al., 2020 ) . The emerging focus on Generative Artificial Intelligence (GenAI), especially Large Language Models ( LLMs ), has spurred a growing interest in Generative Recommendation , driven by the desire to leverage their exceptional generalization capabilities and adeptness at in-context learning ( Zhao et al., 2024 ; Wang et al., 2025b ) . These distinct abilities empower LLM-based RecSys to generalize more effectively across new tasks and domains. Most existing LLM-based RecSys utilize LLM as the backbone, therefore inevitably inherit its core architecture design centered on discrete token representations ( Zhao et al., 2023 ; Hirschberg and Manning, 2015 ) . For example, basic settings for item/user representation in RecSys include numeric indexing (e.g., " item_2025 ") and textual title indexing (e.g., " Apple iPhone 16, 1TB, White ") ( Hua et al., 2023 ; Geng et al., 2022 ; Bao et al., 2023 ; Wei et al., 2024 ) . However, these approaches become inefficient when dealing with large-scale real-world recommender systems, in which the sizes of user and item sets typically reach millions. To address the token set overload issue, Vector-Quantized Variational Autoencoder (VQ-VAE) ( van den Oord et al., 2017 ; Qu et al., 2025 ) has been applied to construct a discrete vocabulary (also referred to as a "codebook", such as " \langle a-3\rangle\langle b-24\rangle\langle c-58\rangle\langle d-12\rangle ") and train a neural tokenizer on users and items. This strategy has been widely employed in a series of LLM-based RecSys, including TIGER ( Rajput et al., 2023 ) , UTGRec ( Zheng et al., 2025 ) , and OneRec ( Deng et al., 2025 ) , among others. These quantization-based solutions may be practical for encapsulating rich item-side information into a sequence of tokens that align with the discrete nature of language models, but they still encounter two critical limitations. First, their reconstruction quality and generation performance are constrained by lossy tokenization, as some information will inevitably be compressed and omitted during the quantization process ( Fan et al., 2025a ; Chen et al., 2025 ; Sun et al., 2024 ) . Second, due to the non-differentiable argmin in typical vector quantization, a “straight-through” trick ( Bengio et al., 2013 ) is normally used, directly copying the gradient from the decoder input to the encoder output. Such a surrogate gradient can be inaccurate, potentially impeding both the achievement of high compression ratios and the effective learning of the latent space in quantization-based methods ( Huh et al., 2023 ; Fifty et al., [n. d.] ) . Such limitations of discrete learning-based models have garnered attention in language tasks, especially for multimodal modeling (e.g., image, video, and protein) ( Wang et al., 2024 ; Wang et al., 2025c ) . For example, Fan et al. (2025a) reveal that quantization-based visual models exhibit poor performance improvements when scaling up model size, compared to models operating on continuous tokens. Furthermore, there have been increasing efforts to introduce continuous tokens for learning better representations ( Li et al., 2024 ) . For example, Chen et al. (2025) introduce an efficient continuous image tokenizer for language models, which achieves a high compression ratio while enriching the semantic content of the latent space for image reconstruction. These advancements in language models inspire the exploration of continuous tokens as a new tokenization paradigm for LLM-based generative recommendation. To achieve the goal, a few LLM-based recommendation methods have attempted to include continuous tokens. For example, CoLLM ( Zhang et al., 2025a ) and LlaRA ( Liao et al., 2024 ) propose directly projecting collaborative item representations learned from traditional recommendation methods into the token space of language models. However, these methods only continualize the input part of the model by directly employing simple projection layers, e.g., a multi-layer perceptron (MLP), for user/item tokenization, while their output part inherits the LLM backbone’s generator, which is learned for in-vocabulary discrete tokens. Such discrete-continuous gaps in one model might degrade its unification ( Sun et al., 2024 ) and result in suboptimal recommendation performance. To enable the seamless incorporation of continuous tokens into LLM-based generative recommendation, we propose leveraging the powerful Diffusion Model to continualize user preference modeling and generation, alongside the tokenization of users and items. It is widely recognized that user preference modeling is the core technical part leading to effective recommendation, while this task is highly