Multimodal Quantitative Language for Generative Recommendation

Jianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, Yonghong Tian · 2025

Read Multimodal Quantitative Language for Generative Recommendation by Jianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, Yonghong Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05314.txt.

Introduction

Jianyang Zhai and co-authors propose a shared language for recommendation items in a paper from 2025. Generative recommendation skips ranking lists and directly writes out the ID of the next item a user may want. Language models hold useful prior knowledge, but it does not match shop data well.

Items also carry text, images, and other modes that most methods ignore. The new approach, MQL4GRec, turns items from many domains into one quantitative language. That language bridges raw item data and the language model's skills. Knowledge can then transfer across shops and item types. The design aims to use the full, multi-sided picture of user taste.

Catalog record

CREATORJianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, Yonghong Tian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05314

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Quantitative Language for Generative Recommendation Jianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, Yonghong Tian Abstract Generative recommendation has emerged as a promising paradigm aiming at directly generating the identifiers of the target candidates. Most existing methods attempt to leverage prior knowledge embedded in Pre-trained Language Models (PLMs) to improve the recommendation performance. However, they often fail to accommodate the differences between the general linguistic knowledge of PLMs and the specific needs of recommendation systems. Moreover, they rarely consider the complementary knowledge between the multimodal information of items, which represents the multi-faceted preferences of users. To facilitate efficient recommendation knowledge transfer, we propose a novel approach called Multimodal Quantitative Language for Generative Recommendation (MQL4GRec). Our key idea is to transform items from different domains and modalities into a unified language, which can serve as a bridge for transferring recommendation knowledge. Specifically, we first introduce quantitative translators to convert the text and image content of items from various domains into a new and concise language, known as quantitative language, with all items sharing the same vocabulary. Then, we design a series of quantitative language generation tasks to enrich quantitative language with semantic information and prior knowledge. Finally, we achieve the transfer of recommendation knowledge from different domains and modalities to the recommendation task through pre-training and fine-tuning. We evaluate the effectiveness of MQL4GRec through extensive experiments and comparisons with existing methods, achieving improvements over the baseline by 11.18%, 14.82%, and 7.95% on the NDCG metric across three different datasets, respectively. 1 1 1 Our implementation is available at: https://github.com/zhaijianyang/MQL4GRec . 1 Introduction Figure 1: Illustration of our MQL4GRec. We translate items from different domains and modalities into a new unified language, which can then serve as a bridge for transferring recommendation knowledge. Recommendation systems (RS) aim to recommend items to users that they may be interested in, and are widely used on many online platforms, such as e-commerce and social networking ( Chaves et al. 2022 ; Covington et al. 2016 ) . For a long time, recommendation models that represent users and items using their unique IDs (known as IDRec) have been dominant in the field of RS ( Kang & McAuley 2018a ; Sun et al. 2019 ; Zhang et al. 2024 ) . However, IDRec may encounter cold start and knowledge transferability issues due to its inherent properties. To address these limitations, some literature ( Hou et al. 2022 ; Sun et al. 2023 ) employs modal encoders ( Devlin et al. 2018 ; He et al. 2016 ) to learn universal representations of items or sequences. While promising, these modal encoders are typically not specifically designed for recommendation tasks, resulting in suboptimal performance. Recently, generative recommendation has emerged as a promising paradigm, which employs an end-to-end generative model to directly predict identifiers of target candidates ( Geng et al. 2022 ; Rajput et al. 2023 ) . Due to the success of PLMs in natural language generation (NLG) ( Raffel et al. 2020a ; Brown et al. 2020 ; Touvron et al. 2023 ) , most existing methods attempt to leverage the prior knowledge of PLMs to improve the recommendation performance ( Bao et al. 2023 ; Zhang et al. 2023 ; Zheng et al. 2023 ) . They formalize the recommendation task as a sequence-to-sequence generation process, where the input sequence contains data of items interacted with users, and the output sequence represent identifiers of target items. Then they enable PLMs to perform recommendation tasks by adding instructions or prompts. Despite achieving decent performance, they suffer from the following limitations: 1) There are significant task differences between PLMs and RS, which may lead to inconsistencies between the general linguistic knowledge of PLMs and the specific requirements of RS; 2) They often overlook the complementary knowledge between the multimodal information of items, which is crucial for capturing the multi-faceted preferences of users. To address these limitations, it is crucial to bridge the gaps between different domains and modalities, leveraging their recommendation knowledge to enhance the performance of the target domains. Inspired by significant advancements in NLG, such as pretraining-finetuning ( Devlin et al. 2018 ; Raffel et al. 2020b ) and prompt-tuning ( Brown et al. 2020 ; Touvron et al. 2023 ) , we propose the idea of transforming items from various domains and modalities into a new and unified language. A key factor contributing to these significant advances is the use of a shared vocabulary, where tokens are endowed with rich semantic information and prior knowledge across various tasks, which can then be effectively transferred to downstream tasks. Thus, we aspire for this new language to encompass a vocabulary in which tokens can represent items from various domains and modalities, as depicted in Figure 1 . Specifically, this language not only serves as a bridge for knowledge transfer but also as identifiers of items, and should be more concise than the original modalities (text and image) to avoid issues in generation ( Hua et al. 2023 ) . To this end, we propose a novel approach known as Quantitative Language for Multimodal Generative Recommendation (MQL4GRec). Specifically, we first introduce quantitative translators to convert the content of items (text and images) into the quantitative language. We train a separate quantitative translator for each modality of the item, each consisting of a modal encoder and a vector quantizer. Together, the codebooks of the two quantitative translators constitute the vocabulary. Then, we design a series of quantitative language generation tasks aiming at endowing quantitative language with rich semantic information and prior knowledge, and these tasks can be viewed as microcosms of NLG tasks. Specifically, we additionally incorporate some special tokens as task prompts. Finally, we transfer the source domain and multimodal recommendation knowledge to the recommendation tasks through pre-training and fine-tuning. To evaluate the effectiveness of our proposed MQL4GRec, we conduct extensive experiments and comparisons with existing methods. Relative to the baseline, we observe improvements of 11.18%, 14.82%, and 7.95% on the NDCG metric across three datasets, respectively. In summary, our proposed MQL4GRec achieves the transfer of recommendation knowledge by breaking down barriers between items across different domains and modalities, demonstrating strong scalability and potential. Our main contributions can be summarized as follows: • We propose MQL4GRec, a novel approach that translates items from various domains and modalities into a unified quantitative language, thereby breaking down the barriers between them and facilitating the transfer of recommendation knowledge. • We design a series of quantitative language generation tasks that endow quantitative language with rich semantic information and prior knowledge, and enhance the performance of recommendation tasks through pre-training and fine-tuning. • We conduct extensive experiments and analyses on three public datasets, and the results validate the effectiveness of our proposed method. 2 Related Works Generative Recommendation. Generative models are one of the hottest research topics in machine learning, resulting in some representative works such as Variational AutoEncoders (VAEs) ( Kingma & Welling 2014 ) , Generative Adversarial Networks (GANs) ( Goodfellow et al. 2014 ) and Diffusion models ( Ho et al. 2020 ) . Generally, generative models aim to learn the distribution of the training data \mathbb{P}(\mathbf{x}) and generate new samples \mathbf{z}\sim\mathbb{P}(\mathbf{x}) .

Read the complete text