Discrete Preference Learning for Personalized Multimodal Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Personalization; Multimedia content creation
Yuting Zhang, Ying Sun, Dazhong Shen, Ziwei Xie, Feng Liu, Changwang Zhang, Xiang Liu, Jun Wang, Hui Xiong · 2026
Read Discrete Preference Learning for Personalized Multimodal Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Personalization; Multimedia content creation by Yuting Zhang, Ying Sun, Dazhong Shen, Ziwei Xie, Feng Liu, Changwang Zhang, Xiang Liu, Jun Wang, Hui Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20434.txt.
Introduction
Zhang and eight colleagues tackle personalized multimodal generation, where text and images fit a user's tastes. They note two gaps: no dedicated way to model tastes well, and outputs stuck in one mode despite users mixing text and images. Their two-stage framework, DPPMG, learns tastes per mode with a graph network, turns them into discrete preference tokens, and feeds those tokens to text and image generators. A cross-mode reward keeps image and text consistent while preserving personalization, and tests on two real datasets support the design.
Catalog record
| CREATOR | Yuting Zhang, Ying Sun, Dazhong Shen, Ziwei Xie, Feng Liu, Changwang Zhang, Xiang Liu, Jun Wang, Hui Xiong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48533 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20434
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Discrete Preference Learning for Personalized Multimodal Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Personalization; Multimedia content creation Yuting Zhang, Ying Sun, Dazhong Shen, Ziwei Xie, Feng Liu, Changwang Zhang, Xiang Liu, Jun Wang, Hui Xiong Abstract . The emergence of generative models enables the creation of texts and images tailored to users’ preferences. Existing personalized generative models have two critical limitations: lacking a dedicated paradigm for accurate preference modeling, and generating unimodal content despite real-world multimodal-driven user interactions. Therefore, we propose personalized multimodal generation, which captures modal-specific preferences via a dedicated preference model from multimodal interactions, and then feeds them into downstream generators for personalized multimodal content. However, this task presents two challenges: (1) Gap between continuous preferences from dedicated modeling and discrete token inputs intrinsic to generator architectures; (2) Potential inconsistency between generated images and texts. To tackle these, we present a two-stage framework called D iscrete P reference learning for P ersonalized M ultimodal G eneration (DPPMG). In the first stage, to accurately learn discrete modal-specific preferences, we introduce a modal-specific graph neural network (a dedicated preference model) to learn users’ modal-specific preferences, which preferences are then quantized into discrete preference tokens. In the second stage, the discrete modal-specific preference tokens are injected into downstream text and image generators. To further enhance cross-modal consistency while preserving personalization, we design a cross-modal consistent and personalized reward to fine-tune token-associated parameters. Extensive experiments on two real-world datasets demonstrate the effectiveness of our model in generating personalized and consistent multimodal content. † † cc-license: by 1. Introduction Aiming to infer user preferences from historical behaviors and provide tailored content accordingly, personalized systems (e.g., recommenders ( 62 ) ) have played a crucial role in various online platforms. However, most personalized systems focus on mining user preferences and retrieving existing contents accordingly ( 52 ) . The rapid development of generative models, such as LLMs ( 1 ; 11 ) and Diffusion models (DMs) ( 35 ; 31 ) , has spurred the emergence of a personalized generation paradigm. It directly generates content tailored to individual preferences, which is becoming increasingly important across various domains, including movie marketing ( 40 ; 57 ) , advertising ( 58 ; 5 ) , music ( 7 ; 30 ) , and fashion designs ( 56 ) . Nevertheless, existing personalized generation methods struggle to accurately extract user preferences. For instance, personalized text generation ( 38 ; 44 ) typically relies on retrieving text from users’ interaction history for generation guidance. The LaMP benchmark ( 39 ) evaluates several retrieval-augmented strategies, including BM25 ( 48 ) , Recency and Contriever ( 17 ) . However, retrieved text often contains noise, mixing users’ preferred and dispreferred features (e.g., style, word choice), hindering preference-aligned text generation. The same issue also persists in personalized image generation. Existing works directly use generation models (DMs ( 10 ; 37 ; 56 ) , LLMs ( 40 ) or MLLMs ( 57 ) ) to learn preferences from users’ historical images. For instance, PMG ( 40 ) converts historical images into textual descriptions and employs generative LLMs to extract keywords for guiding image generation. However, these generative approaches lack a dedicated preference learning paradigm , such as classical collaborative filtering that mines user-item interactions and user/item similarity patterns ( 42 ) . Consequently, they fail to accurately extract user preferences from complex historical data. Furthermore, users’ content demands are inherently multimodal, as evidenced by the multimodal content deployed across mainstream platforms to capture attention. For example, advertising platforms provide visual images with textual copy; movie marketing pairs posters with introductions; short videos integrate text, images, and audio elements. These modalities collectively drive user engagement ( 25 ; 55 ) , making the isolated single-modal generation paradigms inherently inadequate . Thus, we introduce the personalized multimodal generation task, which technically aims to capture users’ modality-specific preferences from multimodal interaction histories and leverage these to generate tailored multimodal content (e.g., text-image pairs). However, this task poses two major challenges: (1) How to accurately extract user preferences and effectively incorporate them into downstream generators. Traditional preference models such as graph neural networks ( 12 ; 53 ; 13 ) excel at extracting users’ preferences using collaborative signals but represent these preferences as continuous embeddings. In contrast, current mainstream generative models (e.g., LLMs with Transformer backbones and diffusion models with attention layers) inherently rely on discrete sequences as input, creating a fundamental gap. (2) How to realize consistent personalized multimodal generation. Performing single-modal generation ( 39 ; 38 ; 44 ) separately might neglect cross-modal alignment , resulting in disjointed image-text semantics and inconsistent styles. For example, as Figure 1 shows, a generated image depicting a piece of modern minimalist furniture may be paired with an elaborate, gorgeous textual description, which will directly degrade user experience and satisfaction. Figure 1. Challenges faced by personalized multimodal content generation: (1) Gap between continuous preference representations and generators’ discrete token sequence; (2) Inconsistency between generated images and texts. To address these challenges, we propose a Discrete Preference Learning framework for Personalized Multimodal Generation. The framework comprises two core stages: (1) We first introduce a modal-aware graph neural network to accurately model users’ modal-specific preference representations, and then quantize them into discrete preference tokens. This process is guided by specially designed cross-modal and preference-oriented tasks. (2) We then integrate the discretized modal-specific preference tokens into the downstream text and image generators. To further enhance cross-modal consistency while preserving personalization, we design a cross-modal and personalized reward function to fine-tune preference token-related parameters. To summarize, our main contributions are listed below: • To our knowledge, we are among the first to explicitly enable personalized multimodal generation, which captures users’ modality-aware preferences from multimodal interaction history data. • We propose discrete preference token learning, which adopts vector quantization to align continuous preference embeddings with discrete inputs natively required by generative architectures. • During training, we freeze the pre-trained generative backbones and optimize lightweight token-related parameters as additional generation conditions, ensuring high training efficiency. • Extensive experiments and human evaluation on two real-world datasets verify the effectiveness of our model in generating personalized, consistent multimodal content. 2. Related Work 2.1. Multimodal Content Generation Multimodal content generation targets cross-modal consistent and preference-aligned content creation. Existing works ( 1 ; 45 ; 15 ) feed explicit instructions (e.g., Pixar Style) for instruction-aligned generation. In contrast, our personalized generation infers implicit preferences from users’ multimodal historical interactions, analogous to recommenders ( 62 ; 12 ; 43 ) , and generates consistent preference-aligned novel content. Few studies explicitly generate personalized multimodal content while ensuring consistency. We thus review unimodal personalized generation (text and image) respectively: Personalized Text Generation . It has been widely applied in various applications, such as dialogue agents ( 64 ; 47 ) , headline ( 2 ) and product description generation ( 9 ; 3 ) . With the rise of LLMs ( 11 ; 1