MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation CCS: Information systems Recommender systems

Yi Xu, Moyu Zhang, Chenxuan Li, Zhihao Liao, Haibo Xing, Hao Deng, Jinxin Hu, Yu Zhang, Xiaoyi Zeng, Jing Zhang · 2025

Read MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation CCS: Information systems Recommender systems by Yi Xu, Moyu Zhang, Chenxuan Li, Zhihao Liao, Haibo Xing, Hao Deng, Jinxin Hu, Yu Zhang, Xiaoyi Zeng, Jing Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15281.txt.

Introduction

Yi Xu and co-authors present a 2025 tokenizer that turns goods into meaning-based codes for shopping suggestions. Plain ID codes fail for new or rare goods and learn little across items. Codes built from text and images can share what goods have in common, yet past tries blurred what each sense adds or drifted from what users truly like.

Their two-stage method first splits shared and sense-specific experts to keep both sides. It then tunes the codes toward real user behavior while keeping their content roots. Lab tests and live trials showed gains for both search and ranking of goods.

Catalog record

CREATORYi Xu, Moyu Zhang, Chenxuan Li, Zhihao Liao, Haibo Xing, Hao Deng, Jinxin Hu, Yu Zhang, Xiaoyi Zeng, Jing Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation CCS: Information systems Recommender systems Yi Xu, Moyu Zhang, Chenxuan Li, Zhihao Liao, Haibo Xing, Hao Deng, Jinxin Hu, Yu Zhang, Xiaoyi Zeng, Jing Zhang Abstract . Recommender systems traditionally represent items using unique identifiers (ItemIDs), but this approach struggles with large, dynamic item corpora and sparse long-tail data, limiting scalability and generalization. Semantic IDs, derived from multimodal content such as text and images, offer a promising alternative by mapping items into a shared semantic space, enabling knowledge transfer and improving recommendations for new or rare items. However, existing methods face two key challenges: (1) balancing cross-modal synergy with modality-specific uniqueness, and (2) bridging the semantic-behavioral gap, where semantic representations may misalign with actual user preferences. To address these challenges, we propose Multimodal Mixture-of-Quantization (MMQ), a two-stage framework that trains a novel multimodal tokenizer. First, a shared-specific tokenizer leverages a multi-expert architecture with modality-specific and modality-shared experts, using orthogonal regularization to capture comprehensive multimodal information. Second, behavior-aware fine-tuning dynamically adapts semantic IDs to downstream recommendation objectives while preserving modality information through a multimodal reconstruction loss. Extensive offline experiments and online A/B tests demonstrate that MMQ effectively unifies multimodal synergy, specificity, and behavioral adaptation, providing a scalable and versatile solution for both generative retrieval and discriminative ranking tasks. 1. Introduction In recommender systems, a common approach represents items using their unique identifiers (ItemIDs) ( singh2024bettergeneralizationsemanticids ; zheng2025enhancingembeddingrepresentationstability ) . However, this paradigm faces critical limitations in real-world scenarios with large, dynamic item corpora. First, high item turnover and shifting popularity trends challenge the stability and scalability of static ID-based embeddings ( Milojevi__2010 ; 10.1145/2523813 ) . Second, data sparsity for long-tail items leads to poorly generalized representations ( Li_2021 ; ni2023contentdrivenmicrovideorecommendationdataset ) , collectively limiting model performance and scalability. These issues motivate exploring alternative item representations. Semantic IDs, generated from item content such as text and images via multimodal tokenizers, provide a promising alternative ( rajput2023recommendersystemsgenerativeretrieval ; zheng2025enhancingembeddingrepresentationstability ; singh2024bettergeneralizationsemanticids ) . By mapping items to a shared semantic space, they enable knowledge transfer among similar items, improving recommendations for new or long-tail items. Moreover, representing items through a combination of tokens from a fixed-size semantic vocabulary enhances scalability and efficiency compared to one-to-one ID mappings ( kudo2018subwordregularizationimprovingneural ) . These advantages have facilitated semantic IDs’ application across diverse recommendation frameworks, from generative retrival ( wang2024eagertwostreamgenerativerecommender ; hou2023learningvectorquantizeditemrepresentation ; Deng_2025 ) to discriminative ranking tasks ( Wang_2025 ) . Existing semantic tokenizers primarily leverage text and vision modalities, which provide complementary information. According to Partial Information Decomposition (PID) theory ( wollstadt2023rigorousinformationtheoreticdefinitionredundancy ; liang2023quantifyingmodelingmultimodal ) , item information can be decomposed into modality-unique and synergistic components. Capturing both is crucial: synergistic cross-modal interactions can reveal fine-grained item characteristics that single modalities cannot, while modality-unique details preserve unique signals essential for personalization. As illustrated in Figure 1 , cross-modal interactions between text and vision reveal finer-grained item characteristics: the left appeals to fashion-forward users, whereas the right attracts casual, vacation-oriented users. Properly modeling this synergy alongside modality-unique details enables more precise recommendations ( liu2018efficientlowrankmultimodalfusion ; xin2025i2moeinterpretablemultimodalinteractionaware ; yu2024mmoeenhancingmultimodalmodels ) , such as suggesting fashion pants for the first user and beach sandals for the second. However, current methods struggle with a trade-off between capturing synergy and uniqueness. Approaches fall into two paradigms: (1) Modality Alignment (MA) merges modalities into a unified representation before tokenization, which may obscure modal-specific details due to dominance of one modality; (2) Modality Separation (MS) maintains separate tokenizers per modality, preserving uniqueness but missing cross-modal synergy. Consequently, generating high-quality semantic IDs that fully leverage multimodal information remains a challenge. Another key challenge is the semantic-behavioral gap : semantic IDs are trained in the semantic space, whereas recommendation relies on user behavior. Purely semantic representations can misrepresent behavioral preferences ( wang2024eagertwostreamgenerativerecommender ) . For example, semantically similar items may elicit distinct user interactions, leading to noisy recommendations. Thus, semantic tokenizers must adapt dynamically to downstream tasks rather than remain static. Figure 1 . Illustration of text and vision modal interaction. To address these challenges, we propose Multimodal Mixture-of-Quantization (MMQ), which is a two-stage framework. 1) Multimodal Shared-Specific Tokenizer Training: We introduce a multi-expert architecture with modality-specific and modality-shared experts to explicitly model both unique and synergistic information. In addition, we leverage orthogonal regularization to encourage expert diversity, preventing collapse and ensuring comprehensive representation. 2) Behavior-Aware Fine-Tuning: We adapt semantic ID clusters dynamically using downstream recommendation objectives, bridging the semantic-behavioral gap. A multimodal reconstruction loss is proposed to preserve modality information while fine-tuning. We conduct extensive offline experiments and online A/B tests to evaluate MMQ in both generative retrieval and discriminative ranking tasks. Our main contributions are summarized as follows: • We propose the first unified semantic ID framework that simultaneously captures multimodal synergy and uniqueness while dynamically adapting to user behavior. • We design a multi-expert architecture with orthogonal regularization to disentangle modality-shared and modality-specific information. We introduce a behavior-aware fine-tuning to align semantic IDs with downstream recommendation objectives, which effectively addressed the semantic-behavioral gap. • Extensive experiments across generative and discriminative tasks demonstrate MMQ’s effectiveness, scalability, and versatility. 2. Related Work Recent efforts in item representation have centered on semantic indexing ( jin2024languagemodelssemanticindexers ; luo2024qarmquantitativealignmentmultimodal ; kuai2024breakinghourglassphenomenonresidual ; zheng2025pretraininggenerativerecommendermultiidentifier ; yang2025sparsemeetsdenseunified ) , which generates discrete item identifiers (i.e., tokens) from content information. A dominant approach within this paradigm is to apply quantization or clustering techniques to pre-trained textual embeddings. In particular, TIGER ( rajput2023recommendersystemsgenerativeretrieval ) is a pioneering work that introduces Residual Quantization (RQ-VAE) to quantize items into semantic IDs, enabling new items to share knowledge through semantic similarity without the pre-trained embeddings. SPM-SID ( singh2024bettergeneralizationsemanticids ) and PMA ( zheng2025enhancingembeddingrepresentationstability ) explored the semantic IDs and subwords of Large Language Models (LLMs), using n-gram and SPM(SentencePiece Model) to combine the IDs. Existing works use different quantization methods for tokenization. RQ-Kmeans better addresses the collapse of RQVAE and is widely applied in many works ( deng2025onerecunifyingretrieverank ) . Unlike the ordered tokenization of RQVAE, unordered tokenization ( hou2025generatinglongsemanticids ) supports faster parallel generation. VQ-Rec ( hou2023learningvectorquantizeditemrepresentation ) uses OPQ (Optimized Product Quantization) ( jegou2010product ) to split and disorderly tokenize text vectors from BERT. ActionPiece ( hou2025actionpiececontextuallytokenizingaction ) adopts unordered feature sets and context-aware token merging. Recognizing the limitations of the single content modality, recent research ( liu2025generativerecommenderendtoendlearnable ) explored the integration of collaborative information into the tokenization. A representative work is EAGER ( wang2024eagertwostreamgenerativerecommender ) , which combines semantic information with collaborative signals to generate more informative tokens. LC-Rec ( zheng2024adaptinglargelanguagemodels ) designs multitask semantic alignment fine-tuning, including sequence item prediction, explicit index-language alignment, and implicit alignment tasks. ColaRec ( Wang_2024 ) utilizes a pre-trained collaborative filtering model to incorporate collaborative signals from user-item interactions into hierarchical K-means clustering ( huang2025augmentnotcomparativestudy ) . IDGenRec ( tan2024idgenrecllmrecsysalignmenttextual ) converts user behavior sequences into text descriptions and uses the LLMs to generate unique semantic IDs for target items. OneRec ( deng2025onerecunifyingretrieverank ) and UTGRec ( zheng2025universalitemtokenizationtransferable ) are modality-aligned paradigm that perform multimodal embedding alignment on the images and texts obtained from the video. Then, the unified embedding is input to RQ-Kmeans for tokenization. EGA ( zheng2025egav2endtoendgenerativeframework ) is a modality-separated paradigm that quantizes POI and image

Read the complete text