Facet-Aware Multi-Head Mixture-of-Experts Model with Text-Enhanced Pre-training for Sequential Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Inf
Mingrui Liu, Sixiao Zhang, Cheng Long · 2026
Read Facet-Aware Multi-Head Mixture-of-Experts Model with Text-Enhanced Pre-training for Sequential Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Inf by Mingrui Liu, Sixiao Zhang, Cheng Long (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12301.txt.
Introduction
Mingrui Liu, Sixiao Zhang, and Cheng Long improve sequential recommendation, which predicts a user's next pick from past acts. Most systems give each item one embedding vector. Models then merge those vectors into a sequence trait for user intent.
One vector per item can hide its many facets. Their facet-aware multi-head mixture model gives items a richer form. Text-enhanced pre-training adds item sense from words. Richer items could yield picks that fit user taste with more nuance. Facet traits may also help cold start items a lot. One trait per item is the weak point the new model aims to fix.
Catalog record
| CREATOR | Mingrui Liu, Sixiao Zhang, Cheng Long |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Facet-Aware Multi-Head Mixture-of-Experts Model with Text-Enhanced Pre-training for Sequential Recommendation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems Mingrui Liu, Sixiao Zhang, Cheng Long Abstract . Sequential recommendation (SR) systems excel at capturing users’ dynamic preferences by leveraging their interaction histories. Most existing SR systems assign a single embedding vector to each item to represent its features, adopting various models to combine these embeddings into a sequence representation that captures user intent. However, we argue that this representation alone is insufficient to capture an item’s multi-faceted nature (e.g., movie genres, starring actors). Furthermore, users often exhibit complex and varied preferences within these facets (e.g., liking both action and musical films within the genre facet), which are challenging to fully represent with static identifiers. To address these issues, we propose a novel architecture titled F acet- A ware M ulti-Head Mixture-of- E xperts Model for Sequential Recommendation ( FAME ). We leverage sub-embeddings from each head in the final multi-head attention layer to predict the next item separately, effectively capturing distinct item facets. A gating mechanism then integrates these predictions by dynamically determining their importance. Additionally, we introduce a Mixture-of-Experts (MoE) network within each attention head to disentangle varied user preferences within each facet, utilizing a learnable router network to aggregate expert outputs based on context. Complementing this architecture, we design a Text-Enhanced Facet-Aware Pre-training module to overcome the limitations of randomly initialized embeddings. By utilizing a pre-trained text encoder and employing an alternating supervised contrastive learning objective, we explicitly disentangle facet-specific features from textual metadata (e.g., descriptions) before sequential training begins. This ensures that the item embeddings are semantically robust and aligned with the downstream multi-facet framework. Extensive experiments on four public datasets demonstrate that our proposed method significantly outperforms existing baselines, validating the effectiveness of both the FAME architecture and the text-enhanced pre-training strategy. 1. Introduction The exponential growth of online information presents users with a vast and ever-expanding sea of items, ranging from e-commerce products ( Geng et al. 2022 ) and digital apps ( Cheng et al. 2016 ) to streaming videos ( Covington et al. 2016 ; Zhao et al. 2019 ) . With limited time to explore this abundance, recommender systems (RS) have become indispensable tools for facilitating efficient and satisfying decision-making. However, user interests are inherently dynamic, evolving continuously over time, which challenges platforms to deliver consistently relevant recommendations ( Wang et al. 2021 ) . To address this, sequential recommendation (SR) has emerged as a predominant paradigm. By leveraging the chronological sequence of user interactions, SR models capture the evolution of user preferences to predict future actions ( Fang et al. 2020 ; Wang et al. 2019 ) . Figure 1. A motivating example illustrating multi-faceted user intent. A motivation example Traditional SR frameworks typically assign a single static embedding vector to represent each item. Architectures such as Recurrent Neural Networks (RNNs) ( Hidasi et al. 2015a ; Hidasi et al. 2015b ) , attention-based models ( Kang and McAuley 2018 ; Sun et al. 2019 ; Zhang et al. 2019 ; Song et al. 2019 ) , and graph-based models ( Wu et al. 2019 ; Chang et al. 2021 ; Yang et al. 2023 ; Guo et al. 2022 ) aggregate these item embeddings into a sequence representation to model user intent. The prediction is then usually performed by selecting the item with the highest compatibility (e.g., inner product) with this sequence vector. However, a monolithic embedding fails to adequately capture the multi-faceted nature of items (e.g., a movie possessing both Genre and Starring attributes) ( Zhang et al. 2022 ; Choi et al. 2024 ) . This limitation is critical when different facets drive user intent in different contexts. As illustrated in Figure 1 , User 1’s history indicates a strong preference for the Action genre; thus, recommending another action movie is appropriate. Conversely, User 2’s interactions span multiple genres but strictly feature the actor Hugh Jackman, suggesting that the Starring facet dominates their decision-making. These examples underscore that user interest is often driven by specific, distinguishable facets. Furthermore, in realistic scenarios, users may exhibit multiple distinct preferences within a single facet. For instance, within the Genre facet, a user might enjoy both Action and Musical movies. Failing to disentangle the dominant facet and the specific preferences within it leads to suboptimal representations and recommendations. Existing research has attempted to address intent complexity by using hierarchical windows ( Guo et al. 2022 ; Zhang et al. 2023 ) or retrieving multi-item representations ( Choi et al. 2024 ) . However, these methods largely focus on aggregating historical interactions and still neglect the inherent multi-faceted semantics of the items themselves. To address these limitations, we propose a novel architecture titled F acet- A ware M ulti-Head Mixture-of- E xperts Model for Sequential Recommendation ( FAME ). We repurpose the multi-head attention mechanism to serve as a multi-facet predictor: the sub-embeddings from each head are utilized to independently predict the next item, with each head specializing in a distinct item facet. A gating mechanism then dynamically integrates these predictions based on their contextual importance. Furthermore, to disentangle complex user preferences within each facet, we replace the standard query projection with a Mixture-of-Experts (MoE) network. A learnable router assigns weights to different experts, effectively identifying the user’s specific sub-preference (e.g., "Action" vs. "Musical") within the active facet. Complementing the FAME architecture, we introduce a Text-Enhanced Facet-Aware Pre-training module to address the initialization bottleneck. Standard ID-based embeddings suffer from the cold-start problem and lack semantic interpretability. To bridge this gap, we leverage a pre-trained text encoder (e.g., BERT ( Devlin et al. 2019 ) ) to extract rich semantic features from item metadata (e.g., product descriptions or movie plots). However, raw textual embeddings often represent semantics in an entangled manner. To explicitly align these features with our multi-facet framework, we project the initial text embeddings into H disjoint subspaces using independent experts (MLP layers), where H matches the number of attention heads in FAME. Within each subspace, we employ an alternating supervised contrastive learning objective. This explicitly pulls items sharing the same facet label (e.g., same Genre) closer while pushing others apart, thereby ensuring that the initialized embeddings are already disentangled and facet-aware before the sequential training begins. We refer to this extended framework with text-enhanced initialization as FAME+ . To summarize, this article extends our preliminary work ( Liu et al. 2025 ) by incorporating semantic initialization and conducting a more comprehensive analysis. Our main contributions are as follows: • (Presented in ( Liu et al. 2025 ) ) We propose a Multi-Head Prediction Mechanism that reinterprets attention heads as facet-specific predictors, enabling the model to capture diverse item attributes without increasing parameter complexity. • (Presented in ( Liu et al. 2025 ) ) We introduce a Mixture-of-Experts (MoE) network within the self-attention layer to disentangle multiple fine-grained user preferences within each facet, seamlessly integrating with existing transformer-based backbones. • (New in this work) We design a text-enhanced pre-training framework that utilizes supervised contrastive learning to explicitly extract and disentangle facet-specific features from