Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information sy
Qiushi Pan, Hao Wang, Guoyuan An, Luankang Zhang, Wei Guo, Yong Liu · 2025
Read Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information sy by Qiushi Pan, Hao Wang, Guoyuan An, Luankang Zhang, Wei Guo, Yong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25285.txt.
Introduction
Qiushi Pan led five co-authors on Fuxi-MME. Hao Wang, Guoyuan An, and Luankang Zhang helped build it. Wei Guo and Yong Liu complete the team. Their 2025 paper tackles scalable sequential recommendation. Items have many traits and facets. Which trait matters shifts with user context.
Fuxi-MME uses a multi-embedding strategy for items. It splits one embedding matrix into several smaller ones. Each smaller matrix can capture different item traits. The framework also uses mixture-of-experts layers. Different expert parts handle different kinds of inputs. That allows adaptive and specialized processing of representations. Tests used public data sets for recommendation. The authors report their framework beat several baselines. The work matters for services scaling up recommendation models.
Catalog record
| CREATOR | Qiushi Pan, Hao Wang, Guoyuan An, Luankang Zhang, Wei Guo, Yong Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems Qiushi Pan, Hao Wang, Guoyuan An, Luankang Zhang, Wei Guo, Yong Liu Abstract . In recommendation systems, how to effectively scale up recommendation models has been an essential research topic. While significant progress has been made in developing advanced and scalable architectures for sequential recommendation(SR) models, there are still challenges due to items’ multi-faceted characteristics and dynamic item relevance in the user context. To address these issues, we propose Fuxi-MME, a framework that integrates a multi-embedding strategy with a Mixture-of-Experts (MoE) architecture. Specifically, to efficiently capture diverse item characteristics in a decoupled manner, we decompose the conventional single embedding matrix into several lower-dimensional embedding matrices. Additionally, by substituting relevant parameters in the Fuxi Block with an MoE layer, our model achieves adaptive and specialized transformation of the enriched representations. Empirical results on public datasets show that our proposed framework outperforms several competitive baselines. 1. Introduction Recommendation systems Kang & McAuley 2018 ; Zhang et al. 2024 ; He et al. 2014 ; Liu et al. 2023 ; Wang et al. 2025a ; Wang et al. 2025b ; Wang et al. 2025 ; Wang et al. 2019 ; Wang et al. 2021a ; Wang et al. 2021 ; Tong et al. 2024 ; Zhang et al. 2024a ; Xu et al. 2024 aim to analyze users’ historical behaviors to identify and present items that align with their potential interests. In industrial applications, Sequential Recommendation (SR) Zhou et al. 2020 ; Zhai et al. 2024 ; Zhang et al. 2025 ; Yin et al. 2024 ; Yin et al. 2023 ; Wang et al. 2024 ; Han et al. 2024 ; Han et al. 2023 has become an indispensable component, as it focuses on predicting a user’s next action by modeling the temporal dynamics of their interaction history. This subfield has consequently attracted substantial attention from both academia and industry. In recent years, the field has embraced the autoregressive paradigm Kang & McAuley 2018 ; Deng et al. 2025 , drawing inspiration from advances in natural language processing Touvron et al. 2023 ; Yang et al. 2025 . Early implementations, such as GRU4Rec Hidasi et al. 2015 , utilized Recurrent Neural Networks (RNNs) Hochreiter & Schmidhuber 1997 ; Medsker & Jain 2001 to capture short-term user preferences but struggled with long-range patterns due to recursive state updates and gradient issues. Subsequently, SASRec Kang & McAuley 2018 marked a significant leap forward by incorporating a Transformer-based Vaswani et al. 2017 architecture that leverages self-attention to capture more complex and dynamic user preferences. Despite these advancements, scalability remains a persistent challenge—simply enlarging model size does not guarantee improved performance Shen et al. 2024a . To overcome this, contemporary research has turned to scaling laws Kaplan et al. 2020 ; Guo et al. 2024 ; Shen et al. 2025a , revealing that model performance can grow predictably with increases in parameters, data, and compute. For instance, HSTU Zhai et al. 2024 modified the attention mechanism for large and non-stationary vocabulary and proposes the M-FALCON algorithm to speed up model inference, which exhibits scaling-up effects in recommendation. Similarly, Fuxi-Alpha Ye et al. 2025 presents a novel architecture that disentangles temporal, positional, and semantic features, offering a more scalable and modular design. Together, these works demonstrate that autoregressive frameworks can successfully harness scaling laws for considerable performance enhancements in recommendation. Despite the considerable progress achieved in designing scalable architectures for SR, most existing studies emphasize model structure while overlooking the embedding layer —a component that often constitutes the majority of parameters and represents a major bottleneck in expressive capacity Jiang et al. 2021 ; Wang et al. 2025b . Consequently, the ability to capture rich item features is often limited by the constraints of a single, monolithic embedding vector. While some approaches attempt to mitigate this by increasing embedding dimensionality, such brute-force scaling yields diminishing returns Guo et al. 2023 . Drawing inspiration from Large Language Models (LLMs) Shen et al. 2024 ; Lv et al. 2025 ; Shen et al. 2025 ; Yin et al. 2024b ; Gu et al. 2025 and their capacity to represent multifaceted semantic spaces Zhang et al. 2024b ; Huang et al. 2024 , we argue for a more principled rethinking of the embedding layer itself. By rearchitecting the embedding space, we can more effectively model complex item attributes and alleviate the representational bottleneck that limits current SR systems. However, advancing this direction introduces new challenges. Research on embedding structures for sequential recommendation remains limited, and two critical challenges: (1) An item’s identity is inherently multi-faceted—encompassing attributes such as category, brand, and style—yet a single dense embedding struggles to disentangle these heterogeneous signals, constraining the model’s expressiveness. Designing a mechanism that can represent such diverse features in a decoupled and structured manner is therefore essential. (2) The relevance of item attributes varies with user context: one user may prioritize brand affinity, while another focuses on visual style. Capturing these differences requires a dynamic, input-aware mechanism that can adaptively transform and weigh features according to sequence context. To tackle these challenges, we propose Fuxi-MME , a framework that integrates a multi-embedding strategy with a Mixture-of-Experts (MoE) architecture built upon the high-performing Fuxi-Alpha model. Our approach decomposes the conventional single embedding into multiple lower-dimensional sub-embeddings, each responsible for capturing a distinct facet of an item’s identity. This design enhances expressiveness without inflating parameter count and can be seamlessly integrated as a plug-and-play component. Furthermore, to enable adaptive and specialized processing of these enriched representations, we introduce an MoE layer within the Fuxi block. This layer serves as a dynamic routing mechanism, directing sequence inputs to specialized expert networks based on their contextual characteristics. Extensive experiments show that the synergy between expressive embeddings and adaptive experts enables end-to-end learning that effectively models heterogeneous user behaviors and nuanced feature interactions. The main contributions of this paper are summarized as follows: • We present Fuxi-MME, a framework that synergizes multi-embedding and expert routing to achieve more context-aware and adaptive sequence modeling. • We propose a multi-faceted embedding strategy for sequential recommendation, designed to overcome the representational bottleneck of conventional single-vector embeddings in a parameter-efficient manner. • We develop a MoE module tailored for Transformer-based architectures, enabling dynamic, input-dependent feature transformation and enhancing interaction modeling. • Comprehensive experiments on three public benchmark datasets demonstrate that Fuxi-MME establishes new state-of-the-art results. Further studies confirm the framework’s effectiveness. 2. Related Work 2.1. Sequential Recommendation Sequential recommendation Boka et al. 2024 ; Pan et al. 2024 ; Xu et al. 2019 aims to predict a user’s next interaction based on their historical behavior. Early approaches predominantly relied on Markov chains He & McAuley 2016 ; Rendle et al. 2010 to model transition probabilities between items. With the rise of deep learning, subsequent research adopted neural architectures such as recurrent neural networks (RNNs) Hidasi & Karatzoglou 2018 ; Hidasi et al. 2015 and convolutional neural networks (CNNs) Chen et al. 2022 ; Yan et al. 2019 to better capture complex temporal dependencies. More recently,