Generative Representational Learning of Foundation Models for Recommendation
Zheli Zhou, Chenxu Zhu, Jianghao Lin, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang · 2025
Read Generative Representational Learning of Foundation Models for Recommendation by Zheli Zhou, Chenxu Zhu, Jianghao Lin, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11999.txt.
Introduction
Zheli Zhou and co-authors wrote this 2025 paper on building one foundation model for recommendation that can both generate and embed. Recommender research has chased general models, but most efforts cover only generative tasks and stumble when many tasks train together: knowledge conflicts, uneven learning speeds, lopsided results.
Their framework, RecFound, starts with the first broad data set spanning generative and embedding tasks across scenarios. Three devices manage the multi-task tangle: task-wise groups of small expert adapters share knowledge without clashing, a scheduler feeds samples by convergence pace, and a merge step balances final performance across tasks. In experiments the model beat existing baselines across the task board. The design offers a template for foundation models in other mixed-task fields.
Catalog record
| CREATOR | Zheli Zhou, Chenxu Zhu, Jianghao Lin, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11999
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generative Representational Learning of Foundation Models for Recommendation Zheli Zhou, Chenxu Zhu, Jianghao Lin, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang Abstract Developing a single foundation model with the capability to excel across diverse tasks has been a long-standing objective in the field of artificial intelligence. As the wave of general-purpose foundation models sweeps across various domains, their influence has significantly extended to the field of recommendation systems. While recent efforts have explored recommendation foundation models for various generative tasks, they often overlook crucial embedding tasks and struggle with the complexities of multi-task learning, including knowledge sharing & conflict resolution, and convergence speed inconsistencies. To address these limitations, we introduce RecFound , a generative representational learning framework for recommendation foundation models. We construct the first comprehensive dataset for recommendation foundation models covering both generative and embedding tasks across diverse scenarios. Based on this dataset, we propose a novel multi-task training scheme featuring a Task-wise Mixture of Low-rank Experts (TMoLE) to handle knowledge sharing & conflict, a Step-wise Convergence-oriented Sample Scheduler (S2Sched) to address inconsistent convergence, and a Model Merge module to balance the performance across tasks. Experiments demonstrate that RecFound achieves state-of-the-art performance across various recommendation tasks, outperforming existing baselines. 1 Introduction It has been a long-standing goal to develop a single foundation model to excel across diverse tasks in the field of artificial intelligence bommasani2021opportunities . Foundation models, characterized by their extensive knowledge bases and large-scale parameters, have demonstrated remarkable generalization capabilities across various domains, including natural language processing and computer vision. These models rely on large-scale pretraining to capture rich representations, enabling them to adapt to a wide range of downstream tasks with minimal finetuning muennighoff2025grit . Recently, the influence of foundation models has extended to the field of recommendation systems, where they offer unique opportunities to enhance personalization and prediction accuracy by effectively modeling complex user-item interactions and contextual information of distinct sources and formats huang2024foundation ; lin2024rella ; lin2024clickprompt ; xi2024memocrs ; wang2024flip . Early efforts in applying foundation models to recommendation systems primarily focus on specific, relatively simple tasks such as tagging konjengbam2020unsupervised and question answering lai2019supervised . These approaches often involve adapting general-purpose language models to handle individual recommendation-related tasks, demonstrating the potential of foundation models in this domain. More recent work has aimed to generalize foundation models to handle a broader spectrum of recommendation tasks shi2024llamae ; li2024EcomGPT ; peng2024eCeLLM ; zhang2024LLaSA . While these works demonstrate promising progress in unifying different recommendation tasks under a single model, they still face the following significant challenges in effectively handling the full complexity of recommendation scenarios. First, existing approaches overlook the crucial embedding 1 1 1 In this work, the term “embedding” is interchangeable with “representational”. tasks , which are fundamental to recommendation performance. Embedding techniques transform high-dimensional, discrete features such as user and item identifiers or descriptions into low-dimensional, continuous vectors, capturing intricate relationships and facilitating efficient computation. High-quality embeddings are essential for various components of recommendation systems, including retrieval, recall, and visualization huang2024comprehensive ; liu2023embedding ; chen2022deep . Second, current models struggle with the complexities of multi-task learning , particularly in managing the knowledge sharing & conflict and the inconsistent convergence speeds among tasks. These issues can hinder the effective training of foundation models intended to perform well across diverse recommendation tasks, limiting their practical applicability in real-world systems xi2024towards ; wu2023robust . To address these limitations, we introduce RecFound , a generative representational learning framework for recommendation foundation models. Our framework begins with the construction of the first comprehensive dataset that covers both generative and embedding tasks across various aspects of the recommendation domain. Building upon this dataset, we propose a novel multi-task training scheme for recommendation foundation models. From the model perspective, we design a Task-wise Mixture of low-rank experts (TMoLE) architecture. TMoLE consists of multiple low-rank experts and a task - oriented router. The router generates a masked routing distribution based on each sample’s task to adaptively aggregate the expert outputs. This mechanism allows for effective knowledge sharing between related tasks while mitigating potential conflict among disparate ones by enabling task-specific specialization via the expert integration. From the data perspective, we propose a Step-wise convergence-oriented sample scheduler (S2Sched) . S2Sched dynamically adjusts the sampling ratio of different tasks within each training batch based on their non-convergence rates estimated on the validation set. Specifically, it prioritizes sampling from tasks that are converging more slowly, thereby addressing the issue of inconsistent convergence speeds across different recommendation tasks within a single model. Finally, we also introduce a Model Merge module to further balance the perfromance across tasks. Our contributions can be summarized as follows: • We construct the first comprehensive dataset for recommendation foundation models, covering both 10 generative tasks and 3 embedding tasks across diverse recommendation scenarios. • We propose RecFound, a novel training framework for recommendation foundation models. We design TMoLE to facilitate effective knowledge sharing & conflict resolution between embedding and generative tasks, S2Sched to address the inconsistency of convergence speeds for different tasks, and the Model Merge module to balance performance across tasks. • Extensive experiments demonstrate that our RecFound framework achieves state-of-the-art performance on a wide range of recommendation tasks, and we provide insightful analysis to support our findings. We release our code, as well as the dataset and model checkpoints, to benefit the research community. 2 Related Works General Foundation Models in NLP Extensive research on general foundation models in natural language processing (NLP) has investigated general-purpose methods for addressing a wide range of downstream tasks. To solve various embedding tasks, early influential works focused on learning static word embeddings from large corpora mikolov2013word2vec ; pennington2014glove . Subsequent research, such as ELMo peters2018elmo , BERT devlin2019bert , and a wave of BERT-family models liu2019roberta ; reimers2019sentence generates token representations that dynamically reflect their surrounding context, yielding substantial gains in tasks like semantic matching and retrieval. Another line of work investigates foundation models designed for various generative tasks, aiming to solve multiple types of text generation tasks simultaneously. The GPT series radford2018gpt1 ; radford2019gpt2 ; brown2020gpt3 demonstrated this by revealing how decoder-only architectures can acquire multitask proficiency only through next-token prediction. More recently, a few studies have explored the possibility of a unified foundation model that can effectively handle both embedding and generative tasks. GritLM muennighoff2025grit trains a large language model to perform both embedding and generative tasks by distinguishing them through instructions. Similarly, UniGen li2023unigen integrates retrieval and question answering into a single model. Recommendation Foundation Models Early research on recommendation focused on optimizing single-task generative models hidasi2015gru4rec ; dong2017seq2seq ; wang2017irgan ; liang2018variational ; lin2025can ; xi2024decoding , such as producing personalized reviews via neural review generators dong2017neural_review_gen or generating item descriptions from user behavior sequences xi2024efficient ; yang2018behavior_driven_desc . These models are typically trained from scratch and lack generalization across tasks. In the wake of large language models, recent works introduce foundation models tailored to e-commerce scenarios shen2023large ; zhou2024large ; wang2024unirec ; liu2023fim . LLaMA - E shi2024llamae tunes LLaMA grattafiori2024llama3herdmodels to support various product authoring tasks. EcomGPT li2024EcomGPT leverages generative pretraining over large-scale e-commerce corpora to unify tasks such as product Q&A, summarization, and marketing text generation. eCeLLM peng2024eCeLLM further develops a