VRAgent-R1: Boosting Video Recommendation with MLLM-based Agents via Reinforcement Learning

Chen Siran, Chen Boyu, Yu Chenyun, Luo Yuxiao, Yi Ouyang, Cheng Lei, Zhuo Chengxiang, Li Zang, Wang Yali · 2025

Read VRAgent-R1: Boosting Video Recommendation with MLLM-based Agents via Reinforcement Learning by Chen Siran, Chen Boyu, Yu Chenyun, Luo Yuxiao, Yi Ouyang, Cheng Lei, Zhuo Chengxiang, Li Zang, Wang Yali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02626.txt.

Introduction

Siran Chen and co-authors present a new agent system for video picks in 2025. Large language models can act as stand-ins for users, but video poses a hard mix of image, sound, and text. Prompt-only setups often model both items and tastes poorly.

Their system uses two agents that work in a loop. One agent studies each video step by step to grasp its hidden appeal. The other agent refines the pick list with careful reasoning and learns from reward. On a large test set, the system beat prior methods. The first agent alone raised one key score by a clear margin.

Catalog record

CREATORChen Siran, Chen Boyu, Yu Chenyun, Luo Yuxiao, Yi Ouyang, Cheng Lei, Zhuo Chengxiang, Li Zang, Wang Yali
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VRAgent-R1: Boosting Video Recommendation with MLLM-based Agents via Reinforcement Learning Chen Siran, Chen Boyu, Yu Chenyun, Luo Yuxiao, Yi Ouyang, Cheng Lei, Zhuo Chengxiang, Li Zang, Wang Yali Abstract Owing to powerful natural language processing and generative capabilities, large language model (LLM) agents have emerged as a promising solution for enhancing recommendation systems via user simulation. However, in the realm of video recommendation, existing studies predominantly resort to prompt-based simulation using frozen LLMs and encounter the intricate challenge of multimodal content understanding. This frequently results in suboptimal item modeling and user preference learning, thereby ultimately constraining recommendation performance. To address these challenges, we introduce VRAgent-R1, a novel agent-based paradigm that incorporates human-like intelligence in user simulation. Specifically, VRAgent-R1 comprises two distinct agents: the Item Perception (IP) Agent and the User Simulation (US) Agent, designed for interactive user-item modeling. Firstly, the IP Agent emulates human-like progressive thinking based on MLLMs, effectively capturing hidden recommendation semantics in videos. With a more comprehensive multimodal content understanding provided by the IP Agent, the video recommendation system is equipped to provide higher-quality candidate items. Subsequently, the US Agent refines the recommended video sets based on in-depth chain-of-thought (CoT) reasoning and achieves better alignment with real user preferences through reinforcement learning. Experimental results on a large-scale video recommendation benchmark have demonstrated the effectiveness of our proposed VRAgent-R1 method, e.g., the IP Agent achieves a 6.0% improvement in NDCG@10 on the MicroLens-100k dataset, while the US Agent shows approximately 45.0% higher accuracy in user decision simulation compared to state-of-the-art baselines. 1 Introduction With the widespread adoption of online multimedia services, particularly the booming popularity of short video platforms like TikTok and Kuaishou, Multimodal Recommendation Systems (MRS) have attracted considerable attention from both academia and industry. Unlike conventional recommendation technologies [ 1 , 2 , 3 , 4 , 5 ] that rely on ID-based user/item representation learning, MRS focuses on effectively integrating data associated with items from diverse modalities, such as text, images, videos, and audio, which aims to offer more personalized recommendations by deep semantic understanding of item and user characteristics. However, in video recommendations, existing approaches predominantly construct item representations based on text and images due to challenges in computational efficiency and processing heterogeneous information across modalities, the full potential of multimodal information in video recommendation systems remains largely unexplored. Benefiting from the significant advancements in Multimodal Large Language Models (MLLMs), recent studies tend to utilize their strengths and potential to boost multimodal recommendation systems. On one hand, some approaches [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] leverage the pre-trained MLLMs to directly convert each item’s multimodal information into a single embedding. However, fine-tuning MLLMs and achieving semantic alignment for recommendation requires abundant high-quality interaction data and computational resources. On the other hand, some methods [ 14 , 15 , 16 , 17 ] directly apply LLMs or MLLMs to recommendation tasks, transforming the recommendation into a language generation problem. For instance, by analyzing user interaction history, item descriptions, or conversational context, LLMs perform in-depth inference to generate recommendation results. While effective in addressing the data sparsity issue, these methods often face challenges such as limited input length, computational inefficiency, and hallucinations, making them unsuitable for large-scale recommendations. Recently, growing attention has been paid to using LLM-based agents to enhance recommendation systems’ personalization and intelligence (e.g., user profiling, simulating interactions, improving satisfaction). However, existing methods [ 18 , 14 , 19 ] mostly use frozen LLMs, with the knowledge gap between vertical domains and LLMs/MLLMs limiting their adaptability and effectiveness. Figure 1: Qualitative examples of VRAgent-R1 for Video Recommendation. Our VRAgent-R1 stands out from previous supervised fine-tuning of MLLM, which fails to give the correct prediction due to the lack of understanding of video items and deep thinking on user status. Based on the above discussion, we propose to use LLMs/MLLMs to attain enhanced multimodal content comprehension and simulate user decision-making. Our objective is to optimize representation learning and human-centric recommendation outcomes, so as to align with real user preferences. In order to realize satisfactory MLLM-based user simulation for video recommendation, we are required to address the following challenges. 1) How to discriminately exploit the recommendation-relevant semantics hidden in video items? Most video recommendation systems heavily rely on understanding video items, which presents two major difficulties. Firstly, previous methods ignore the temporal relation within video frames and fail to identify key information among numerous contents. For example, some methods [ 20 , 21 ] directly use randomly selected, pre-processed visual features, while MLLM-MSR [ 15 ] only takes the first image for video modeling, all of which fail to effectively utilize the characteristics of videos. Secondly, previous methods process each modality independently and conduct late feature fusion, which faces the problem of modality competition [ 22 , 23 ] , even leading to inferior performance compared to using a single modality [ 24 ] . Additionally, the lack of in-depth semantic interaction between raw modalities may also cause a misunderstanding of the high-level semantics of the video. As shown in Fig. 3 (a), (b), these methods merely obtain superficial modality embeddings while failing to identify the political topics expressed in the video. 2) How to effectively simulate user behavior that mirrors human deep thinking? LLMs show great promise for user simulation in the recommendation systems [ 18 , 17 , 14 , 25 ] due to their excellent linguistic understanding capabilities. However, existing LLMs struggle with processing sequential multimodal inputs, which prevents direct modeling of multimodal user behavior sequences. Moreover, most user simulators [ 18 , 14 , 17 , 19 ] primarily rely on prompt engineering to instruct frozen LLMs to generate responses without feedback, potentially leading to discrepancies between the agent and real user behavior when the model cannot be optimized. Some methods [ 16 , 15 ] attempt to fine-tune the large model, but simple fine-tuning only yields binary ‘Yes’ or ‘No’ judgments without deep analysis of the user’s status, thus limiting the model’s generalizability as shown in Fig. 1 . To address the aforementioned challenges, we introduce VRAgent-R1, a novel agent-based paradigm for video recommendation. Unlike prior approaches built on frozen LLMs, VRAgent-R1 effectively mimics the human-like thinking for recommendation by leveraging multimodal collaborative understanding and reinforcement fine-tuning (RFT) on user simulation. Specifically, it consists of two distinct agents: the Item Perception (IP) Agent and the User Simulation (US) Agent. As illustrated in Fig. 2 , The IP Agent aims to establish comprehensive multimodal content understanding for videos to improve item modeling through multi-round, in-depth semantic interaction with the MLLM. This process enables it to progressively discover key video content and effectively extract recommendation-relevant semantics from the items. Subsequently, the semantic summarization of videos generated by the IP Agent is used to optimize the fundamental video recommendation model via feature augmentation. This not only enhances the recommendation model but also facilitates the US Agent to capture user preferences and predict the next item based on historical interactions. The US Agent focuses on deep user behavior simulation and provides proxy feedback to refine the candidate set provided by the video recommendation system. To align user simulation with real decision-making, reinforcement learning is

Read the complete text