AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
Yu Shang Peijie Liu Yuwei Yan Zijing Wu Leheng Sheng Yuanqing Yu, Chumeng Jiang An Zhang Fengli Xu Yu Wang Min Zhang Yong Li · 2025
Read AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems by Yu Shang Peijie Liu Yuwei Yan Zijing Wu Leheng Sheng Yuanqing Yu, Chumeng Jiang An Zhang Fengli Xu Yu Wang Min Zhang Yong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19623.txt.
Introduction
Yu Shang and colleagues benchmark language model agents as personal recommenders, in work from 2025. Agent recommenders reason and role play to craft picks that adapt to each user. No shared test ground existed to grade those agents against classic methods fairly.
An interactive text simulator stages classic, shifting taste, and cold start scenarios. Rich user and item records feed the agents inside each test scene. A modular frame lets builders swap agent parts for study and comparison. Ten classic and agent methods faced the same benchmark tasks head on. Agent systems won overall, and part level lessons guide future builds.
Catalog record
| CREATOR | Yu Shang Peijie Liu Yuwei Yan Zijing Wu Leheng Sheng Yuanqing Yu, Chumeng Jiang An Zhang Fengli Xu Yu Wang Min Zhang Yong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems Yu Shang Peijie Liu Yuwei Yan Zijing Wu Leheng Sheng Yuanqing Yu, Chumeng Jiang An Zhang Fengli Xu Yu Wang Min Zhang Yong Li Abstract The emergence of agentic recommender systems powered by Large Language Models (LLMs) represents a paradigm shift in personalized recommendations, leveraging LLMs’ advanced reasoning and role-playing capabilities to enable autonomous, adaptive decision-making. Unlike traditional recommendation approaches, agentic recommender systems can dynamically gather and interpret user-item interactions from complex environments, generating robust recommendation strategies that generalize across diverse scenarios. However, the field currently lacks standardized evaluation protocols to systematically assess these methods. To address this critical gap, we propose: (1) an interactive textual recommendation simulator incorporating rich user and item metadata and three typical evaluation scenarios (classic, evolving-interest, and cold-start recommendation tasks); (2) a unified modular framework for developing and studying agentic recommender systems; and (3) the first comprehensive benchmark comparing 10 classical and agentic recommendation methods. Our findings demonstrate the superiority of agentic systems and establish actionable design guidelines for their core components. The benchmark environment has been rigorously validated through an open challenge and remains publicly available with a continuously maintained leaderboard 2 2 2 https://tsinghua-fib-lab.github.io/AgentSocietyChallenge/pages/overview.html , fostering ongoing community engagement and reproducible research. The benchmark is available at: https://huggingface.co/datasets/SGJQovo/AgentRecBench . 1 Introduction Recommender systems have become indispensable for addressing information overload on web platforms by modeling user preferences [ 1 , 2 , 3 , 4 ] . Over the decades, the field has evolved from rule-based recommendation [ 5 , 6 ] to data-driven deep learning approaches [ 7 , 8 ] , achieving remarkable gains in accuracy and efficiency. Despite these advances, there still remains some critical challenges, including (1) the black-box nature of recommendations limits interpretability of recommendation results [ 9 , 10 ] ; (2) most methods rely predominantly on historical interaction data, failing to fully leverage rich contextual information in real-world applications [ 11 ] ; and (3) existing methods often depend on handcrafted features with fixed recommendation strategies [ 12 ] , which constrain their adaptability to different scenarios. Recent advancements in LLM-based agents present a transformative opportunity to revolutionize the paradigm of recommender systems [ 13 , 14 , 15 , 16 , 17 ] . Equipped with complex reasoning and role-playing capabilities brought by LLMs, LLM-based agents can automatically tackle complex tasks through human-like planning [ 18 , 19 ] , reflective reasoning [ 20 , 21 ] , and iterative action [ 22 , 23 ] . Compared to traditional recommender systems, agentic recommender systems offer several key advantages. Firstly, by leveraging natural language as a medium, these systems can explicitly articulate their recommendation logic, enhancing transparency and fostering user trust. Secondly, with advanced perception and reasoning abilities, agentic recommender systems can autonomously gather and integrate rich, personalized contextual information from the environment, beyond ID-based interactions to enrich user modeling. Thirdly, these systems exhibit continuous self-improvement through environmental interactions, dynamically refining their recommendation strategies by incorporating external feedback and internal memory to achieve unprecedented adaptability across various scenarios. Despite this promising paradigm shift, the field currently lacks standardized benchmarks to systematically evaluate agentic recommendation approaches, hindering our understanding of their effectiveness and practical deployment. To bridge this gap, we introduce AgentRecBench, the first comprehensive benchmark for agentic recommender systems. Our benchmark makes four key contributions to advance agentic recommender systems. First, we construct a unified textual interaction environment by processing Yelp, GoodReads, and Amazon datasets. This environment provides standardized function interfaces that enable agents to perform flexible information retrieval under various conditions, while maintaining a comprehensive multi-domain database. Second, we establish three carefully designed evaluation scenarios: classic recommendation tasks for general performance assessment, evolving-interest tasks to evaluate dynamic adaptation capabilities, and cold-start tasks to measure generalization ability. These scenarios collectively reflect the core challenges in real-world recommendation systems. Third, we propose a modular agent framework with essential cognitive components [ 24 , 25 ] for rapidly building agentic recommender systems. The framework includes four core modules: dynamic planning for task decomposition, complex reasoning for decision-making, tool utilization for environment interaction, and memory management for experience retention and utilization. Finally, through extensive evaluation of 10 existing recommendation agents and traditional methods, we establish the first comprehensive benchmark for agentic recommenders. Our analysis reveals critical insights into current capabilities and limitations, from which we derive practical design guidelines to facilitate future development in this emerging field. The benchmark’s practical utility has been demonstrated through the AgentSociety Challenge [ 13 ] , which attracted 295 competing teams worldwide and received over 1,400 submissions during the 37-day competition. Participants achieved 20.3% performance improvement in Recommendation Track during the Development Phase, with a further 15.9% gain in the Final Phase, validating the practical utility of our evaluation framework. The complete benchmark environment remains publicly available with continuously maintained leaderboards to support ongoing research. In summary, this work has the following main contributions: • We propose AgentRecBench, the first large-scale and comprehensive benchmark that systematically evaluates both emerging agentic recommender systems and traditional recommendation methods across diverse scenarios. • We provide a textual environment simulator equipped with multi-domain recommendation datasets and a standardized agent development framework, establishing a closed-loop development-evaluation pipeline. This toolkit helps facilitate rapid prototyping and systematic testing of recommendation agents. • Through in-depth analysis of 10 approaches, we distill key insights into the superior designs of current approaches. Our findings provide actionable design guidelines to inspire more powerful agentic recommender systems. 2 Definition of Agentic Recommender System We formalize an agentic recommender system as a system where an LLM-based agent interacts with an environment \mathcal{E}=(\mathcal{U},\mathcal{I},\mathcal{H}) to produce recommendations. Here, \mathcal{U} and \mathcal{I} denote the user and item feature spaces respectively, while \mathcal{H}=\{(u,i,c,\tau)\} represents user-item historical interactions with contextual metadata c (e.g., ratings, comments) and timestamp \tau . At each step t , the agent observes a state s_{t}\in\mathcal{S} (encoding user and item profile, interactions, and other environmental feedback) and selects an action a_{t}\in\mathcal{A} via policy \pi_{\theta} : a_{t}\sim\pi_{\theta}(a|s_{t}),\quad\pi_{\theta}:\mathcal{S}\rightarrow p(\mathcal{A}), (1) where p(\mathcal{A}) is a probability distribution over the action space \mathcal{A}=\mathcal{I}\cup\mathcal{A}_{seek} including item recommendations and information-seeking actions, and \theta parameterizes the agent’s internal functional modules such as reasoning and memory. The agentic recommender system exhibits three distinguishing characteristics that differentiate it from traditional approaches. First, it can adaptively collect personalized contextual information from the environment while maintaining generalized, zero-shot adaptable recommendation policies across diverse scenarios. Second, the system demonstrates self-improving capability by continuously refining its internal representations and decision strategies through accumulated experience and external feedback, enabling ongoing performance enhancement. Third, the system is promising to enhance proactive engagement by strategically initiating targeted interactions beyond passive response paradigms. 3 Textual Environment Simulator The textual experiment environment simulates the information retrieval logic and functionalities typically found in social and web platforms. It provides a standardized interaction space, enabling recommendation agents to query and receive feedback systematically. This environment addresses two primary concerns: defining a clear interaction space and ensuring controlled data accessibility. The overall framework is shown in Figure 1 . Figure 1 : The overall framework of our interactive textual environment simulator. 3.1 Data and Interaction Space A clearly defined interaction space is crucial for the reliable evaluation of recommending agents.