Retrieval Augmented Conversational Recommendation with Reinforcement Learning
Zhenrui Yue, Honglei Zhuang, Zhen Qin, Zhankui He, Huimin Zeng, Julian McAuley, Dong Wang · 2026
Read Retrieval Augmented Conversational Recommendation with Reinforcement Learning by Zhenrui Yue, Honglei Zhuang, Zhen Qin, Zhankui He, Huimin Zeng, Julian McAuley, Dong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04457.txt.
Introduction
Yue and colleagues present RAR, a retrieval-aided chat recommender aligning search and generation to improve scores and cut errors. Large language models now serve as chat recommenders, but existing methods rely on pretrained knowledge alone. The team builds a movie corpus of over 300,000 titles with metadata. A finder first picks options from user history, and a language model refines them with conversational context. RL feedback updates the finder toward higher-ranking option sets. Tests on multiple benchmarks show RAR beats state-of-the-art baselines.
Catalog record
| CREATOR | Zhenrui Yue, Honglei Zhuang, Zhen Qin, Zhankui He, Huimin Zeng, Julian McAuley, Dong Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52737 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04457
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval Augmented Conversational Recommendation with Reinforcement Learning Zhenrui Yue, Honglei Zhuang, Zhen Qin, Zhankui He, Huimin Zeng, Julian McAuley, Dong Wang Abstract Large language models (LLMs) exhibit enhanced capabilities in language understanding and generation. By utilizing their embedded knowledge, LLMs are increasingly used as conversational recommender systems (CRS), achieving improved recommendation performance across diverse scenarios. However, existing LLM-based methods rely on pretrained knowledge without external retrieval mechanisms for novel items. Additionally, the lack of a unified corpus poses challenges for integrating retrieval augmentation into CRS. Motivated by these challenges, we present RAR, a novel two-stage r etrieval a ugmented conversational r ecommendation framework that aligns retrieval and generation to enhance both performance and factuality. To support this framework and provide a unified corpus, we construct a large-scale movie corpus, comprising over 300k movies with rich metadata, such as titles, casts and plot summaries. Leveraging this data, our primary contribution is RAR, the first framework to departs from standard two-stage CRS by dynamically bridging retrieval and generation. First, a retriever model generates candidate items based on user history; in the subsequent stage, an LLM refines the recommendations by incorporating conversational context with retrieved results. In addition, we introduce a novel reinforcement learning (RL) method that leverages LLM feedback to iteratively update the retriever. By creating a collaborative feedback loop that reinforces sampled candidate sets with higher ranking metrics, RAR effectively mitigates the misalignment between the retrieval and generation stages. Furthermore, grounding the LLM in factual metadata allows our RL-driven approach to capture subtle user intentions and generate context-aware recommendations with reduced hallucinations. We validate our approach through extensive experiments on multiple CRS benchmarks, where RAR consistently outperforms state-of-the-art baseline methods. 1 Introduction Conversational recommender systems (CRS) have emerged as a promising paradigm for providing personalized recommendations through natural language interactions ( Zhang et al. 2018 ; Li et al. 2018 ; Kang et al. 2019 ; Hayati et al. 2020 ; Gao et al. 2023 ; He et al. 2023 ) . Recent advancements in large language models (LLM) ( Achiam et al. 2023 ; Dubey et al. 2024 ; Comanici et al. 2025 ) have further enabled CRS methods to utilize their extensive knowledge and diverse generation capabilities, demonstrating improvements across different recommendation scenarios ( Feng et al. 2023 ; Yang & Chen 2024 ; Li et al. 2024 ; Hui et al. 2026 ) . For example, He et al. 2023 adopt LLMs to generate movie candidates and achieve superior performance compared to traditional CRS methods. Figure 1: Our retrieval-augmented conversational recommendation framework, where a retriever gathers candidate items and the LLM generates the response conditioned on them. Nevertheless, language models often lack the essential knowledge to recommend the most suitable items (e.g., being unaware of relevant options) ( He et al. 2023 ; Yang & Chen 2024 ; Wu et al. 2024 ) . As a solution, traditional CRS methods exploit search modules or knowledge graphs to provide additional contextual information ( Zhang et al. 2018 ; Chen et al. 2019 ; Zhou et al. 2020 ) . Similarly, LLM-based CRS methods leverage knowledge graph retrieval or training on a broad range of items to improve recommendation relevance ( Li et al. 2024 ; Jeon et al. 2025 ; Zare & Alamdari 2025 ) . A notable example is ReFICR, where LLMs are trained on extensive curated data to perform sub-tasks including indexing, retrieval and generation ( Yang & Chen 2024 ) . In Figure 1 , we illustrate this general approach of the two-stage retrieval augmented conversational recommendation. While LLMs can accommodate novel items through additional training, scaling this process to extensive items and conversations is often infeasible, particularly when managing large volumes of items and noisy conversations ( Jannach et al. 2021 ; Zhu et al. 2025 ; Surana et al. 2025 ) . On the other hand, retrieving from knowledge graph requires significant efforts in data preprocessing, modeling and graph indexing. Graph retrieval can also be more computationally intensive when indexing or traversing through large-scale graphs ( Li et al. 2024 ; Peng et al. 2024 ) . Although embedding-based retrieval is a straightforward alternative ( Reimers & Gurevych 2019 ; Wang et al. 2022a ; Chen et al. 2024 ) , its effectiveness is limited by the lack of a unified corpus that provides comprehensive attribute-level metadata (e.g., plot, cast) corpus. As such, embedding-based retrieval remains largely under-explored in conversational recommendation. Furthermore, LLM-based CRS often exhibits retrieval–generation misalignment: when the retriever returns sub-optimal candidates (e.g., low-relevance items), LLM could amplify such deficiencies and cause deteriorated accuracy on cold-start or less-popular items ( He et al. 2023 ; Kemper et al. 2024 ) . In this paper, we investigate how embedding-based retrieval augmentation can be used to improve two-stage conversational recommendation. To address the limitations of existing, small-scale movie corpora (e.g., REDIAL with \sim 7k titles), we curate a comprehensive evaluation benchmark of over 300k movies, systematically enriched with metadata (e.g., titles, casts, and plot summaries). By establishing this robust, large-scale foundation, our primary contribution is RAR, a novel LLM-based r etrieval a ugmented conversational r ecommendation framework. In the first stage, we employ a retriever model to select candidate items based on the conversational history. Subsequently, an LLM refines the recommendations by incorporating conversational context with retrieved candidates, enabling personalized and fine-grained recommendation results. To enhance retrieval-generation alignment, we further optimize the retriever by leveraging LLM feedback for reinforcement learning (RL). This feedback enables RAR to iteratively update the retriever through online, on-policy preference optimization. As a result, RAR can be adapted for any black-box LLMs and deliver high-quality, context-aware recommendations. We demonstrate the efficacy of RAR with extensive experiments, where RAR consistently outperforms baselines on multiple benchmark datasets. We summarize our contributions in the following 1 1 1 Our data and code can be accessed at https://github.com/Yueeeeeeee/RAR. : 1. We present the first large-scale study of embedding-based retrieval augmentation for conversational recommendation. To support this scale, we synthesize a highly-enriched, structured baseline of over 300k films with comprehensive metadata. 2. We propose RAR, an LLM-based CRS framework with two-stage retrieval augmentation. By utilizing LLM feedback as reward signals, we design an online, on-policy reinforcement learning framework that enhances retrieval–generation alignment. 3. We demonstrate the effectiveness our approach by utilizing our curated retrieval corpus, where the RL post-trained RAR consistently outperforms state-of-the-art baseline methods, achieving considerable improvements in recommendation performance on established benchmark datasets. 2 Related Work 2.1 Retrieval Augmented Generation Retrieval augmented generation (RAG) enhances language modeling by integrating external knowledge into context ( Lewis et al. 2020 ; Guu et al. 2020 ; Karpukhin et al. 2020 ) . Beyond naïve RAG, performance is improved by refining the retriever (e.g., via joint optimization ( Lin et al. 2024 ) ) ( Trivedi et al. 2023 ; Jiang et al. 2023 ; Shi et al. 2024 ; Sarthi et al. 2024 ) , upgrading document encoding ( Izacard & Grave 2021 ; Borgeaud et al. 2022 ; Izacard et al. 2023 ) , and selectively filtering retrieved knowledge to minimize irrelevant context ( Yoran et al. 2024 ; Yan et al. 2024 ; Yue et al. 2024b ; Jin et al. 2025 )