A Standardized Re-evaluation of Conversational Recommender Systems on the ReDial Dataset Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808573 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Recommender systems
Ivica Kostric, Krisztian Balog · 2026
Read A Standardized Re-evaluation of Conversational Recommender Systems on the ReDial Dataset Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808573 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Recommender systems by Ivica Kostric, Krisztian Balog (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13053.txt.
Introduction
Kostric and Balog retest seven conversational recommender systems on the ReDial dataset under shared rules. Shifting preprocessing and ground-truth definitions have made cross-study comparison shaky, and backbone capacity often drives gains more than architectural ideas. They find fine-grained ranking swings wildly with implementation detail, while nearly half of reported accuracy comes from repetition shortcuts absent in novelty-focused evaluation. Plain recall, they show, often inflates real conversational effectiveness.
Catalog record
| CREATOR | Ivica Kostric, Krisztian Balog |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44935 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13053
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Standardized Re-evaluation of Conversational Recommender Systems on the ReDial Dataset Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808573 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Recommender systems Ivica Kostric, Krisztian Balog Abstract . Recent years have seen a surge of research into conversational recommender systems (CRS). Among existing datasets, ReDial is the most widely used benchmark, cited in hundreds of studies. However, variations in how the dataset is preprocessed and used in experiments, particularly in the definition of ground-truth items, make it difficult to compare results across studies. These comparisons are further complicated by confounding factors such as the choice of the underlying large language model (LLM) and the use of external data sources. In this work, we revisit seven prominent CRS methods across three architectural families and evaluate them under standardized conditions. Our reproducibility study reveals a “granularity gap,” where fine-grained ranking (Recall@1) is highly sensitive to implementation details, while our replicability analysis shows that nearly 50% of reported accuracy stems from “repetition shortcuts” that are absent in novelty-focused evaluation. Furthermore, we find that performance gains are often driven more by the capacity of the LLM backbone than by specific architectural innovations. Finally, by applying user-centric utility metrics, we demonstrate that traditional recall frequently overstates a system’s actual conversational effectiveness. This work establishes a transparent, controlled baseline and promotes evaluation practices that prioritize novelty and interaction efficiency. † † cc-license: by 1. Introduction Figure 1. Recall@1 changes across evaluation settings in our reproducibility study. The first column reflects the results reported in the original papers, while the remaining columns use our standardized evaluation settings across all methods. The resulting changes in ordering highlight how sensitive method comparisons are to the evaluation setup. Conversational recommender systems (CRS) model user preferences and provide personalized item recommendations through natural language interaction. In recent years, end-to-end conversational recommender systems have become the standard approach, but they require large amounts of training data. ( 7 ) . Several datasets have been developed, including ReDial ( 9 ) , INSPIRED ( 5 ) , GoRecDial ( 8 ) , TG-ReDial ( 20 ) , DuRecDial ( 11 ) , and LLM-REDIAL ( 10 ) among others. In this work, we focus on ReDial as it is by far the most widely used and has over 600 citations according to Google Scholar at the time of writing. ReDial is a large-scale human-human conversational recommendation dataset, where two crowdworkers engage in a dialogue: one acts as a seeker, stating their preferences, while the other plays the role of a recommender providing personalized recommendations throughout the conversation. The dialogues are natural language interactions and include explicit mentions of recommended items, making the dataset a standard benchmark for conversational movie recommendation (see Figure 2 for an example dialogue). However, the use of ReDial for training and evaluation raises several issues. Studies increasingly utilize larger and more capable large language models (LLMs) as backbones. Yet, performance gains are often attributed solely to the proposed methods, while comparisons are frequently made against baselines that use smaller or pre-LLM models. Evaluation practices also vary, as ground-truth items are often constructed by extracting mentioned movies from the conversations and cross-referenced with an external movie database, resulting in differing test sets across papers. These inconsistencies make direct comparison and reproducibility difficult. Furthermore, 6 showed in their analysis that the dataset suffers from repeated recommendation issues, where the same item is recommended over multiple turns, as shown in Figure 2 . This results in an unreasonably high performance for a naive recommender that simply recommends the last items mentioned previously in the conversation. Figure 2. Example dialogue from the ReDial dataset. The dialogue is split into two evaluation instances: in Turn 1, the input is the initial seeker utterance, while in Turn 2, the input is the preceding dialogue context. Movies mentioned by the seeker are shown in blue , and recommendations are shown in green . Red highlights a recommendation that repeats an item already introduced earlier in the conversation. Since such repeated mentions are included in the ground truth, models can obtain artificially high scores by repeating previously mentioned movies rather than providing novel recommendations. More broadly, these challenges reflect concerns raised in the wider recommender systems literature about the reliability of reported progress. 4 have shown that many claimed improvements do not come from fundamentally better modeling, but from uneven experimental conditions, such as stronger tuning effort or more modern underlying architectures for proposed methods than for baselines. At the same time, evaluation protocols often differ substantially across studies, making results difficult to compare and easy to overstate. Reproducibility is further weakened by incomplete reporting of implementation details, hyperparameter choices, and dataset construction procedures. Additionally, 1 highlight two key limitations of current CRS evaluation practices: the reliance on static offline benchmarks that fail to capture interactive user behavior, and the overemphasis on recall-based accuracy metrics, which have been shown to correlate poorly with actual user satisfaction and perceived recommendation quality. To address these challenges, we systematically re-evaluate the current landscape of ReDial-based research by controlling for three critical variables: the underlying LLM backbone, the utilization of external knowledge, and the definition of ground-truth items—specifically focusing on an evaluation setting that considers only the newly mentioned items. We select seven influential CRS models for reproduction: KBRD ( 2 ) , KGSF ( 19 ) , UniCRS ( 14 ) , ECR ( 17 ) , MESE ( 15 ) , PECRS ( 12 ) , and ReFICR ( 16 ) . This selection is intentionally structured along two dimensions to capture the evolution of the field: (1) an architectural dimension, which categorizes models by how they integrate user modeling, recommendation, and response generation; and (2) a maturity dimension, which distinguishes between established, widely-cited baselines and contemporary state-of-the-art (SOTA) methods. By mapping these seven models across both architectural pipelines and temporal maturity, we can decouple performance gains derived from fundamental modeling innovations from those merely resulting from more modern backbones or differing evaluation protocols. Our findings reveal a significant “granularity gap” in the reproducibility of current CRS models: while performance trends are stable at higher recall cutoffs, results at Recall@1 are highly sensitive to implementation details, with over half of the reproduced models deviating from reported scores by more than 10%. Furthermore, our replicability analysis shows that “repetition shortcuts”—where models suggest items already mentioned in the dialogue—account for nearly 50% of reported accuracy. When these shortcuts are removed, even the strongest SOTA models suffer substantial performance drops. Our findings reveal a significant “granularity gap” in the reproducibility of current CRS models: while performance trends are stable at higher recall cutoffs, results at Recall@1 are highly sensitive to implementation details, with over half of the reproduced models deviating from reported scores by more than 10%. Furthermore, our replicability analysis shows that “repetition shortcuts”—where models suggest items already mentioned in the dialogue—account for nearly 50% of reported accuracy. When these shortcuts are removed, even the strongest SOTA models