FACE: A Fine-grained Reference Free Evaluator for Conversational Recommender Systems

Hideaki Joko, Faegheh Hasibi · 2025

Read FACE: A Fine-grained Reference Free Evaluator for Conversational Recommender Systems by Hideaki Joko, Faegheh Hasibi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00314.txt.

Introduction

Hideaki Joko and Faegheh Hasibi grade chat-based recommendation and search systems without needing a gold-standard answer, in a 2025 paper. The head of the text gives the work a slightly wider title, covering conversational information access as a whole.

Judging such chats is costly, and scoring whole dialogues hides where a system went wrong. Their method, FACE, scores small units of information and many aspects of each turn and dialogue. A search procedure picks the best grading instructions for each aspect. Scores from the small units are then combined into one result. FACE matched human judgments closely, with a system-level correlation of 0.9. It also points to the exact turns where a system fails, which single scores cannot do.

Catalog record

CREATORHideaki Joko, Faegheh Hasibi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00314

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FACE: A Fine-grained Reference Free Evaluator for Conversational Recommender Systems Hideaki Joko, Faegheh Hasibi Abstract A systematic, reliable, and low-cost evaluation of Conversational Recommender Systems (CRSs) remains an open challenge. Existing automatic CRS evaluation methods are proven insufficient for evaluating the dynamic nature of recommendation conversations. This work proposes FACE : a F ine-grained, A spect-based C onversation E valuation method that provides evaluation scores for diverse turn and dialogue level qualities of recommendation conversations. FACE is reference-free and shows strong correlation with human judgments, achieving system correlation of 0.9 and turn/dialogue-level of 0.5, outperforming state-of-the-art CRS evaluation methods by a large margin. Additionally, unlike existing LLM-based methods that provide single uninterpretable scores, FACE provides insights into the system performance and enables identifying and locating problems within conversations. Figure 1: Illustration of FACE for a turn-level aspect. Instruction optimization generates a set of diverse evaluation instructions for the given aspect (e.g., relevance) based on an initial prompt; see example instructions in Appendix E.2 . For evaluation, each conversation is decomposed into particles containing a dialogue act, a mention (text span from a system turn), and user feedback from the following turn. A response distribution is created for each instruction-particle pair and the weighted summation of the scores is computed. The final score is obtained by aggregating the scores across all instructions and particles. For turn-level aspects, aggregation is performed per turn, while for dialogue-level aspects, scores of particles across the entire dialogue are aggregated. 1 Introduction Evaluation is vital for developing powerful Conversational Recommender Systems (CRSs), where users are provided with relevant and personalized recommendations Bernard et al. 2025 ; Wang et al. 2023a ; Zhang et al. 2022 . While human evaluation is considered the gold standard, it cannot be used intensively during the development of CRSs, due to its cost- and time-intensive nature Zhang and Balog 2020 . Automatic evaluation methods fill this gap and serve as invaluable aids for early diagnosis of known problems and biases during development of CRSs Dey and Desarkar 2023 ; Dubois et al. 2024b There are a number of shortcomings in existing automatic evaluation methods that make them unreliable for CRS evaluation: (i) Reference-based metrics such as Recall, ROUGE-L, and BERTScore Zhang et al. 2020 cannot capture the dynamic and evolving user-system interactions and limit the evaluation process to assessing single conversation turns given fixed conversation histories. (ii) Recently proposed reference free LLM-based evaluation methods Liu et al. 2023 ; Zhong et al. 2022 , while showing higher correlation with humans, provide a single uninterpretable 1 1 1 In this paper, interpretability is less ambitious than what is defined in the AI field Barredo Arrieta et al. 2020 ; Perrella et al. 2024 and concerns evaluation methods that enable humans to gain insights into a system’s behavior and identify issues within conversations; i.e., interpretability of the evaluation process ( Perrella et al. 2024 ) . score for each evaluation aspect, which cannot be traced back to its contributing factors. (iii) Automatic evaluations of CRSs focus mainly on turn-level aspects (e.g., recommendation effectiveness), without providing insights into dialogue-level aspects (e.g., interest arousal and task completion), which are more indicative of why a user is (dis)satisfied Siro et al. 2022 with a conversation. In this paper, we propose FACE, a F ine-grained A spect-based C onversation E valuation method. FACE is reference-free and handles diverse conversation trajectories; see Figure 1 . It first decomposes system responses into self-contained, contextualized information fragments, termed conversation particles . They are then evaluated by an LLM using a set of optimized instructions, through beam search and a bandit algorithm. These sub-scores are then aggregated into a single score per aspect. Therefore, FACE scores are interpretable , in the sense that they can be traced back to their contributing factors, providing insights into problems within the conversation. FACE evaluates CRSs on two turn-level aspects: relevance and interestingness , and five dialogue-level aspects: understanding, task completion, conversation efficiency, interest arousal, and overall impression . To evaluate FACE, we collect 20,962 human annotations for 467 human-system conversations, covering the aforementioned evaluation aspects and nine diverse CRSs that are trained on the ReDial Li et al. 2018 and OpenDialKG Moon et al. 2019 datasets. Our experiments demonstrate that FACE outperforms state-of-the-art methods by a large margin, achieving system and turn/dialogue-level Spearman of 0.9 and 0.5, respectively, without observing any system-human conversations for instruction optimization. We further show that FACE generalizes to chatbots trained on Topical-Chat Gopalakrishnan et al. 2019 and PersonaChat Zhang et al. 2018 datasets, outperforming strong baselines. Importantly, we demonstrate how FACE scores can be interpreted by humans to diagnose issues of two competitive CRSs. Key contributions of this paper include: (i) We propose FACE, a strong CRS evaluation method that evaluates dynamic user-system interactions for diverse turn- and dialogue-level evaluation aspects. (ii) We show FACE is generalizable to other LLMs and chitchat conversations, while offering interpretable scores that helps humans to identify and locate potential system issues. (iii) We develop and release a dataset for evaluating CRS evaluation methods, which covers high-quality human annotations of human-system conversations. This provides a meta-evaluation dateset that facilitate future benchmarking of CRS evaluation methods. 2 2 2 The resources of FACE, including codes/prompts used, the interface developed, and collected annotations, will be available on the GitHub repository upon acceptance. 2 Method Our Fine-Grained Aspect-based Conversation Evaluation (FACE) approach handles the one-to-many nature of conversations and provides detailed scores for turn and dialogue-level evaluation aspect. Figure 1 illustrates the FACE method. Using beam search and bandit algorithms, FACE first optimize a set of instructions for a given evaluation aspect. During evaluation, a dialogue is decomposed into conversation particles, each containing a dialogue act (e.g., “Recommendation”), mention (e.g., “How about Inception”), and corresponding user feedback from the user response (e.g., “Inception seems interesting”). Each particle is independently evaluated with optimized instructions via an LLM, generating score distributions and resulting in turn/dialogue-level scores for a given aspect. We note, without detailed elaboration, that FACE is applicable to a broad range of evaluation aspects ( Sakai 2023 ) and conversation types (e.g., task-oriented and chit-chat dialogues). The primary focus of this paper, however, is on CRSs and seven widely recognized turn- and dialogue-level evaluation aspects, following ( Siro et al. 2022 ) ; see Section 3.1 for description of these aspects. This section describes the evaluation steps of FACE: particle generation (Sec. 2.1 ) and evaluation score computation (Sec. 2.2 ), followed by the instruction optimization process (Sec. 2.3 ). 2.1 Conversation Particle Generation FACE sets two goals: (1) enable reference-free evaluations to address state explosion in natural conversation evaluation, and (2) provide fine-grained scores at both turn- and dialogue-level to locate undesired system behavior within the conversation Sakai 2023 . To achieve these, we introduce conversation particle , a self-contained information unit decomposed from conversations. Each particle is composed of three parts: (i) Dialogue act is the system’s action associated with the particle, such as “recommendation” or “preference elicitation;” (ii) Mention denotes the particle text within the system’s response, like “How about the movie A?”; and (iii) Feedback is the user’s evaluative reply, for instance, “The movie A seems interesting.” Following Joko et al. 2024 ),

Read the complete text