Towards a Signal Detection Based Measure for Assessing Information Quality of Explainable Recommender Systems

Yeonbin Son, Matthew L. Bolton, Senior Member, IEEE · 2025

Read Towards a Signal Detection Based Measure for Assessing Information Quality of Explainable Recommender Systems by Yeonbin Son, Matthew L. Bolton, Senior Member, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01168.txt.

Introduction

Yeonbin Son and Matthew Bolton propose an objective way to grade the explanations that recommender systems give for their suggestions. Most evaluations score the recommendations themselves, and explanation quality is usually judged by asking users for opinions about style rather than content. The authors target veracity, the truthfulness of what an explanation says, and split it into two parts.

Fidelity asks whether the explanation states accurate facts about the item. Attunement asks whether it matches the target user's actual tastes. Borrowing from signal detection theory, they score hits and errors on each part and combine them into one sensitivity value. They then test the metric on four cases built to differ in information quality. The metric separated the cases as intended, which the authors take as a first validation.

Catalog record

CREATORYeonbin Son, Matthew L. Bolton, Senior Member, IEEE
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01168

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards a Signal Detection Based Measure for Assessing Information Quality of Explainable Recommender Systems Yeonbin Son, Matthew L. Bolton, Senior Member, IEEE Abstract There is growing interest in explainable recommender systems that provide recommendations along with explanations for the reasoning behind them. When evaluating recommender systems, most studies focus on overall recommendation performance. Only a few assess the quality of the explanations. Explanation quality is often evaluated through user studies that subjectively gather users’ opinions on representative explanatory factors that shape end-user’s perspective towards the results, not about the explanation contents itself. We aim to fill this gap by developing an objective metric to evaluate Veracity : the information quality of explanations. Specifically, we decompose Veracity into two dimensions: Fidelity and Attunement . Fidelity refers to whether the explanation includes accurate information about the recommended item. Attunement evaluates whether the explanation reflects the target user’s preferences. By applying signal detection theory, we first determine decision outcomes for each dimension and then combine them to calculate a sensitivity, which serves as the final Veracity value. To assess the effectiveness of the proposed metric, we set up four cases with varying levels of information quality to validate whether our metric can accurately capture differences in quality. The results provided meaningful insights into the effectiveness of our proposed metric. I Introduction With the rapid development of recommender systems, research has been attempting to develop methods for explaining recommendation results [ 1 ] . Explainable recommender systems (XRS) not only provide recommendation results, but also explain why the results were generated [ 1 ] . The purpose is to enhance users’ trust in the system by offering transparent and scrutable results to users [ 2 ] . Such approaches typically utilize users’ behavior and the characteristics of items. The way of explaining recommendation results varies by the type of data that a recommender system used. When evaluating the performance of XRS, most studies focus on examining recommendation performance. The most representative metric for recommendation performance is accuracy, which measures the ratio of user-preferred items among the recommended item list, where a user-preferred item is one that a user either rated 4 or higher, or previously purchased by the user. Another commonly used metric is root mean squared error (RMSE) [ 3 ] , which is applied when the problem is rating prediction. This metric compares pre-rated and predicted rating values to calculate the difference. Explanatory criteria Definition Transparency Explain how the system works Efficiency Help users make decisions faster Trust Increase users’ confidence in the system Satisfaction Increase the ease of use or enjoyment Persuasiveness Convince users to try or buy Effectiveness Help users make good decisions Scrutability Allow users to tell the system it is wrong TABLE I: The Seven Explanatory Factors From [ 4 ] . XRS research evaluates explanations as well, but there are rare cases where the objective information quality of the explanations is measured. General evaluation methods are divided into offline methods, online methods, and user studies. In offline methods, the assumption is that not every recommendation case has an explanation [ 5 ] . Only few of the recommendations can explain the reason, so the quality is calculated by checking whether each recommendation has explanations or not. In online methods, the underlying assumption is that researchers can track end-user behavior after receiving recommendation results [ 6 ] . The researchers compare the user’s behavior with and without explanations to determine if their is a valid difference. For user studies, researchers interview or poll end-users to assess their satisfaction with the explanations based on seven representative explanatory factors, as shown in Table I [ 1 ] . Each explanation is optionally rated on these factors using psychometric scales such as Likert scale. Fig. 1: Illustration of our approach to assessing explanation versatility in terms of Fidelity and Attunement. First, recommendation–explanation pairs are generated for a target user. Next, Fidelity and Attunement SDT outcomes are measured based on the reality of information contained about the product (Fidelity) and the user’s feedback about feature preferences (Attunement). Veracity is calculated from the paired fidelity and attunement SDT outcomes. Sensitivity ( A^{\prime} ) and bias ( B_{D}^{\prime\prime} ) metrics can be computed for all three measured concepts: fidelity, attunment, and veracity. While user studies provide subjective assessments, these factors do not directly account for the information quality. This is a critical dimension because modern systems, including but not limited to large language models, can potentially create explanations that are not rooted in reality, are highly stochastic, or based on imperfect information. This led us to consider: what if there were an objective measure to evaluate the quality of explanations, particularly in terms of information quality? Thus, the goal for this paper was to develop a metric to assess explanation quality, specifically focusing on the quality/truthfulness of provided information content, a concept we call Veracity . Because recommender explanations make statements that concern the truthfulness of both the product being recommended (e.g., this item has a given feature) and the person the recommendation is targeted to (e.g., you like things with the feature), Veracity needs to assess the quality of both. Thus, we divide Veracity into two sub-factors. The first, Fidelity , evaluates whether the explanation contains accurate information related to the recommended item. The second, Attunement , assesses whether the explanation effectively captures the user’s actual preferences. To measure these sub-factors and combine them, we utilize signal detection theory (SDT). SDT is a method that quantifies a decision maker’s ability to detect the presence of a phenomenon. SDT’s critical feature is its ability to distinguish between the decision maker’s sensitivity (their ability to differentiate between signal and noise/uncertainty) and the criteria it uses to make decision (its response bias) [ 7 ] . This paper explores how SDT can be used to assess explanation Veracity. This is accomplished by using SDT to quantify explanation Fidelity and Attunement separately. Then, these are combined together into a proper composite SDT measure of Veracity. The overview of our concept is described in Figure 1 . In what follows, we provide background for understanding our approach to measuring explanation Veracity. This is followed by the details of our SDT-based approach. We then present an experiment to evaluate different formulations of our approach and discuss their significance. II Background II-A Explainable Recommender Systems XRS is an extension of traditional recommender systems designed to BOTH provide recommendations and explain why a particular item is recommended to the end-user. Depending on the way explanations are generated, methods are categorized into model-dependent and model-independent approaches. The model-dependent approach relies on the specific structure and internal mechanisms of the recommendation algorithm to generate explanations. It is typically applied to collaborative filtering or content-based recommendation algorithms. In contrast, the model-independent approach generates explanations independently of the underlying model, focusing on analyzing the recommendation output rather than the algorithm itself. Explanations are often produced through a post-hoc analysis of recommendation results. The examples of the representative methods are local interpretable model-agnostic explanations (LIME) [ 8 ] or shapley additive explanations (SHAP) [ 9 ] . There are different types of explanations, and the type is heavily influenced by the nature of the utilized data.

Read the complete text