Identifying Explanation Needs: Towards a Catalog of User-based Indicators

Hannah Deters, Laura Reinhardt, Jakob Droste, Martin Obaidi, Kurt Schneider · 2025

Read Identifying Explanation Needs: Towards a Catalog of User-based Indicators by Hannah Deters, Laura Reinhardt, Jakob Droste, Martin Obaidi, Kurt Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16997.txt.

Introduction

Software that explains itself only when asked misses the moments users silently need help. Deters and co-authors hunt for runtime signals, in behavior, system events, and feelings, that betray a need for explanation before it is voiced.

In an online study they collected self-reported signs from users. They sorted them into a catalog: 17 indicators in user behavior, 8 in system events, and 14 in emotional or physical reactions. They also mapped how each sign relates to different kinds of explanation need. The catalog can guide designers building prototypes, help teams mine telemetry from software already in use, and trigger explanations at the right moment during a session. The authors frame it as a step toward systems that notice confusion on their own.

Catalog record

CREATORHannah Deters, Laura Reinhardt, Jakob Droste, Martin Obaidi, Kurt Schneider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying Explanation Needs: Towards a Catalog of User-based Indicators Hannah Deters, Laura Reinhardt, Jakob Droste, Martin Obaidi, Kurt Schneider Abstract In today’s digitalized world, where software systems are becoming increasingly ubiquitous and complex, the quality aspect of explainability is gaining relevance. A major challenge in achieving adequate explanations is the elicitation of individual explanation needs, as it may be subject to severe hypothetical or confirmation biases. To address these challenges, we aim to establish user-based indicators concerning user behavior or system events that can be captured at runtime to determine when a need for explanations arises. In this work, we conducted an online study to collect self-reported indicators that could indicate a need for explanation. We compiled a catalog containing 17 relevant indicators concerning user behavior, 8 indicators concerning system events and 14 indicators concerning emotional states or physical reactions. We also analyze the relationships between these indicators and different types of need for explanation. The established indicators can be used in the elicitation process through prototypes, as well as after publication to gather requirements from already deployed applications using telemetry and usage data. Moreover, these indicators can be used to trigger explanations at appropriate moments during the runtime. © 2025 IEEE. This document is a preprint. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. DOI: tbd I Introduction Explainability is a quality aspect that describes the ability of a software system to explain itself [ 1 ] . This includes explaining various system aspects such as system behavior [ 2 , 3 ] , interactions with the system [ 4 , 5 ] , or privacy [ 6 , 7 ] . One challenge in the area of explainability is the highly individual needs of users [ 8 , 9 ] . Depending on the user and the context of use, different explanations may be needed [ 1 ] . For example, a user’s prior knowledge can influence whether an explanation is needed at all [ 10 ] , and personal preferences can influence which type of presentation is preferred [ 11 ] . Additionally, the same user may require different explanations in different contexts [ 1 ] . In calm settings, users can process longer, more in-depth explanations, whereas in stressful, hectic contexts, only very brief, essential explanations should be provided. This challenge is further compounded the fact that explanations should only be shown when they are actually needed, as they can also have negative effects. For instance, explanations can be stressful for the user [ 12 , 13 ] and increase their mental load [ 14 ] . Explanations can also reduce trust in the system [ 15 ] if they reveal flaws in what users perceive as a flawless system. Therefore, the correct identification of the need for explanations is very important in order to provide only desired explanations. However, eliciting the need for explanations has proven to be challenging [ 4 ] . When users are simply asked whether they want a particular explanation, they tend to accept the offer since it appears to come at no cost. Droste et al. [ 8 ] referred to this phenomenon as the why-not mentality . In contrast, providing open-ended questions instead of concrete explanation examples avoids the why-not mentality but introduces a so-called hypothetical bias [ 16 ] , since users have to think about which explanations they might have wanted. Therefore, asking users about their need for explanations is difficult to do without introducing bias. To provide an alternative that prevents these biases, we aim to establish user-based indicators that can be recorded at runtime and that objectively indicate a need for explanation. These indicators include specific user behaviors within in the system (e.g., back-and-forth navigation) or physical reactions of the users (e.g., facial expressions). To identify these indicators, we conducted an online study with 66 participants. Participants reported their needs for explanation for three software systems they had recently used. They then reported what they did and how they felt when they encountered the uncertainty that led to a need. These self-reported indicators were then coded and categorized by two researchers. Overall, identified 17 indicators related to user behavior within the system and eight system event-based indicators. We also identified six physical reactions and eight emotions related to the need for explanation. Our work contributes to the fields of requirements engineering and software engineering in several ways. First, the established indicators can be used to identify the need for explanations before the development using prototypes. Second, they can be used to automatically detect the explanation needs in existing applications using telemetry and usage data. Furthermore, these indicators can be used to detect the need for explanations at runtime in order to trigger explanations at the appropriate time to introduce explainability into adaptive personalized systems. The remainder of the paper is structured as follows: Section II provides an overview of the background and related work. The study design is described in Section III . Section IV and Section V present the study results, which are further analyzed and discussed in Section VI . Finally, Section VII summarizes the conclusions and outlines directions for future work. II Background and Related Work II-A Explainability In the context of requirements engineering, explainability is considered a non-functional requirement and software quality aspect [ 17 , 1 , 12 , 18 ] . In general terms, explainability describes the ability of a software to be explained to its stakeholders [ 18 ] . Usually, explanations are implemented with the goal of improving the users’ understanding of [ 2 , 17 , 19 ] and experience with the software [ 12 , 4 ] . Chazette et al [ 20 , 1 ] conducted a systematic literature review to research the relationship between explainability and other software quality aspects. Their findings indicate that while software-sided explanations can lead to positive effects on system aspects such as transparency, learnability and understandability, they might also lead to negative effects on other aspects, such as performance, security and development cost. Deters et al. [ 12 ] made similar observations in the context of user experience: For one, explanations may improve aspects like user efficiency and trust, by providing the necessary information to properly navigate and understand a system. However, the same aspects might also be negatively affected by overly complicated or complex explanations, confusing users and obstructing their experience rather than supporting them. In the field of explainable artificial intelligence (XAI), the focus usually lies on reasoning decision making processes and justifying outputs, making the system more interpretable [ 2 , 3 ] . However, explanations may also be used to make other system aspects more understandable. In the context of end-user privacy, Brunotte et al. [ 6 , 7 ] demonstrated that explanations are a suitable means to make software more transparent and trustworthy. This is also apparent in concepts such as explainable security [ 21

Read the complete text