Clinicians don’t know what explanations they need: A case study on eliciting AI software explainability requirements

Tor Sporsem, Stine Rasdal Finserås, Inga Strümke · 2025

Read Clinicians don’t know what explanations they need: A case study on eliciting AI software explainability requirements by Tor Sporsem, Stine Rasdal Finserås, Inga Strümke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09592.txt.

Introduction

Tor Sporsem and co-authors reported in 2025 on an eighteen-month case study of an AI tool built to flag risk of cerebral palsy in infants. The team followed the builders and hospital doctors as they tried to design clear explanations for the tool's scores. The common belief was that doctors would want deep detail on how the tool works inside.

That belief proved wrong. Doctors trusted the tool when they could check its scores against their own judgment, and a simple graph of each score did that job well. The authors call the needs behind this evaluative requirements: features that let expert users test outputs for themselves. Such needs, they argue, are best found by building early drafts and watching real use.

Catalog record

CREATORTor Sporsem, Stine Rasdal Finserås, Inga Strümke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clinicians don’t know what explanations they need: A case study on eliciting AI software explainability requirements Tor Sporsem, Stine Rasdal Finserås, Inga Strümke Abstract This paper analyses how software developers elicit explainability requirements when creating a software application with an AI component, through a case study using AI in the medical context of predicting cerebral palsy (CP) risk in infants. Following a small software development team at a Norwegian hospital, we observe their process of simultaneously developing the AI application and discovering what explanations clinicians require from the AI predictions. Since clinicians struggled to articulate their explainability needs before interacting with the system, an iterative approach proved effective: the team started with minimal explanations and refined these based on clinicians’ responses during real patient examinations. Our preliminary findings from the first two iterations show that clinicians valued “interrogative explanations” - i.e., tools that let them explore and compare the AI predictions with their own assessments - over detailed technical explanations of the AI model’s inner workings. Based on our analysis, we suggest that successful explainability requirements emerge through iterative collaboration between developers and users rather than being fully specified upfront. To the best of our knowledge, this is the first empirical case study on eliciting explainability requirements in software engineering. 1 Introduction Hanne, a physiotherapist who specializes in examining infants for cerebral palsy (CP), has just finished examining a three month old infant. I observe from beside her, wearing medical scrubs. She notes in the patient journal that while the overall assessment is optimistic, she and two colleagues have identified some slightly concerning signals. “I will call the child back in three months for another examination as a precaution.” I turn my laptop toward her to display a CP risk prediction generated by a machine learning (ML) model, developed in a collaboration between her hospital, a startup, and a university. After analyzing a three-minute video of the infant, the model shows “Total CP risk score 1%.” “Well, it is good to see that we agree with the model, I guess.” She leans back in her chair with a troubled expression on her face. “However, the machine seems very certain, while we are slightly worried. The machine isn’t worried at all. This would be an easier pill to swallow if the machine agreed with us.” She looks out the window, deep in thoughts, before she turns to me and says: “It would be nice to have some kind of explanation about why the model is so certain. Something to compare with my own assessment.” In this short-paper, we outline the process through which a small team of developers identify the explanations needed for clinicians to trust in ML application predicting the risk of CP. Specifically, we focus on how they elicited the necessary explanations to ensure clinicians at a Norwegian hospital could trust the model. The infants included in this study are part of a screening program due to complications during pregnancy or childbirth. This screening process can continue until the age of six, during which the infants are assessed for various diagnoses, including CP. Notably, nine out of ten infants in the screening program do not have CP, making it crucial to spare parents from the nerve-wracking process of screening their child for CP as early as possible. Furthermore, early identification of CP is beneficial for the child’s further development due to the plasticity of their young brains, which allows for more effective interventions and support [ 14 ] . At present, the screening process for CP is performed by specifically trained clinicians. However, even highly skilled clinicians are typically unable to confidently determine a CP diagnose before the infant reaches seven months of age. The absence of a biomarker for CP makes early diagnosis particularly challenging. Clinicians undergo specialized training to recognize CP and accumulate many years of experience to develop this expertise. The ability to predict CP is largely based on intuition and tacit knowledge, as described by one clinician: “It is a holistic assessment; you get this gut feeling, this suspicion, from observing the child.” This rare skill is primarily found among clinicians working in Norway’s largest hospitals, which often necessitates parents traveling considerable distances to have their child examined. The scarcity of this expertise highlight the need for more accessible methods for early CP detection. Over the course of 17 years, a clinician has collected 557 videos of infants participating in the screening program as part of CP research. The recent advancements in ML technology, particularly deep learning, have enabled the collaboration between the hospital and university to develop an image recognition model using this unique dataset [ 7 ] . The model, completed in 2022, predicts CP risk from a three-minute video of the infant, typically recorded by parents using their smartphones. The model’s output is a CP risk prediction ranging from 0 to 1 (0-100%). To develop and validate the model, the dataset was divided into a training set consisting of videos from 418 infants (75% of the total) and an external validation sample (test set) comprising videos from 139 infants (25%). For a comprehensive description of the model development process and performance scores, refer to [ 7 ] . Given the CP prediction model, a small team of developers set out to create an application making the model accessible to clinicians. Early in the process, the developers expected that clinicians would likely require explanations of the model’s predictions. This expectation is based on several facts, primarily that clinicians are accountable for any erroneous diagnoses. The large variety of explanation methods available for ML models, and the lacking consensus regarding what constitutes a valid and complete explanation, led to considerable uncertainty in how to proceed. Furthermore, a constrained budget necessitated careful decision-making regarding which explanation methods to develop and implement. Considering the challenge described above, we pose the following research question: How do developers elicit explainability requirements? To address this question, we present preliminary findings from a case study that chronicles the journey of a small development team as they navigated the process of discovering and prioritizing the need for explanations from users while adhering to a limited resource budget. By examining their experiences, we aim to provide insights into the strategies and methods employed by developers when faced with the task of eliciting explainability requirements in resource-constrained settings. Through this narrative, we seek to contribute to the Software Engineering literature on developing AI systems and offer practical guidance for developers and researchers working in similar contexts. 2 Background Recent advancements in ML have accelerated the integration of AI capabilities into software, significantly transforming software engineering processes [ 12 ] . Developing AI software differs from traditional software development in three ways [ 2 ] . First, it is more challenging and complex to discover and manage the data required for ML applications. Second, it demands skills that are often not present in typical software teams. Third, AI components are harder to handle as modular units. Another significant change is that users now require explanations of how AI applications work, which compels developers to elicit requirements for these explanations to effectively meet user needs. Studies on explainability requirements first surfaced in the Software Engineering

Read the complete text