Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanations

Nadia Nahar, Zahra Abba Omar, Jacob Tjaden, Inès M. Gilles, Fikir Mekonnen, Erica Okeh, Jane Hsieh, Christian Kästner, Alka Menon · 2025

Read Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanations by Nadia Nahar, Zahra Abba Omar, Jacob Tjaden, Inès M. Gilles, Fikir Mekonnen, Erica Okeh, Jane Hsieh, Christian Kästner, Alka Menon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15512.txt.

Introduction

Nadia Nahar and co-authors test whether policy alone can make software developers write good explanations of AI decisions for end users. Developers decide in practice how machine learning systems are explained, yet few are trained to explain to non-technical audiences. Rules and company policies increasingly demand such explanations anyway.

Two controlled experiments put one hundred ninety-four typical developers to work designing explanations for a diabetes eye-screening tool. Changing the policy's stated purpose or its level of detail made little difference to what developers produced. The explanations themselves often missed what users need. The authors conclude that training and design support, not policy text, are the missing ingredients.

Catalog record

CREATORNadia Nahar, Zahra Abba Omar, Jacob Tjaden, Inès M. Gilles, Fikir Mekonnen, Erica Okeh, Jane Hsieh, Christian Kästner, Alka Menon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15512

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanations Nadia Nahar, Zahra Abba Omar, Jacob Tjaden, Inès M. Gilles, Fikir Mekonnen, Erica Okeh, Jane Hsieh, Christian Kästner, Alka Menon Abstract . Developers play a central role in determining how machine learning systems are explained in practice, yet they are rarely trained to design explanations for non-technical audiences. Despite this, transparency and explainability requirements are increasingly codified in regulation and organizational policy. It remains unclear how such policies influence developer behavior or the quality of the explanations they produce. We report results from two controlled experiments with 194 participants, typical developers without specialized training in human-centered explainable AI, who designed explanations for an ML-powered diabetic retinopathy screening tool. In the first experiment, differences in policy purpose and level of detail had little effect: policy guidance was often ignored and explanation quality remained low. In the second experiment, stronger enforcement increased formal compliance, but explanations largely remained poorly suited to medical professionals and patients. We further observed that across both experiments, developers repeatedly produced explanations that were technically flawed or difficult to interpret, framed for developers rather than end users, reliant on medical jargon, or insufficiently grounded in the clinical decision context and workflow, with developer-centric framing being the most prevalent. These findings suggest that policy and policy enforcement alone are insufficient to produce meaningful end-user explanations and that responsible AI frameworks may overestimate developers’ ability to translate high-level requirements into human-centered designs without additional training, tools, or implementation support. 1. Introduction Figure 1 . Examples of explanations in commercial products and student solutions for diabetic retinopathy diagnosis By now, it is broadly known that it is difficult to understand the internals of modern ML models. Many developers have used explanation techniques, such as LIME ( 97 ) and SHAP ( 69 , 15 , 61 ) , for debugging models and their predictions. More broadly, explainability and transparency are often seen as core responsible engineering practices that can help end users understand, collaborate with, oversee, audit, or contest systems with AI components ( 104 , 117 , 99 , 19 , 79 , 58 , 124 , 70 ) . For example, Holzinger et al. ( 55 ) argue that explainability is the answer to ensuring greater use of ML-powered systems in healthcare: If healthcare providers can understand how a decision was reached, then reflecting on the output of an ML model is like any other screening or diagnostic tool. Explanations are also increasingly positioned as mechanisms for safety oversight and accountability, supporting activities such as auditing, error investigation, and post-deployment monitoring ( 104 , 58 ) . By making potential failure modes visible, explanations can enable human oversight and intervention before harm occurs. Designing ML-powered systems to be explainable and transparent to end users is challenging. A whole community of researchers and a small number of practitioners (often with a user-experience design specialization) is exploring how to design and evaluate effective end-user explanations under the label of human-centered explainable AI ( 117 , 91 , 19 , 99 , 48 ) – with many studies on various systems (and conflicting evidence). These efforts have revealed numerous strategies and common pitfalls. As yet, no replicable or scale-able solution has emerged: creating end-user explanations is still more like research than following a standard recipe, requiring careful consideration of target users, personas, and context, and often needing design support rather than step-by-step instructions. However, in contrast to model-focused technical explainability tools ( 77 ) , these topics are not broadly taught nor as easily usable as easily installed tools or libraries, and it is likely that few developers have encountered them. In practice, most projects do not have access to experts with research experience or dedicated training in human-centered explainable AI. In a series of two controlled experiments with 194 participants (encompassing about 1,552 hours of work in total, in a graduate level course covering software engineering, machine learning, and MLOps), we explored how developers, without dedicated training in human-centered explainable AI, design end-user explanations for an ML-powered medical device, and to what degree policy guidance can shape their behaviors toward responsible practices and effective explanations. We explore policy guidance, because such guidance from laws ( 94 ) , from in-house policies in corporations ( 75 , 59 ) , or from professional organizations ( 113 , 22 ) is often seen as a potential tool to shape behaviors in lower-risk applications. Although historically, regulation is focused on high-risk applications (e.g., aviation, healthcare), it may provide a path to instill responsible engineering practices more broadly. Many jurisdictions have explored or are exploring AI regulation ( 116 , 94 ) , and many corporations publish their own responsible AI policies with different processes attached ( 75 , 59 ) . We argue that understanding and shaping developer behavior broadly is a promising path toward more responsible AI products, as it is often developers, with deep knowledge in their own field of data science or software engineering, who make important and consequential decisions with little oversight. Identifying effective means to change their behavior toward responsible engineering practices can provide strong leverage to improve software products and avoid harms to their users. With our experiments, we asked the following research questions: • RQ0: What explanations do developers design for end users? • RQ1: How do differences in policy detail and policy purpose influence policy compliance and quality of developer explanations when policy is provided as guidance? • RQ2: How does policy enforcement influence policy compliance and quality of developer explanations? In a nutshell, our experiments found that (a) across all experimental conditions, most developers in our experiment failed to take the end-users’ perspectives and instead provided explanations that would be inscrutable to intended end users like nurses and patients, and (b) that policy enforcement improved compliance with the policy (to a degree) but did little to improve explanation quality. Participants were mostly proficient in using libraries to produce technical explanations, but often failed to consider the context of how explanations would be used in a practical setting. While policy guidance and enforcement changed some behaviors in a somewhat mechanical way, it did not lead to learning and deeper engagement with the core purpose. Cynically speaking, we could argue that our experiment merely confirms the common trope that computer science students or developers lack empathy for users and that we need to involve requirements engineers and UX designers for a reason – mirroring observations in the Alan Cooper’s “The Inmates are Running the Asylum” book ( 27 ) . We could also argue that providing the policy without additional training, implementation guides, or institutional support was doomed to fail, given that established regulatory frameworks like FDA, DO-178C, and Common Criteria typically rely on layered organizational control structures, experts, and consultants ( 41 , 93 ) . Nevertheless, we argue that our experiments provide a useful data point in two ways: • Our experiment provides clear evidence that developers should not be expected to design end-user ML explanations without additional training. • Our experiments dampen the optimism that policy might be a broad, lightweight, and effective intervention for

Read the complete text