The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System
Anton Hummel, Håkan Burden, Susanne Stenberg, Jan-Philipp Steghöfer, Niklas Kühl · 2025
Read The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System by Anton Hummel, Håkan Burden, Susanne Stenberg, Jan-Philipp Steghöfer, Niklas Kühl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20311.txt.
Introduction
Anton Hummel and co-authors link explainable AI to European law in a 2025 paper on clinical decision support. The EU AI Act makes explainability urgent for high-risk systems, yet most explainable AI work uses technical scores alone. Legal duties fall on providers and deployers, groups that research often overlooks.
The authors argue the link runs both ways between methods and rules. Rules pull research toward real deployment needs in clinics and other high-risk sites. Skilled practitioners can also shape how explanations prove that a system follows the law. Their analysis maps stakeholder needs to Act duties. End users, whose needs matter most, gain a clearer place in design.
Catalog record
| CREATOR | Anton Hummel, Håkan Burden, Susanne Stenberg, Jan-Philipp Steghöfer, Niklas Kühl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20311
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The EU AI Act, Stakeholder Needs, and Explainable AI: Aligning Regulatory Compliance in a Clinical Decision Support System Anton Hummel, Håkan Burden, Susanne Stenberg, Jan-Philipp Steghöfer, Niklas Kühl Abstract Explainable AI (XAI) is a promising solution to ensure compliance with the EU AI Act, the first multi-national regulation for AI. XAI aims to enhance transparency and human oversight of AI systems, particularly “black-box models”, which are criticized as incomprehensible. However, the discourse around the main stakeholders in the AI Act and XAI appears disconnected. While XAI prioritizes the end user’s needs as the primary goal, the AI Act focuses on the obligations of the provider and deployer of the AI system. We aim to bridge this divide and provide guidance on how these two worlds are related. By fostering an interdisciplinary discussion in a cross-functional team with XAI, AI Act, legal, and requirements engineering experts, we walk through the steps necessary to analyze an AI-based clinical decision support system to clarify the end-user needs and assess AI Act applicability. By analyzing our justified understanding using an AI system under development as a case, we show that XAI techniques can fill a gap between stakeholder needs and the requirements of the AI Act. We look at the similarities and contrasts between the legal requirements and the needs of stakeholders. In doing so, we encourage researchers and practitioners from the XAI community to reflect on their role towards the AI Act by achieving a mutual understanding of the implications of XAI and the AI Act within different disciplines. AI artificial intelligence ML machine learning CNN convolutional neural network DNN deep neural network ICU intensive care unit LSTM long short-term memory LOS length of stay NN neural network MTS multivariate time series RNN recurrent neural network XAI explainable artificial intelligence CDSS clinical decision support system 1 Introduction After long and complicated negotiations between the European Parliament, the EU Commission and the EU Member States, the AI Act entered into force in August 2024 ( European Parliament and Council 2024 ) . AI-based systems are now regulated under a comprehensive set of product safety rules for the first time. In the coming years, different parts of the EU AI Act will become applicable. Providers and deployers of AI systems need to learn how to apply the different articles of the act, while notified bodies and certification authorities will need to establish a process to assess these systems according to the regulations. Many standardization efforts are already underway, but there will be significant uncertainty about how the AI Act has to be applied and how it relates to different aspects of AI currently under research. Regulations such as the AI Act are especially important in high-risk ai ( ai ) systems to ensure that they are safe and respect fundamental rights and values ( Van Kolfschooten and Van Oirschot 2024 ) . The AI Act follows a risk-based approach: the higher the risk, the stricter the rule. Especially in high-risk systems, it describes many obligations for the provider and the deployer of ai systems. \Acp cdss, which may fall under the definition of a high-risk ai system in the EU AI Act, place the responsibility for the high-risk decision on the end-user (typically, clinicians). However, the AI Act does not directly address end-user needs and instead primarily focuses on regulating the providers and deployers of ai systems. One of the few mentions of end-users is the statement that high-risk AI systems must be designed so that humans can effectively oversee these systems. But there are no obligations in terms of technical requirements that put the user’s needs at the center of its goals ( Van Kolfschooten and Van Oirschot 2024 ) . In contrast, nowadays more xai ( xai ) researchers suggest putting end-users and their needs at the center of attention ( Langer et al. 2021 ) . xai methods provide descriptions of the outputs of opaque AI models, thereby satisfying the needs of the end-users. This is very helpful in high-risk applications such as healthcare. Previous work has started exploring the role that xai plays w.r.t. ensuring the trustworthiness of high-risk applications under the EU AI Act ( Panigutti et al. 2023 ; Kiseleva et al. 2022 ; Longo et al. 2024 ; Esmaeilzadeh 2024 ) . xai can contribute to implementing transparency and human oversight, which are key principles of the AI Act ( Panigutti et al. 2023 ) . Taking this line of thinking further, we believe that xai —if desigend correctly—could play a crucial role in fulfilling the obligations of deployers and providers of AI systems. Therefore, we ask: Research Question: How can Explainable AI bridge the gap between end-user needs and requirements of the AI Act? To answer the research question, we analyze the relationship between the end-user’s xai needs and the AI Act implications from a legal and technical perspective. We do this by using a concrete, ai -based cdss ( cdss ) under development. This real-world application is currently under development in collaboration with clinicians and academic partners. The cdss helps icu ( icu ) coordinators who make choices about which patients can be moved out of the icu with predicted key indicators such as the remaining los ( los ). It is important to note that clinicians can use the predictions to make a decision in our cdss — the system never decides on its own. To overcome acceptance and trust issues, we use xai methods to provide human-understandable explanations of the system. Research on the AI Act and research on xai are often conducted independently, lacking interdisciplinary collaboration ( Gyevnar et al. 2023 ) . By providing a joint understanding of the role of xai in the ai Act from an interdisciplinary perspective representing various stakeholder needs, we… 1. provide an analysis of the implications of the AI Act in relation to adopting AI in healthcare (see Section 4 ) 2. describe how the needs of the cdss stakeholders can be captured through XAI (see Section 5 ) 3. assess the similarities and contrasts between the legal requirements of the AI Act and the stakeholder desiderata of XAI (see Section 6 ). With this, our work is the first that analyzes the role of xai in the ai Act in a cross-functional team, on a real-world cdss that takes user-centered needs into account and contributes to the state-of-art by mapping obligations of the EU AI Act with the obligations of stakeholders and how to fulfill them using xai . With this, we encourage researchers and practitioners from the xai community to reflect on their role towards the ai Act by bridging the gap towards end-users. 2 Related Work and Background In this section, we give an overview of the AI Act and xai . We also discuss relevant related work. A number of studies already addressed the question of how ai can be applied in healthcare, using ethical and legal considerations ( Aboy et al. 2024 ; Porsdam Mann et al. 2024 ; Karimian et al. 2022 ; Kiseleva et al. 2022 ) . The function of xai in fostering transparency and human oversight has also been the subject of considerable debate ( Panigutti et