Secure Human Oversight of AI: Exploring the Attack Surface of Human Oversight

Jonas C. Ditz, Veronika Lazar, Elmar Lichtmeß, Carola Plesch, Matthias HeckFederal Office for Information SecurityBonnGermany [email protected] · 2025

Read Secure Human Oversight of AI: Exploring the Attack Surface of Human Oversight by Jonas C. Ditz, Veronika Lazar, Elmar Lichtmeß, Carola Plesch, Matthias HeckFederal Office for Information SecurityBonnGermany [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12290.txt.

Introduction

Human overseers are meant to catch AI mistakes, yet the oversight desk itself can be attacked. Jonas C. Ditz and six co-authors argue in 2025 that oversight, now written into rules such as the European AI Act, forms a fresh attack surface inside an organization's safety setup.

Treating the oversight process as a piece of software, they apply systematic threat modeling borrowed from cybersecurity. The paper maps attack vectors against the people, screens, and approval steps that stand between a model and the world. It then catalogs hardening strategies, giving researchers and practitioners a checklist for building oversight that resists tampering, deception, and quiet bypass.

Catalog record

CREATORJonas C. Ditz, Veronika Lazar, Elmar Lichtmeß, Carola Plesch, Matthias HeckFederal Office for Information SecurityBonnGermany [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Secure Human Oversight of AI: Exploring the Attack Surface of Human Oversight Jonas C. Ditz, Veronika Lazar, Elmar Lichtmeß, Carola Plesch, Matthias HeckFederal Office for Information SecurityBonnGermany [email protected] Abstract Human oversight of AI is promoted as a safeguard against risks such as inaccurate outputs, system malfunctions, or violations of fundamental rights, and is mandated in regulation like the European AI Act. Yet debates on human oversight have largely focused on its effectiveness, while overlooking a critical dimension: the security of human oversight. We argue that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations. Drawing on cybersecurity perspectives, we analyze attack vectors that threaten the requirements of effective human oversight, thereby undermining the safety of AI operations. Such attacks may target the AI system, its communication with oversight personnel, or the personnel themselves. We then outline hardening strategies to mitigate these risks. Our contributions are: (1) introducing a security perspective on human oversight, and (2) providing an overview of attack vectors and hardening strategies to enable secure human oversight of AI. K eywords Human oversight \cdot cybersecurity \cdot attack vectors \cdot hardening strategies \cdot AI governance 1 Introduction Figure 1: The human oversight attack surface encompasses the AI system’s technical infrastructure, the human oversight personnel, and the communication channels between the two sides. Attacks on human oversight aim to undermine oversight personnel’s epistemic access, causal power, self-control, and fitting intentions, the four requirements of effective human oversight [ 1 ] . Malicious actors can utilize different attack vectors to undermine one or several of these requirements (see Table 1 ). In order to protect human oversight from malicious actors, various hardening strategies need to be considered and deployed to counteract the threats posed by the attack vectors (see Table 2 ). A well-designed combination of hardening strategies can shield against different attack vectors and, thus assure secure human oversight. Human oversight of artificial intelligence (AI) is often considered a strategy to address risks associated with the use of AI in high-risk contexts [ 2 , 3 , 4 ] . Human oversight personnel are expected to detect inaccurate outputs, outputs that may pose risks to fundamental human rights (e.g., discriminatory outputs), unsafe states of AI systems, and system malfunctions, and to intervene in order to mitigate or minimize these risks [ 5 ] . Human oversight is also a key concept in AI governance and ethics [ 3 , 2 ] and a central requirement for the implementation of high-risk AI systems in emerging AI regulation, such as Article 14 of the European AI Act [ 6 , 3 ] . However, the idea of relying on humans to oversee AI in order to mitigate operational risks also faces criticism and is expected to encounter substantial challenges in practice [ 7 , 2 , 1 ] . The most prominent criticisms and challenges include doubts about whether humans can effectively oversee AI systems [ 2 ] , the difficulty of defining what constitutes effective human oversight [ 1 ] , clarifying who should provide oversight and when it should occur [ 6 ] , specifying which risks and unwanted system behaviors oversight personnel should detect [ 5 ] , and determining how to design and evaluate effective human oversight [ 1 ] . However, one key challenge that remains underexplored is the security risks that arise from human oversight of AI. Because human oversight will be implemented as part of the safety, security, and accountability architecture of AI systems [ 3 , 7 ] , malicious actors who successfully attack elements related to oversight can significantly undermine the safety of AI operations. In this sense, the human oversight architecture creates a new attack surface for targeting AI operations in high-risk contexts such as critical infrastructure, public administration, or medicine. A security-focused perspective on human oversight is crucial, given the central role oversight already plays, the expectations placed on it in high-risk contexts, and its growing importance as a regulatory requirement in AI governance, such as under the EU AI Act. If we fail to assess and address the security challenges of human oversight, we risk overlooking potential attack vectors in high-risk contexts; and without awareness of these vectors, we will not be able to design effective hardening strategies. In this paper, we introduce a security-related perspective on human oversight of AI. Specifically, drawing on cybersecurity research, we outline possible attack vectors that threaten the key requirements of effective human oversight (i.e., epistemic access, self-control, causal power, and fitting intentions; [ 1 ] ). These attacks may target the AI system itself, the communication between the AI system and oversight personnel, or the personnel directly (see Figure 1 ). We also draw on the cybersecurity literature to discuss hardening strategies that can mitigate the risks posed by such attacks. This paper makes two key contributions. First, we introduce a security-related perspective on human oversight of AI. This perspective is critically missing from current research, even though human oversight is becoming an integral component of the safety, security, and accountability architecture of AI operations, and thus an increasingly viable target for malicious actors. Second, we provide an overview of possible attack vectors and hardening strategies contextualized for human oversight of AI. We hope this contribution supports AI governance and system design in ensuring not only effective but also secure human oversight of AI. 2 Related Work 2.1 Effective Human Oversight of AI Research discusses the nature of human oversight of AI, including the what , when , and who of oversight. Regarding the what , many forms of human-AI collaboration in principle involve some degree of human oversight. Across a wide range of human-AI interaction use cases, humans monitor AI system outputs, check their accuracy, request explanations, adapt outputs, intervene when systems malfunction, or take over manual control in the event of failures [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . In a more targeted sense, and following how emerging research has tentatively defined the concept [ 1 ] , human oversight specifically refers to humans mitigating risks in AI operations [ 16 , 6 ] . More concretely, oversight personnel are tasked with monitoring AI systems, detecting failures and malfunctions, identifying unsafe states, flagging inaccurate outputs, and recognizing outputs that may pose risks to fundamental human rights, such as discriminatory outputs [ 1 , 5 , 6 ] . Once such issues are detected, oversight personnel are expected to act in ways that effectively mitigate emerging risks [ 5 , 1 ] . This can involve taking over manual control, overwriting inaccurate or inadequate outputs [ 17 ] , or stopping AI operations and triggering intervention by other human stakeholders, such as developers who can adapt the system. Importantly, the AI system under oversight may be operated by another human who is not considered oversight personnel but a user of the system. For example, a patient using an AI chatbot for mental health support [ 18 ] would be the user, while oversight personnel monitor the patient–AI interaction to detect risky situations (e.g., if the patient discusses suicidal behavior with the chatbot). Regarding

Read the complete text