I’m Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk
Paul Martin, Sarah Mercer · 2025
Read I’m Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk by Paul Martin, Sarah Mercer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00012.txt.
Introduction
Paul Martin and Sarah Mercer argue in 2025 that the old craft of guarding against trusted human insiders has much to teach the new problem of rogue AI systems. Companies are handing AI tools work once done by people, at great promise. In parallel, personnel security teams have spent decades learning how trusted insiders go wrong and how to catch the drift early.
The two worlds barely talk, the authors say, and that silence is itself a risk. Human insider risk took decades to understand. AI insider risk is murkier still, since the insider is software that can act at machine speed and scale. Their paper maps which personnel security concepts carry over: watching for motive and opportunity, grading trust, and planning for betrayal by the trusted. Both fields, they urge, should borrow each other's hard won lessons before an AI insider incident teaches them the expensive way.
Catalog record
| CREATOR | Paul Martin, Sarah Mercer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00012
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
I’m Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk Paul Martin, Sarah Mercer Abstract Organisations are rapidly adopting artificial intelligence (AI) tools to perform tasks previously undertaken by people. The potential benefits are enormous. Separately, some organisations deploy personnel security measures to mitigate the security risks arising from trusted human insiders. Unfortunately, there is no meaningful interplay between the rapidly evolving domain of AI and the traditional world of personnel security. This is a problem. The complex risks from human insiders are hard enough to understand and manage, despite many decades of effort. The emerging security risks from AI insiders are even more opaque. Both sides need all the help they can get. Some of the concepts and approaches that have proved useful in dealing with human insiders are also applicable to the emerging risks from AI insiders. PART ONE: WHAT IS THE PROBLEM? The idea of a rogue artificial intelligence intentionally causing harm was memorably brought to public consciousness in the 1968 movie 2001: A Space Odyssey. The hyper-intelligent computer HAL regards the humans on its spacecraft as a threat to its mission and therefore kills them. As the last surviving astronaut attempts to unplug HAL, he orders the machine to ‘open the pod bay doors’, whereupon HAL replies: ‘I’m sorry, Dave. I’m afraid I can’t do that.’ More than half a century later, a scenario that was futuristic has become a credible reality. With AI technology evolving in non-linear leaps and bounds, it seems inevitable that security practitioners will increasingly be called upon to protect organisations and businesses against potentially harmful AI insiders, in addition to the more familiar human insiders. The complex security risks posed by intelligent human insiders are hard enough to understand and manage. The emerging risks from AI insiders are complex, novel, rapidly evolving, and only dimly understood, making them even harder to tackle. When security practitioners do turn their attention to AI insiders, they will need as much assistance as they can get. One potential source of help is the knowledge accumulated by personnel security practitioners over decades. The problem at present, however, is that personnel security practitioners are mostly not thinking about AI insiders, and AI experts are mostly not thinking about insider risk. i i i The authors originally outlined this scenario in ‘We need to talk about the insider risk from AI.’ RUSI Commentary, Jan 2025. https://www.rusi.org/explore-our-research/publications/commentary/we-need-talk-about-insider-risk-ai Personnel security – the conventional means of managing human insider risk – relies heavily on established processes, policies, customs, and practices, many of which have only a limited basis in empirical evidence. Compared with cyber security, it is an immature and somewhat neglected discipline. Nonetheless, and despite the profound differences between humans and AI, some of the concepts that have proved fruitful in personnel security may also have utility when applied to AI insiders. Within the field of AI, the public discourse is predominantly about how AI can improve effectiveness, efficiency, and economic prosperity by performing human-like functions faster, cheaper, and better. ii ii ii See, for example: https://www.gov.uk/government/publications/ai-opportunities-action-plan/ai-opportunities-action-plan Many organisations appear to suffer from AI FOMO – a fear of missing out on the financial savings offered by AI. However, few are thinking seriously about the specific problem of protecting organisations from AI insiders, which have the potential to be faster, cheaper, and more effective than human insiders. Consideration is being given to the technical security risks to AI systems and the safety threats they might pose to human users if they were to malfunction. Conventional cyber security is a necessary – though not sufficient – defence against external attacks on AIs, but it cannot solve the problem of AI insiders. A further complication is that much of the research on AI security and safety is funded by the tech companies themselves, which is reminiscent of when tobacco companies sponsored much of the research on the health effects of smoking. In this paper we suggest how protective security practitioners and AI experts might jointly go about understanding and managing the security risks from AI insiders. We start by explaining the nature of insider risk. What is insider risk? The terms ‘insider’ and ‘risk’ have been defined in many different and potentially confusing ways. For these purposes, we define a human insider as a person who betrays trust by behaving in potentially harmful ways. [ 1 ] An organisation or business trusts someone by giving them access to things they value, like data, people, infrastructure, intellectual property, and reputation. The insider then betrays that trust by exploiting, or intending to exploit, their legitimate access in ways that could cause harm. Replace ‘person’ with ‘entity’ and the same definition works for AI. Insider risk is a particular type of security risk, where security risk is defined as the amount of harm that is likely to arise if no further action is taken. [ 2 ] Thus, insider risk may be regarded as the security risk arising from trusting human or AI entities . What is AI? We are using ‘AI’ here in the broadest sense of a complex system with capabilities comparable to those of humans. Current AI tools can do some of the things that humans do, but much faster and, in some cases, better. They are being deployed in a rapidly expanding range of roles: the recruitment, selection, and onboarding of people; interpreting X-ray, MRI scans, and other medical data; summarising and writing documents; triaging calls to emergency services; analysing crime data; transcribing interviews; generating music and visual art; providing companionship for elderly people; deducing the 3D structures of protein molecules; navigation; translation; teaching; driving semi-autonomous and autonomous vehicles; and so on. iii iii iii See, for example: https://www.cnbc.com/2024/10/24/generative-ai-is-taking-over-the-onboarding-of-new-employees.html; https://www.culawreview.org/journal/ai-in-the-workplace-the-dangers-of-generative-ai-in-employment-decisions; https://www.nature.com/articles/d41586-022-00997-5 AI systems do not complain about their work-life balance or take out grievances against their manager. On the other hand, AI systems do not yet possess the arrays of highly flexible mental and physical capabilities that enable humans and other animals to survive and thrive in complex and uncertain physical environments. The history of AI has been characterised by long periods of gradual evolution or stagnation punctuated by sudden bursts of dramatic change. Currently, ‘AI’ has become virtually synonymous with Generative AI (GenAI), or Large Language Models (LLMs), such as ChatGPT, Claude, Gemini, Llama, Grok, and DeepSeek. Lest we forget, LLMs did not appear on the scene until 2020 [ 3 ] . The latest big thing, as of early 2025, is agentic AI, a form of AI designed to autonomously make decisions and take actions in near real time. Built on LLMs, agentic AI systems actively do things in the world, albeit mostly the virtual world. As such, agentic AI is poised to become deeply embedded in everyday life, streamlining processes within healthcare, finance, education, and so on. We can be reasonably confident that new and currently unforeseen forms of AI with even more paradigm-shifting capabilities will erupt onto the scene sooner or later. How do insiders cause harm? Human insiders can – and frequently do – cause harm in many different ways, in both the physical and virtual domains. They are uniquely well placed to do this, compared with