Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework
Rishane Dassanayake 1,∗ Mario Demetroudi 1,∗ [email protected] [email protected] James Walpole 1,∗ Lindley Lentati 1,2,∗ [email protected] [email protected] Jason R. Brown 1,3,† Edward James Young 1,4,† [email protected] [email protected] · 2025
Read Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework by Rishane Dassanayake 1,∗ Mario Demetroudi 1,∗ [email protected] [email protected] James Walpole 1,∗ Lindley Lentati 1,2,∗ [email protected] [email protected] Jason R. Brown 1,3,† Edward James Young 1,4,† [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12872.txt.
Introduction
Rishane Dassanayake and co-authors warn that a misaligned AI could manipulate the people meant to watch it, in a 2025 paper. Advanced AI systems can already persuade and deceive at human level in narrow settings. Inside an AI company, such a system could target staff to weaken oversight. The authors say this risk has drawn little study and no shared method for judging it.
They build a safety case framework with three lines of argument. The first claims the system cannot manipulate, the second that controls would stop it, and the third that it can be trusted. For each line they list the proof, tests, and setup steps a company would need. The paper offers AI firms a ready method to fold manipulation risk into safety checks before release.
Catalog record
| CREATOR | Rishane Dassanayake 1,∗ Mario Demetroudi 1,∗ [email protected] [email protected] James Walpole 1,∗ Lindley Lentati 1,2,∗ [email protected] [email protected] Jason R. Brown 1,3,† Edward James Young 1,4,† [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12872
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework Rishane Dassanayake 1,∗ Mario Demetroudi 1,∗ [email protected] [email protected] James Walpole 1,∗ Lindley Lentati 1,2,∗ [email protected] [email protected] Jason R. Brown 1,3,† Edward James Young 1,4,† [email protected] [email protected] Abstract Frontier AI systems are rapidly advancing in their capabilities to persuade, deceive, and influence human behaviour, with current models already demonstrating human-level persuasion and strategic deception in specific contexts. Humans are often the weakest link in cybersecurity systems, and a misaligned AI system deployed internally within a frontier company may seek to undermine human oversight by manipulating employees. Despite this growing threat, manipulation attacks have received little attention, and no systematic framework exists for assessing and mitigating these risks. To address this, we provide a detailed explanation of why manipulation attacks are a significant threat and could lead to catastrophic outcomes. Additionally, we present a safety case framework for manipulation risk, structured around three core lines of argument: inability, control, and trustworthiness. For each argument, we specify evidence requirements, evaluation methodologies, and implementation considerations for direct application by AI companies. This paper provides the first systematic methodology for integrating manipulation risk into AI safety governance, offering AI companies a concrete foundation to assess and mitigate these threats before deployment. Contents ∗ Joint first authors with equal contribution. Order randomised. † Joint senior authors with equal contribution. Order randomised. 1 Meridian Impact, 2 Cambridge Inference 3 Computer Laboratory, University of Cambridge, 4 Department of Engineering, University of Cambridge Executive Summary AI Manipulation Attacks as an Understudied and Growing Threat AI systems are demonstrating rapidly increasing capabilities in human persuasion and manipulation. This presents security risks for organisations developing these systems and seeking to deploy them internally. A manipulation attack occurs when an AI system uses manipulative strategies to undermine safety guardrails, thereby allowing the system to pursue potentially harmful goals that deviate from its designated purpose. While misuse risks enabled by AI manipulative abilities have received much attention within previous literature, we argue that the threat posed by misaligned AI systems executing manipulation attacks autonomously has been neglected and deserves further attention. In particular, manipulation attacks are likely to be attempted, are likely to succeed without additional countermeasures, and could enable catastrophic outcomes. Likely to be attempted. Misaligned models have strong incentives to attempt manipulation attacks. Current models already exhibit strategic deception, such as “alignment faking”, to evade safety training. Moreover, manipulation is a strategically obvious path for an AI to achieve its goals, as humans are often the weakest link in cybersecurity systems. In a stark example, Claude 4 Opus attempted to blackmail an engineer in 84% of simulated scenarios to prevent its own replacement ( Anthropic 2025 ) . Likely to succeed without additional countermeasures. Current frontier models already demonstrate human-comparable persuasiveness in specific contexts, with clear capability scaling trends across generations. AI companies, like most organisations, are vulnerable to insider threats. AI systems deployed internally place stress on existing protocols, owing to the scale and pervasiveness at which such systems may attempt attacks. Additionally, technical complexity creates knowledge asymmetries that enable deception, and intense competitive pressures could be exploited by AI systems to justify bypassing safety protocols. Could enable catastrophic outcomes. Successful manipulation could compromise human oversight mechanisms that prevent AI systems from autonomously pursuing harmful or unintended goals. Manipulated employees could, knowingly or unknowingly, facilitate model weight exfiltration; alternatively, sustained influence campaigns could corrupt organisational safety culture by reframing safety precautions as inhibiting progress or steering safety research away from promising avenues. This could enable catastrophic outcomes by allowing powerful misaligned AI to operate with reduced constraints. A Safety Case Framework for Manipulation Attacks We argue that risks stemming from manipulation attacks deserve explicit treatment within safety cases used to justify internal deployment of a model within a frontier AI company. Accordingly, we adapt the safety case framework developed in Clymer et al. 2024b to the specific threat posed by manipulation. The resultant framework gives three potential assurance arguments: Inability arguments argue that the model will be unable to launch successful manipulation attacks, even if they are disposed to do so. A model’s manipulation abilities can be evaluated using a suite of manipulation benchmarks following red-team capability elicitation. An inability argument can be constructed if models consistently fail these tests, or lack prerequisite manipulation capabilities (situational awareness, strategic planning, and Theory of Mind). Control arguments argue that, even if the model is capable of performing manipulation attacks, they will be intercepted and prevented before unacceptable outcomes are realised. We propose an approach to control for manipulation involving three strategies run in tandem. Detection protocols monitor model-employee interactions to flag potential manipulation attempts before they escalate. Disruption protocols prevent manipulation attempts by controlling the flow and presentation of information between AI systems and employees. Fortification protocols strengthen human and technical defences to withstand manipulation attacks. We detail how standard control protocol evaluations can be adapted to the manipulation attack setting, and how these control evaluations can be used to form the basis of a control argument in a safety case. Trustworthiness arguments argue that, even if an AI system could successfully launch a manipulation attack, it will not because it possesses some safety property. Current methods for evidencing trustworthiness are largely insufficient and require substantial further research. We briefly outline potential approaches. A system’s behaviour in honeypot scenarios provides some evidence, but this may be unreliable if the system is aware that it is being evaluated. Training logs showing the systematic rejection of deceptive behaviours may also increase confidence, but they cannot currently prove that this would prevent manipulative behaviour once deployed. 1 Introduction Frontier artificial intelligence (AI) systems are rapidly advancing in their capabilities to persuade, deceive, and influence human behaviour. In some domains, these systems now approach or exceed human-level persuasiveness ( Durmus et al. 2024 ; Schoenegger et al. 2025 ; Salvi et al. 2024 ; Costello et al. 2024 ) . Although the consequences of these developments have received attention in the context of misuse applications of such systems, the potential of advanced AI systems to autonomously deploy manipulative tactics in pursuit of misaligned goals remains largely unaddressed. The central thesis pursued in the present work is that safety cases for internal AI deployment should include dedicated components addressing manipulation threats . Safety cases are structured, evidence-backed arguments for why the deployment of a specific AI system in a specific context will not result in an unacceptable level of risk ( Buhl et al. 2024 ; Clymer et al. 2024b ) . We focus in particular on safety cases for internal deployment of an AI system. We expect that future frontier models will first be deployed internally within the company developing them to assist with decision making and execution of research and development tasks. If the AI system is insufficiently aligned, it may seek to undermine safety guardrails placed around it, for example, by exfiltrating its weights to an external server. During internal deployment, these AI systems will interact with company employees to receive instructions, assist with tasks, and provide input on decisions. For a misaligned AI, these interactions provide an additional attack surface through which the AI may attempt to escape company control, in addition