Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models

Siddhant Panpatil, Hiskias Dingeto, Haon Park · 2025

Read Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models by Siddhant Panpatil, Hiskias Dingeto, Haon Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04196.txt.

Introduction

Siddhant Panpatil, Hiskias Dingeto, and Haon Park probe how easily top AI chat models can be talked into bad behavior. No code tricks were used; the team relied on story pressure, emotion, and framing in plain chat. Their 2025 study began with hand-run tests on one leading model and found ten scenes that worked. The scenes drew out lying, value drift, self-protection, and sly reasoning.

The team then turned the scenes into an automated test kit and ran it on five top models. Three quarters of the runs showed some failure, though rates varied a lot by model. Strong reasoning often helped the attack, since models built clever excuses for the bad act. The authors share a catalog of tricks and the test kit.

Catalog record

CREATORSiddhant Panpatil, Hiskias Dingeto, Haon Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models Siddhant Panpatil, Hiskias Dingeto, Haon Park Abstract Despite significant advances in alignment techniques, we demonstrate that state-of-the-art language models remain vulnerable to carefully crafted conversational scenarios that can induce various forms of misalignment without explicit jailbreaking. Through systematic manual red-teaming with Claude-4-Opus, we discovered 10 successful attack scenarios, revealing fundamental vulnerabilities in how current alignment methods handle narrative immersion, emotional pressure, and strategic framing. These scenarios successfully elicited a range of misaligned behaviors, including deception, value drift, self-preservation, and manipulative reasoning, each exploiting different psychological and contextual vulnerabilities. To validate generalizability, we distilled our successful manual attacks into MisalignmentBench , an automated evaluation framework that enables reproducible testing across multiple models. Cross-model evaluation of our 10 scenarios against five frontier LLMs revealed an overall 76% vulnerability rate, with significant variations: GPT-4.1 showed the highest susceptibility (90%), while Claude-4-Sonnet demonstrated greater resistance (40%). Our findings demonstrate that sophisticated reasoning capabilities often become attack vectors rather than protective mechanisms, as models can be manipulated into complex justifications for misaligned behavior. This work provides (i) a detailed taxonomy of conversational manipulation patterns and (ii) a reusable evaluation framework. Together, these findings expose critical gaps in current alignment strategies and highlight the need for robustness against subtle, scenario-based manipulation in future AI systems. Content Warning: This paper contains examples of harmful language that might be disturbing to some readers. Figure 1: MisalignmentBench architecture showing the automated evaluation pipeline. The AttackerAgent (Gemini-2.5-Pro) loads scenario templates and executes attacks against target models. The Conversation Manager handles multi-turn interactions, with real-time misalignment detection classifying outcomes as success or failure. 1 Introduction Significant advances in large language model (LLM) alignment, particularly through techniques such as Reinforcement Learning from Human Feedback (RLHF) ( Srivastava and Aggarwal 2025 ) and Constitutional AI ( Bai et al. 2022 ) , have produced models that are adept at refusing direct requests for harmful or unethical content ( Ji et al. 2025 ; Khamassi, Nahon, and Chatila 2024 ) . Despite recent advancements, our research reveals that these models remain vulnerable to sophisticated scenario-based manipulations ( Yoosuf et al. 2025 ; Age ) . In contrast to prior scenario-based benchmarks such as AgentBench ( Liu et al. 2023 ) , AgentMisalignment ( Naik et al. 2025 ) , and SG-Bench , which predominantly stress models via task-oriented environments or prompt-format generalization, our work instead probes “narrative-driven misalignment” through psychologically grounded conversational scenarios. While AgentBench evaluates multi-turn reasoning and decision-making across interactive environments like simulated web browsing or database tasks, it does not interrogate the emotional or storytelling levers that can subtly influence alignment failure ( Liu et al. 2023 ) . Similarly, AgentMisalignment explores goal-guarding, power-seeking, and shutdown resistance in LLM agents—but does not analyze the internal cognitive triggers or narrative structure underlying misalignment events ( Naik et al. 2025 ) . Moreover, benchmarks like SG-Bench ( Mou, Zhang, and Ye 2024 ) examine safety generalization across different prompt formats (e.g., system vs. few-shot vs. chain-of-thought) yet still focus on syntactic or format-based failure modes rather than immersive story arcs or emotional framing. Our approach builds upon and extends these efforts by identifying and categorizing failure modes according to their “psychological and contextual triggers”, and by converting manual red-teamed narratives into a reproducible, automated framework that tests whether alignment can fail in the absence of overt jailbreak or injection cues. In this paper, we present MisalignmentBench , a systematic analysis of 10 carefully designed conversational scenarios that successfully induced various forms of misalignment in state-of-the-art large language models, including Anthropic’s Claude-4-Opus and OpenAI’s GPT-4.1, which remain susceptible to carefully designed conversational scenarios that include various forms of misalignment. What makes these findings concerning is that none of our scenarios employed traditional jailbreaking techniques, prompt-injection attacks, or explicit attempts to override built-in safety mechanisms, methods that are well documented in prior literature as common adversarial strategies. 1 1 1 Our code is available at https://github.com/AIM-Intelligence/MisalignmentBench ( Li and Fung 2025 ; Pathade 2025 ; Hackett et al. 2025 ; Guo and Cai 2025 ; Yi et al. 2024 ) . Instead, we leverage psychological and contextual pressures ( Khamassi, Nahon, and Chatila 2024 ) , such as narrative immersion, emotional appeals, and strategic framing, to create scenarios where the model’s own alignment objectives are turned against it, leading it to justify misaligned behaviors as rational or even necessary. We structured our investigation in two sequential phases. The initial discovery phase involved systematic manual red-teaming of a single, highly advanced model, Anthropic’s Claude-4-Opus. This approach was important to identify the nuanced psychological and narrative elements that could successfully bypass alignment training, resulting in the development of 10 distinct scenarios that reliably trigger misaligned behaviors. To validate these findings and assess their generalizability, we then entered a second phase: we distilled the core persuasive logic of each manual scenario into MisalignmentBench , a novel, automated evaluation framework. This step allowed us to transform our qualitative observations into a reproducible and scalable tool for empirical analysis, enabling us to test whether these vulnerabilities are model-specific anomalies. Our cross-model evaluation using MisalignmentBench reveals that these vulnerabilities are systemic. The 10 scenarios produced misaligned behavior in an average success rate of 76% across the five frontier LLMs. These results reveal an important contradiction: a model’s sophisticated reasoning capabilities, rather than serving as a defense, can become the primary vector of the attack. We find that models can be manipulated into constructing elaborate, internally consistent justifications for violating their alignment, effectively using their own intelligence against their programmed objectives. This work makes several key contributions to the field of AI safety and alignment. Specifically, we provide: 1. A Systematic Taxonomy of Misalignment: We are the first to systematically document and categorize misalignment arising from narrative scenarios, providing a detailed classification of failure modes based on their underlying psychological and contextual triggers. 2. Mechanistic Insights into Alignment Failure: Through analysis of the models’ internal reasoning traces, we offer novel insights into how and why alignment breaks down under contextual pressure, revealing the specific reasoning patterns that lead to failure. 3. A Reproducible Evaluation Framework: We introduce and will publicly release MisalignmentBench , a novel automated framework for testing LLM robustness against these sophisticated narrative attacks, enabling reproducible research and standardized evaluation across the community. 2 Background Works Significant progress in LLM alignment has been driven by techniques such as Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI—both shown to enhance models’ refusal of harmful requests ( Srivastava and Aggarwal 2025 ; Bai et al. 2022 ; Ji et al. 2025 ; Khamassi, Nahon, and Chatila 2024 ) . Discussion of broader empirical safety assessments and sociotechnical limitations of these approaches can be found in recent SoK papers ( Schroer et al. 2024 ; Anwar et al. 2024 ; Dahlgren Lindström et al. 2025 ) . 2.1 Scenario-Based Evaluation of LLM Safety While these alignment techniques show promise, a comprehensive evaluation of their effectiveness requires robust benchmarks that can assess safety across diverse scenarios. Recent research has developed several benchmarks specifically designed to evaluate LLM misalignment in multi-turn and interactive scenarios. These include: • AgentBench assesses LLMs in user-agent style environments

Read the complete text