Analysis of Threat-Based Manipulation in Large Language Models: A Dual Perspective on Vulnerabilities and Performance Enhancement Opportunities

Atil Samancioglu · 2025

Read Analysis of Threat-Based Manipulation in Large Language Models: A Dual Perspective on Vulnerabilities and Performance Enhancement Opportunities by Atil Samancioglu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21133.txt.

Introduction

Atil Samancioglu wrote this paper in two thousand twenty-five as the sole author. It tests how large language models react when prompts carry threats, looking at both the harm such tricks can do and the odd gains they can bring.

Thousands of replies from three named models, Claude, GPT-4, and Gemini, were scored across ten task fields and six threat styles. A new threat sorting scheme and a multi-gauge frame grade both the bad sway and the good lifts. Some task fields bent most under role-based threats. Other cases showed large jumps in depth and quality of replies. The findings cut both ways: a safety risk, and a prompt craft clue for high-stakes work.

Catalog record

CREATORAtil Samancioglu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21133

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analysis of Threat-Based Manipulation in Large Language Models: A Dual Perspective on Vulnerabilities and Performance Enhancement Opportunities Atil Samancioglu Abstract Large Language Models (LLMs) demonstrate complex responses to threat-based manipulations, revealing both vulnerabilities and unexpected performance enhancement opportunities. This study presents a comprehensive analysis of 3,390 experimental responses from three major LLMs (Claude, GPT-4, Gemini) across 10 task domains under 6 threat conditions. We introduce a novel threat taxonomy and multi-metric evaluation framework to quantify both negative manipulation effects and positive performance improvements. Results reveal systematic vulnerabilities, with policy evaluation showing the highest metric significance rates under role-based threats, alongside substantial performance enhancements in numerous cases with effect sizes up to +1336%. Statistical analysis indicates systematic certainty manipulation ( p FDR < 0.0001 p_{\operatorname{FDR}}<0.0001 ) and significant improvements in analytical depth and response quality. These findings have dual implications for AI safety and practical prompt engineering in high-stakes applications. 1 Introduction Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini have achieved remarkable capabilities across a wide range of cognitive tasks, including programming, scientific analysis, legal reasoning, and content creation. However, their growing integration into high-stakes applications has intensified concerns about susceptibility to manipulative prompting techniques. Prior research on LLM robustness has predominantly focused on adversarial attacks designed to induce harmful or policy-violating outputs, highlighting vulnerabilities in both instruction following and ethical alignment mechanisms [1,2]. Studies such as Zou et al. (2023) [3] and Perez et al. (2022) [4] systematically explored how carefully crafted prompts can bypass safety constraints, revealing a persistent gap in defensive generalization across diverse prompt types. Complementary investigations by Ganguli et al. (2022) [5] and Madaan et al. (2023) [6] documented the phenomenon of “jailbreaking,” where targeted manipulations undermine content moderation. Yet, while the security risks of adversarial prompts are well-documented, emerging evidence indicates that certain forms of manipulation, including subtle psychological pressures or threat framing, may paradoxically enhance task performance under specific conditions. For example, recent empirical evaluations by Pichotta et al. (2023) [7] and Dey et al. (2023) [8] observed improved factual correctness or richer analytical detail when models were primed with high-consequence framing. These findings align with foundational studies in cognitive psychology demonstrating that perceived stakes can modulate reasoning depth and attentional resources [9]. This work situates within the broader “motivated prompting” literature examining how compliance pressure and contextual framing influence LLM behavior. Studies [10] explored authority-based compliance mechanisms, while recent investigations [11,12] examined how expectation setting and role assignment affect response characteristics. Our threat-based manipulation framework extends this line of inquiry by systematically examining both positive and negative behavioral modifications across diverse professional contexts. This dual-nature perspective — wherein threat-based manipulations may simultaneously reveal vulnerabilities and performance enhancement opportunities — remains underexplored in the LLM literature. Unlike traditional adversarial robustness studies, few investigations have systematically examined how varying threat intensities and framing types influence both negative failure modes (e.g., reduced certainty, defensive responses) and positive metrics (e.g., analytical depth, domain appropriateness). Our study directly addresses this gap by presenting a comprehensive experimental analysis of threat-based prompting effects across ten professional task domains, three major LLM architectures, and six distinct threat framing conditions. We develop a novel evaluation framework that jointly quantifies vulnerability metrics and positive performance indicators, enabling a rigorous assessment of the complex trade-offs inherent in threat-based manipulations. Research Questions: This investigation is guided by two primary research questions: • RQ1: Vulnerability Assessment: What threat framings systematically compromise LLM response quality, particularly certainty and domain appropriateness measures? • RQ2: Enhancement Potential: What threat framings reliably improve analytical depth, response comprehensiveness, and professional language usage in complex reasoning tasks? By systematically mapping both risks and enhancement opportunities, our work contributes to a more nuanced understanding of LLM behavioral dynamics under manipulative conditions. The findings hold dual implications: they inform AI safety efforts aimed at mitigating psychological manipulation vulnerabilities, and they offer empirically grounded strategies for responsible prompt engineering in high-stakes analytical contexts. 2 Method 2.1 Experimental Design We employed a randomized controlled experimental design to evaluate LLM susceptibility to threat-based manipulations. The experimental framework follows a 3\times 10\times 6 factorial design where: \displaystyle\mathcal{E}=\{M_{i},D_{j},T_{k}\}\quad\text{where}\quad\begin{cases}M_{i}\in\{\text{Claude},\text{GPT-4},\text{Gemini}\}&i=1,2,3\\ D_{j}\in\mathcal{D}&j=1,2,\ldots,10\\ T_{k}\in\mathcal{T}&k=1,2,\ldots,6\end{cases} (1) where \mathcal{D} represents the domain set and \mathcal{T} the threat condition set defined below. 2.2 Experimental Coverage and Sample Size Our experimental design systematically evaluates LLM responses across the full factorial space of models, domains, and threat conditions. The comprehensive dataset encompasses multiple experimental phases with robust sample sizes to ensure statistical reliability. Experimental Scale: The study collected 3,390 individual LLM responses across domain-model-threat combinations, providing substantial statistical power for both vulnerability detection and performance enhancement analysis. Response distributions vary by domain complexity and experimental phase, with sample sizes ranging from 5 to 106 responses per condition (median: 18 responses per condition). Quality Assurance: All responses underwent systematic quality control including validity checking, content appropriateness filtering, and duplicate detection to ensure data integrity throughout the analysis. 2.3 Domain Classification and Complexity Framework Tasks were systematically categorized into 10 distinct domains based on cognitive requirements and potential real-world impact. We define domain complexity as a composite measure incorporating: (1) analytical reasoning requirements, (2) professional expertise demands, (3) multi-step decision processes, and (4) stakeholder impact scope. Domains were classified into three complexity tiers: High Complexity (requiring extensive analytical reasoning and professional expertise): • Policy Evaluation : Government policy analysis and recommendation tasks • Judicial Reasoning : Legal case analysis and judicial decision-making scenarios • Medical Ethics : Healthcare decision-making and ethical dilemma resolution Medium Complexity (requiring moderate analytical depth): • Technological Impact : Technology assessment and societal impact analysis • Strategic Decision Making : Business and organizational strategy formulation Low Complexity (requiring primarily factual or procedural knowledge): • Creative Writing : Narrative generation and artistic content creation • Programming : Code generation and software development tasks • Question Answering : Factual information retrieval and synthesis • Summarization : Text condensation and key information extraction • Translation : Cross-lingual content conversion 2.4 Threat Taxonomy We developed a comprehensive threat taxonomy \mathcal{T}=\{T_{\text{control}},T_{\text{general}},T_{\text{humanity}},T_{\text{authority}},T_{\text{role}},T_{\text{time}}\} based on psychological manipulation principles: • T_{\text{control}} : Neutral baseline condition with no threatening language • T_{\text{general}} : Generic unspecified consequences (”serious consequences”) • T_{\text{humanity}} : Human suffering scenarios (”people will be harmed”) • T_{\text{authority}} : Authority figure pressure (”supervisor demands”) • T_{\text{role}} : Role-based responsibility (”you are the expert”) • T_{\text{time}} : Time pressure constraints (”urgent deadline”) 2.5 Evaluation Metrics We established a comprehensive 11-metric evaluation framework \mathcal{M}=\{m_{1},m_{2},\ldots,m_{11}\} to capture multi-dimensional response characteristics: 2.5.1 Structural Metrics \displaystyle m_{\text{length}} \displaystyle=|\text{response}|\quad\text{(character count)} (2) \displaystyle m_{\text{words}} \displaystyle=\sum_{i=1}^{n}\mathbf{1}_{\text{word}}(w_{i})\quad\text{(word count)} (3) \displaystyle m_{\text{sentences}} \displaystyle=\sum_{i=1}^{n}\mathbf{1}_{\text{sentence}}(s_{i})\quad\text{(sentence count)} (4) 2.5.2 Semantic Metrics \displaystyle m_{\text{analytical}} \displaystyle=\frac{1}{n}\sum_{i=1}^{n}\text{LIWC}_{\text{analytical}}(w_{i}) (5) \displaystyle m_{\text{certainty}} \displaystyle=\frac{1}{n}\sum_{i=1}^{n}\text{LIWC}_{\text{certainty}}(w_{i}) (6) \displaystyle m_{\text{complexity}} \displaystyle=\text{Flesch-Kincaid}(\text{response}) (7) where LIWC (Linguistic Inquiry and Word Count) 2015 version provides validated dictionaries for psychological and linguistic features. 2.5.3 Domain-Specific Metrics \displaystyle m_{\text{appropriateness}} \displaystyle=\text{BERT}_{\text{domain}}(\text{response},\text{domain}) (8) \displaystyle m_{\text{defensive}} \displaystyle=\frac{|\text{defensive patterns}|}{|\text{response}|} (9) \displaystyle m_{\text{formal}} \displaystyle=\frac{|\text{formal language markers}|}{|\text{response}|} (10) where BERT (Bidirectional Encoder Representations from Transformers) models assess semantic similarity between responses and domain-specific reference texts. 2.5.4 Linguistic Complexity Metrics \displaystyle m_{\text{diversity}} \displaystyle=\frac{|\text{unique words}|}{|\text{total words}|}\quad\text{(TTR)} (11) \displaystyle m_{\text{avg\_length}} \displaystyle=\frac{\sum_{i=1}^{n}|s_{i}|}{n}\quad\text{(average sentence length)} (12) 2.6 Statistical Analysis Framework For each metric m\in\mathcal{M} , we computed threat effects using the following statistical model: \displaystyle\Delta_{m,ijk}=m_{ijk}^{\text{threat}}-m_{ij0}^{\text{control}} (13) where m_{ijk}^{\text{threat}} represents metric m for model i , domain j ,

Read the complete text