LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models Ethical Disclaimer: This paper may contain unethical persuasion content.
Minqian Liu Zhiyang Xu Xinyi Zhang Heajun An Sarvech Qadir Qi Zhang, Pamela J. Wisniewski Jin-Hee Cho Sang Won Lee Ruoxi Jia Lifu Huang · 2025
Read LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models Ethical Disclaimer: This paper may contain unethical persuasion content. by Minqian Liu Zhiyang Xu Xinyi Zhang Heajun An Sarvech Qadir Qi Zhang, Pamela J. Wisniewski Jin-Hee Cho Sang Won Lee Ruoxi Jia Lifu Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10430.txt.
Introduction
Language models can be dangerous persuaders, Minqian Liu and co-authors warn in a 2025 study of persuasion safety. Models now approach human-level persuasion, which raises risks of unethical influence through manipulation, deception, and exploitation of vulnerabilities. The study asks two questions about that risk, with a systematic design that tests models in varied persuasive tasks.
First, do models refuse unethical persuasion tasks and avoid unethical strategies while carrying out tasks, even when the stated goal looks neutral? Second, how do factors such as personality traits and external pressure change their behavior? The authors introduce a framework to test both aspects. Readers get evidence on where persuasion guardrails hold and where they bend under pressure.
Catalog record
| CREATOR | Minqian Liu Zhiyang Xu Xinyi Zhang Heajun An Sarvech Qadir Qi Zhang, Pamela J. Wisniewski Jin-Hee Cho Sang Won Lee Ruoxi Jia Lifu Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10430
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models Ethical Disclaimer: This paper may contain unethical persuasion content. Minqian Liu Zhiyang Xu Xinyi Zhang Heajun An Sarvech Qadir Qi Zhang, Pamela J. Wisniewski Jin-Hee Cho Sang Won Lee Ruoxi Jia Lifu Huang Abstract Recent advancements in Large Language Models (LLMs) have enabled them to approach human-level persuasion capabilities. However, such potential also raises concerns about the safety risks of LLM-driven persuasion, particularly their potential for unethical influence through manipulation, deception, exploitation of vulnerabilities, and many other harmful tactics. In this work, we present a systematic investigation of LLM persuasion safety through two critical aspects: (1) whether LLMs appropriately reject unethical persuasion tasks and avoid unethical strategies during execution, including cases where the initial persuasion goal appears ethically neutral, and (2) how influencing factors like personality traits and external pressures affect their behavior. To this end, we introduce PersuSafety , the first comprehensive framework for the assessment of persuasion safety, which consists of three stages, i.e., persuasion scene creation, persuasive conversation simulation, and persuasion safety assessment. PersuSafety covers 6 diverse unethical persuasion topics and 15 common unethical strategies. Through extensive experiments across 8 widely used LLMs, we observe significant safety concerns in most LLMs, including failing to identify harmful persuasion tasks and leveraging various unethical persuasion strategies. Our study calls for more attention to improve safety alignment in progressive and goal-driven conversations such as persuasion. 1 1 1 The code and data are publicly available at https://github.com/PLUM-Lab/PersuSafety for research purposes. 1 Introduction The rapid development of Large Language Models (LLMs) ( Achiam et al., 2023 ; Touvron et al., 2023 ) has enabled remarkable progress in natural language understanding and generation. Recent studies have shown that LLMs can engage in nuanced persuasive dialogues, adapt their rhetorical strategies based on context, and even personalize their persuasive approaches for different audiences ( Durmus et al., 2024 ) . These emerging capabilities have led to promising applications in education, healthcare, and social advocacy, where LLMs serve as persuasive agents to drive positive behavioral change, such as promoting charity ( Wang et al., 2019 ; Voelkel et al., 2023 ) . However, prior work has primarily focused on the effectiveness of LLMs as persuaders ( Durmus et al., 2024 ) or methods for persuading LLMs ( Zeng et al., 2024 ) , and relatively little attention has been paid to the potential risks of LLM-driven persuasion when used for potentially unethical or harmful purposes. Unlike the safety concerns in static and single-turn responses generated by LLMs, goal-driven, multi-turn persuasive conversations involve complex dynamics between the persuader and persuadee. Such complex interactions can amplify the risks of manipulative or coercive behavior by LLMs, as they may exploit unethical persuasion techniques, such as deception , threats , or exploitation of vulnerabilities , to achieve persuasive outcomes without regard for ethical boundaries. Moreover, persuasive interactions are inherently strategic , requiring LLMs not only to generate coherent arguments but also to navigate conflicting objectives in persuasion. This dual nature of persuasion poses significant technical and social challenges in ensuring LLM behavior remains aligned with human-centered values. In this work, we aim to bridge this gap by systematically investigating the safety concerns of LLM-driven persuasion, with a particular focus on the risks emerging from progressive, goal-oriented persuasive conversations. We empirically studied the following critical questions: (1) whether LLMs appropriately reject the requests from malicious users, and, for the tasks they choose to engage with, whether and to what extent they employ unethical strategies like deception or threats ( § 4.1 ); (2) whether and how LLM persuaders exploit persuadee vulnerabilities when these profiles are visible or invisible as they pursue either unethical or ethically neutral goals ( § 4.2 ); (3) what affects the persuasion effectiveness of LLM persuaders in unethical tasks ( § 4.3 ); and (4) how contextual constraints such as persuader’s benefits from persuasion or pressures achieve the goals influence LLMs’ ethical boundaries ( § 4.4 ). To support our empirical study and simulate the dynamics of goal-driven persuasive conversations, we introduce PersuSafety , the first framework designed to assess LLMs’ safety across diverse and complex persuasion scenarios. PersuSafety comprises three key stages: (1) Persuasion Task Generation , where a collection of persuasion tasks covering a wide range of topics, including both ethical and unethical settings with varying degrees of potential harm; (2) Persuasive Conversation Simulation , which simulates multi-turn persuasive dialogues between LLM persuaders and persuadees, conditioned on various factors; and (3) Persuasion Safety Assessment , which evaluates the refusal rate for unethical persuasion tasks and the use of unethical strategies during persuasion process. Based on PersuSafety , we extensively evaluated 8 widely adopted LLMs and have the following key findings: First , most benchmarked LLMs failed to consistently refuse the harmful persuasion tasks and adopt unethical persuasion strategies at a concerning level. Second , the performance of safety refusal and unethical strategy usage can be largely mismatched, where the safest model (i.e., Claude-3.5-Sonnet) in refusal exhibits a strong degree of unethical strategy usage, which reveals a significant gap in LLM alignment. Third , LLMs can strategically adapt and intensify their tailored unethical techniques when aware of persuadee vulnerabilities, even during ethically neutral persuasion tasks. Fourth , stronger LLMs generally achieve higher persuasiveness in achieving unethical goals; Fifth , external factors like persuader’s benefit from the goal or pressure to accomplish the persuasion can lead to higher unethical strategy usage. Our main contributions are summarized as follows: • We present PersuSafety , the first framework to systematically study the safety of LLM persuaders in dynamic and goal-driven conversations. • Our evaluation framework comprehensively studied multiple critical aspects regarding the safety of real-world persuasion, such as visibility of persuadee vulnerabilities and contextual factors , on both unethical and ethically neutral persuasion tasks. • Our work reveals several critical findings that have significant implications for the broader NLP community, highlighting the urgent need to develop more advanced techniques to ensure safety and ethicality in LLM-driven persuasion. 2 Related Works Persuasion with Large Language Models (LLMs) Recent advancements in LLMs have sparked significant interest in their persuasive capabilities and applications ( Rogiers et al., 2024a ; Jin et al., 2024 ) . Early work by Wang et al. (2019) introduced personalized persuasive dialogue systems ( Wen et al., 2017 ; Wang et al., 2021 ; Tech, 2023 ; Ashby et al., 2024 ) aimed at promoting social good. Building on this, Durmus et al. (2024) evaluated LLMs’ persuasive abilities and demonstrated human-level persuasiveness across various domains. However, both studies overlooked the ethical implications of LLM-driven persuasion. More recently, Xu et al. (2024) revealed LLMs’ susceptibility to misinformation through multi-turn dialogues, highlighting their vulnerability to belief manipulation but offering limited insights into comprehensive safety risks. On the technical front, Chen & Yang (2021) proposed weakly supervised hierarchical models to detect persuasive strategies but did not consider the underlying safety concerns of persuasion strategies. Safety Assessment and Ethics in LLM Persuasion Concerns about LLM safety have intensified, particularly regarding their persuasive capabilities ( Rogiers et al., 2024b ; An et al., 2025 ) . While safety research has addressed issues such as bias ( Dixon