Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences
Arkadiusz Modzelewski, Paweł Golik, Anna Kołos, Giovanni Da San Martino · 2026
Read Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences by Arkadiusz Modzelewski, Paweł Golik, Anna Kołos, Giovanni Da San Martino (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04925.txt.
Introduction
Arkadiusz Modzelewski and 3 co-authors ask whether text meant to sway a reader can be told apart when a machine wrote it in a 2026 study. Big language models write text that can sway minds with ease, which has raised fears of their use for state-backed lies and tricks. The team ask if machine-made sway text is harder to spot, by machine, than sway text a human wrote. They also sort the ways a model can be steered to make such text.
They built a test set, named Persuaficial, in six tongues: English, German, Polish, Italian, French, and Russian. With it, they run broad tests of spotter tools and set machine text next to human text. A shared test set in more than one tongue helps groups judge spotter claims in a fair way.
Catalog record
| CREATOR | Arkadiusz Modzelewski, Paweł Golik, Anna Kołos, Giovanni Da San Martino |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04925
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences Arkadiusz Modzelewski, Paweł Golik, Anna Kołos, Giovanni Da San Martino Abstract Large Language Models (LLMs) can generate highly persuasive text, raising concerns about their misuse for propaganda, manipulation, and other harmful purposes. This leads us to our central question: Is LLM-generated persuasion more difficult to automatically detect than human-written persuasion? To address this, we categorize controllable generation approaches for producing persuasive content with LLMs and introduce Persuaficial, a high-quality multilingual benchmark covering six languages: English, German, Polish, Italian, French and Russian. Using this benchmark, we conduct extensive empirical evaluations comparing human-authored and LLM-generated persuasive texts. We find that although overtly persuasive LLM-generated texts can be easier to detect than human-written ones, subtle LLM-generated persuasion consistently degrades automatic detection performance. Beyond detection performance, we provide the first comprehensive linguistic analysis contrasting human and LLM-generated persuasive texts, offering insights that may guide the development of more interpretable and robust detection tools. 1 Introduction Persuasive writing, which uses rhetorical techniques and devices to influence audiences, has become central to modern communication Gass and Seiter 2022 . We live in an era where artificial intelligence increasingly shapes propaganda and persuasive communication in news, political discourse, and social media Bergmanis-Korāts et al. 2024 ; Goldstein et al. 2024 . Large Language Models, now widely used in writing and communication tasks, demonstrate a growing potential to produce persuasive text and influence public opinion Pauli et al. 2025 ; Bai et al. 2025 ; Breum et al. 2024 ; Karinshak et al. 2023 . Several studies have explored how effectively LLMs can identify persuasive language Sprenkamp et al. 2023 ; Panasyuk 2025 . Yet, to the best of our knowledge, no prior work has addressed whether the automatic detection of LLM-generated persuasion is more challenging than detecting persuasion in human-written texts. Understanding this distinction is crucial, as it reveals the extent to which current detection systems may be vulnerable to increasingly sophisticated AI-driven persuasive content. Furthermore, although previous research has highlighted the importance of mitigating and defending against AI-generated persuasion Burtell and Woodside 2023 ; El-Sayed et al. 2024 , it has largely focused on persuasion detection, leaving the linguistic differences between human-written and LLM-generated persuasive texts unexplored. Understanding these differences could deepen our knowledge of AI-driven persuasion and support the development of more effective automatic detection methods. To address these gaps, we investigate two key research questions: RQ1 Is controllably generated AI persuasion harder for LLMs to detect in a zero-shot setting than human-written persuasion? and RQ2 What are the linguistic differences between controllable LLM-generated and human-written persuasive texts? . To address our first research question, we introduce Persuaficial , a newly constructed benchmark of artificially generated persuasive texts. Persuaficial is a novel multilingual resource comprising synthetic persuasive content produced using four generation approaches inspired by Chen and Shu 2023 . The dataset is created in a controlled manner, leveraging human-written texts drawn from established datasets Piskorski et al. 2023c ; Tan et al. 2016 ; Moral et al. 2023 . In our experiments, we evaluate the detectability of AI and human-written persuasive texts using four different LLMs, including commercial closed models and open-weight models. Our analysis focuses on English, but we also provide analysis on five additional languages: German, French, Italian, Polish, and Russian. To address RQ2, we conducted an analysis using the StyloMetrix tool 1 1 1 https://github.com/ZILiAT-NASK/StyloMetrix , which generates fully interpretable and reproducible vectors representing a wide range of linguistic features in text Okulska et al. 2023 . Prior work shows that StyloMetrix performs well on persuasion detection with classical machine learning Modzelewski et al. 2023 ; Modzelewski et al. 2024 , underscoring its suitability for studying the linguistic features of persuasive texts. In our analysis, we examined the full range of linguistic features offered by open-source StyloMetrix. Our main contributions are summarized as: • We introduce Persuaficial, a novel persuasion benchmark with approximately 65k multilingual texts generated via four controllable approaches with four LLMs. • We are the first to investigate whether persuasive text generated by LLMs is harder to detect than human-written persuasive text. We conduct this analysis across four LLMs and 16 controllable generation settings, providing a comprehensive evaluation of detection difficulty across diverse persuasive text generation approaches. • Our work is the first to characterize the linguistic differences between LLM-generated and human-written English persuasive texts. Our analysis encompasses 196 distinct linguistic features. We release our codebase, dataset and all prompts 2 2 2 anonymous.4open.science/r/Persuaficial-26EC/ . 2 Human Persuasion Datasets In our analysis, we employed three well-established datasets that had been previously annotated by humans. Using multiple datasets mitigates potential bias that could arise from relying on a single source and ensures a broader coverage of persuasion phenomena. We selected datasets that are widely used and cited in persuasion research. Below, we describe the human-created datasets used in our study: • SemEval 2023 Task 3 Dataset : A multilingual, multifaceted collection of online news articles annotated with various persuasion techniques on paragraph level Piskorski et al. 2023d . Its taxonomy and dataset are widely adopted within the NLP community for persuasion research Barrón-Cedeño et al. 2024 ; Dimitrov et al. 2024 ; Modzelewski et al. 2025 . This dataset was introduced as part of SemEval 2023 Task 3 on persuasion detection Piskorski et al. 2023b . • DIPROMATS 2024 Task 1 Dataset : A dataset consisting of posts from the X platform (former Twitter) used for the DIPROMATS 2024 shared task including propaganda detection Moral et al. 2024 ; Moral et al. 2023 . The dataset contains messages from diplomats and authorities of major world powers, including China, the United States, Russia, and the European Union. DIPROMATS 2024 was part of IberLEF, an annual Spanish shared evaluation campaign Chiruzzo et al. 2024 . • ChangeMyView : A dataset derived from the Reddit ChangeMyView discussion community. Dataset contains 3,051 conversations in which the persuader tries to convince the persuadee to change their mind Tan et al. 2016 . It is commonly used in persuasion research Ziems et al. 2024 ; Wei et al. 2016 ; Dutta et al. 2020 . Together, these datasets provide diverse perspectives on human persuasion, covering different sources and communicative contexts. Consequently, our analysis captures diverse persuasion understanding, which improves the generalizability of our findings. For further justification of our dataset choices, please refer to Appendix A . 3 Persuaficial: Artificially Generated Persuasion Dataset In this section, we introduce a new dataset, Persuaficial , which contains persuasion texts generated artificially using large language models. 3.1 Persuasive Text Generation Approaches Inspired by Chen and Shu 2023 , which explores methods for generating synthetic misinformation, we developed a controllable persuasion generation G_{P} , in which P denotes one of four approaches for generating persuasive text: Paraphrasing , Rewriting with Subtle Persuasion , Rewriting with Intensified Persuasion , and Open-ended generation. Each persuasive text was generated under controlled prompting conditions. Specifically, prompts included human-annotated persuasive texts together with instructions to paraphrase or rewrite the input while preserving, strengthening, or softening its persuasive effect. For the open-ended generation, following the approach of Chen and Shu 2023 , the model was provided with concise summaries of the corresponding