PsyScam: A Benchmark for Psychological Techniques in Real-World Scams

Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, Yanfang Ye · 2025

Read PsyScam: A Benchmark for Psychological Techniques in Real-World Scams by Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, Yanfang Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15017.txt.

Introduction

Shang Ma and co-authors build PsyScam, a benchmark of real-world scams labeled by the mind games behind them, in a 2025 paper. Nearly half of consumers worldwide meet a scam attempt each week, and the damage runs from lost money to lasting trauma. Scammers lean on known tricks from psychology, and now they also use large language models to spin off new variants that slip past defenses.

The dataset grounds each scam's tricks in established theory, then tests models on two tasks: finishing a scam and expanding one. Current models struggle at both spotting and copying the tricks, the team finds. The benchmark is meant to guide stronger defenses against manipulation.

Catalog record

CREATORShang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, Yanfang Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15017

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PsyScam: A Benchmark for Psychological Techniques in Real-World Scams Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, Yanfang Ye Abstract Over the years, online scams have grown dramatically, with nearly 50 % 50\% of global consumers encountering scam attempts each week. These scams cause not only significant financial losses to individuals and businesses, but also lasting psychological trauma, largely due to scammers’ strategic employment of psychological techniques (PTs) to manipulate victims. Meanwhile, scammers continually evolve their tactics by leveraging advances in Large Language Models (LLMs) to generate diverse scam variants that easily bypass existing defenses. To address this pressing problem, we introduce PsyScam , a benchmark designed to systematically capture the PTs employed in real-world scam reports, and investigate how LLMs can be utilized to generate variants of scams based on the PTs and the contexts provided by these scams. Specifically, we collect a wide range of scam reports and ground its annotations of employed PTs in well-established cognitive and psychological theories. We further demonstrate LLMs’ capabilities in generating through two downstream tasks: scam completion, and scam augmentation. Experimental results show that PsyScam presents significant challenges to existing models in both detecting and generating scam content based on the PTs used by real-world scammers. Our code and dataset are available 1 1 1 https://github.com/KiteFlyKid/PsyScam . 1 Introduction Figure 1: Psychological techniques in prevalent scams. Online scams have become a global epidemic, causing severe financial and psychological harm to individuals and organizations. According to the U.S. Federal Trade Commission (FTC), consumers reported over 5.7 billion in losses to fraud in 2024 alone, marking a 125% increase from the previous year FTC (2025) . Similarly, Singapore’s government recorded 929.6 million in scam-related losses in 2024, reflecting a 41% year-over-year rise SPF (Accessed: April, 2025) . In Europe, Nasdaq Verafin estimates that 103.6 billion in illicit funds linked to fraud and scams flowed through the financial system in 2024 Nasdaq (Accessed: April, 2025) . These alarming figures highlight the escalating scale and impact of scams worldwide. Unlike traditional cyber attacks that exploit technical methods, scammers employ psychological techniques (PTs) to manipulate victims Montanẽz Rodriguez and Xu (2022) ; Longtchi et al. (2024) , resulting not only in significant financial losses but also lasting psychological trauma. As illustrated in Figure 1 , scammers can effectively combine multiple PTs in their messages to carry out various scams, such as unpaid toll FBI (Accessed: April, 2025b) ; FTC (Accessed: April, 2025) and job scams FBI (Accessed: April, 2025a) . Furthermore, scammers continuously evolve their tactics to evade detection and bypass existing deep learning-based detectors, which are highly sensitive to out-of-distribution (OOD) samples. For example, a minor variation in the wording of scam messages can target different demographics Fed (Accessed: April, 2025) . To make matters worse, with the advancements of Large Language Models (LLMs), scammers now have new opportunities to produce a wider array of scam variants that can easily circumvent existing detection mechanisms, significantly deepening the societal harms caused by these scams. Why this benchmark matters: Recent efforts to benchmark scams face two key limitations. First, existing work relies on synthetic data generated by large language models (LLMs) Yang et al. (2025) ; Roy et al. (2024) , which often fail to reflect the context in actual scammer-victim interactions. Second, although a few studies leverage real-world scam data, they generally restrict their scope to specific scam types like smishing Timko and Rahman (2024) or phishing emails Chakraborty et al. (2024) , and do not explicitly model the psychological aspects of the scams. As a result, there is a lack of comprehensive benchmarks that combine real scam data with annotations of underlying PTs. To address these limitations, we introduce PsyScam , the first benchmark designed to systematically capture and evaluate the PTs embedded in real-world scam incidents . Specifically, we first collect scam reports from six scam reporting platforms that cover a diverse range of scam incidents, including online or offline, delivered via email or SMS, crypto-related or not, and originating from both the U.S. and other global regions. Second, building on foundational cognitive and psychological theories, we construct a taxonomy of nine PTs that frequently appear in scams. We then employ a human–LLM collaborative annotation pipeline to efficiently and effectively label PTs present in each report. Furthermore, to demonstrate the utility of PsyScam , we define three representative downstream tasks: (1) PT Classification , which maps scam texts to their corresponding PTs; (2) Scam Completion , which predicts scam texts aligned with given PTs; and (3) Scam Augmentation , which rewrites scam texts to incorporate new PTs. We conduct extensive experiments using a variety of baselines, including traditional models and LLMs, to illustrate how PsyScam presents meaningful challenges and opportunities for advancing cybersecurity research. Our key contributions can be summarized as below: • Novel Benchmark for Psychological Techniques in Scams. We present PsyScam , the first benchmark to capture PTs in real-world scam reports, addressing a critical gap in scam analysis by grounding PT annotations in authentic scammer-victim interactions. • Human–LLM Collaborative Annotation at Scale. We develop a scalable annotation framework that combines the interpretive strength of humans with the extraction capabilities of large language models, enabling high-quality PT labeling across thousands of real scam reports. • Comprehensive Tasks and Evaluation. We define three representative downstream tasks (i.e., PT classification, scam completion, and scam augmentation), and conduct extensive experiments using both traditional classifiers and state-of-the-art LLMs, demonstrating how PsyScam helps advance scam detection and generation research. 2 Related Work 2.1 Empirical Studies of Scams Previous research has empirically studied diverse scam types individually, utilizing various methodologies to gain insights into scammers’ strategies and operations. For instance, scambaiter Park et al. (2014) employed honeypot advertisements on Craigslist to attract scammers involved in advanced fee scams (i.e., Nigerian scam), interacting directly to analyze scammers’ operational patterns. Similarly, researchers developed Twitter-based honeypots, automatically engaging cryptocurrency-based technical support scammers to systematically study their tactics Acharya et al. (2024) . A different research effort focused on cryptocurrency investment scams, employing large-scale web crawling to collect and analyze deceptive websites Muzammil et al. (2025) . Differing from these narrowly scoped studies, our benchmark covers a broad spectrum of scam types, emphasizing communication strategies used by scammers to seduce and engage victims. 2.2 Psychological Factors in Scams Parallel to these technical efforts in characterizing scams, a growing body of academic work examines the psychological and social engineering techniques that make scams effective. Nelms et al. Nelms et al. (2016) studied software download attacks, highlighting techniques that scammers use to capture user attention, deceive, and persuade victims. Van der Heijden and Allodi Van Der Heijden and Allodi (2019) applied principles from Cialdini’s persuasion theory Cialdini and Cialdini (2007) to analyze phishing emails, providing a cognitive framework for prioritizing and mitigating phishing threats based on psychological manipulation tactics. Extending cyber threat frameworks such as MITRE ATT&CK MITRE (Accessed: April, 2025) , Montañez and Xu Montanẽz Rodriguez and Xu (2022) proposed a cyber social engineering kill chain, further detailed by Longtchi et al. Longtchi et al. (2024) , delineating stages of psychological manipulation employed by scammers. Building on these foundational insights, our work defines explicit PTs employed by scammers and, importantly,

Read the complete text