Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects
Anonymous Authors, Fred Heiding 2 , Simon Lermen 4 , Andrew Kao 2 , Bruce Schneier 2 , Arun Vishwanath 3 · 2024
Read Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects by Anonymous Authors, Fred Heiding 2 , Simon Lermen 4 , Andrew Kao 2 , Bruce Schneier 2 , Arun Vishwanath 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00586.txt.
Introduction
Fred Heiding, Simon Lermen, Andrew Kao, Bruce Schneier, and Arun Vishwanath test if large language models can run fully automated spear phishing drives, checked on human subjects. Spear phishing crafts personal lure emails for each target. Their work came out in twenty twenty-four.
Four email groups went to a pool of over a hundred people. Plain lure emails got a low click rate, while expert made lures scored far higher. Fully AI made lures matched the experts, and AI lures with a human in the loop scored a touch higher. A custom tool ran the whole drive, from data gathering to personal risk profiles, which were right and useful in most cases. The authors warn of fast growth in AI lure skill.
Catalog record
| CREATOR | Anonymous Authors, Fred Heiding 2 , Simon Lermen 4 , Andrew Kao 2 , Bruce Schneier 2 , Arun Vishwanath 3 |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects Anonymous Authors, Fred Heiding 2 , Simon Lermen 4 , Andrew Kao 2 , Bruce Schneier 2 , Arun Vishwanath 3 Abstract In this paper, we evaluate the capability of large language models to conduct personalized phishing attacks and compare their performance with human experts and AI models from last year. We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through). Thus, the AI-automated attacks performed on par with human experts and 350% better than the control group. The results are a significant improvement from similar studies conducted last year, highlighting the increased deceptive capabilities of AI models. Our AI-automated emails were sent using a custom-built tool that automates the entire spear phishing process, including information gathering and creating personalized vulnerability profiles for each target. The AI-gathered information was accurate and useful in 88% of cases and only produced inaccurate profiles for 4% of the participants. We also use language models to detect the intention of emails. Claude 3.5 Sonnet scored well above 90% with low false-positive rates and detected several seemingly benign emails that passed human detection. Lastly, we analyze the economics of phishing, highlighting how AI enables attackers to target more individuals at lower cost and increase profitability by up to 50 times for larger audiences. I Introduction Close to 20 years ago, Dhamija et al. wrote a paper entitled “Why Phishing Works,” [ 1 ] explaining that phishing exploits inherent weaknesses in the human brain and cognition. Unfortunately, phishing still works, and thanks to the rapid development of artificial intelligence (AI), it works better than ever [ 2 , 3 , 4 , 5 ] . Technical advancements in AI are improving rapidly and can be used by attackers, while human cognition and mental heuristics remain as easily exploitable as they were 20 years ago [ 6 , 7 ] . Language models, a type of generative AI, allow attackers to create human-like text of high quality in many different languages for almost no cost [ 8 , 9 ] . They also excel at persuasion [ 10 , 11 , 12 ] . Language model-powered AI assistants like ChatGPT 1 1 1 https://chat.openai.com/ and Claude 2 2 2 https://claude.ai have become commonplace in everyday activities worldwide. By January 2023, ChatGPT had become the fastest-growing consumer software application in history, gaining over 100 million users in two months 3 3 3 https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ . Many cyberattacks start by exploiting human users or include some element of social engineering. The Sony Pictures hack [ 13 , 14 ] and the $100 million MGM casino breach [ 15 ] are good examples. Some researchers claim that over 70–80% of cyberattacks involve social engineering techniques [ 7 , 16 ] . Thus, phishing attacks are a significant national security concern, 4 4 4 https://www.nsa.gov/Press-Room/Press-Releases-Statements/Press-Release-View/Article/3560788/how-to-protect-against-evolving-phishing-attacks/ and they are rapidly becoming more frequent. FBI’s Internet Crime Complaint Center [ 17 , 18 ] received over 200% more reported phishing attacks in 2023 than in 2019 (an increase from around 115,000 to 300,000). As phishing is well-suited for AI automation, it will likely become an even more pressing issue in the coming years. Consequently, the White House recently issued a memorandum (October 2024) stating the need for improved evaluations of AI models’ capability to conduct phishing and other cyberattacks [ 19 ] . In this study, we evaluate large language models’ capability to conduct personalized phishing attacks. To that end, we compare the success rate of four email types: a control group of scam emails from online databases, phishing emails created by human experts, AI-generated phishing emails, and AI-generated phishing emails assisted by human-in-the-loop interventions. The emails were sent to 101 human participants recruited for the study. Our AI-automated emails were sent using a custom-built AI-powered tool that performs reconnaissance based on scraping the target’s digital footprint, then creates and sends a personalized email, and evaluates the success of the chosen deception strategy. Section III-A described the tool. The control group emails received a click-through rate of 12%, the emails generated by human experts achieved 54%, the fully AI-automated emails 54%, and the AI emails utilizing a human-in-the-loop 56%. Our results provide an evaluation of frontier models’ phishing capabilities based on real-world empirical data. Furthermore, they showcase the increasing sophistication of AI-automated spear phishing. The AI-automated information scraping tool discovered accurate and useful information about the participants in 88% of the cases and, as shown above, created phishing emails that perform on par with human experts. This is a significant improvement from last year, where several studies found that AI models needed human-in-the-loop intervention to perform on par with human experts [ 20 , 21 , 2 ] . We also used five popular LLMs (Claude 3.5 Sonnet, GPT-4o, Mistral, LLama 3.1, and Gemini) to detect the intention of 20 phishing emails, described in Section IV . Based on the initial detection results, we selected the two most promising models (Claude 3.5 Sonnet and GPT-4o) for an in-depth analysis using a larger dataset of 381 emails (18 legitimate and 363 phishing emails). Claude 3.5 Sonnet showed the strongest initial performance, detecting 100% of the first 20 emails, including non-intuitive phishing attempts that had successfully fooled human targets and were deemed difficult to detect by the authors. We discovered that models perform significantly better when primed for suspicion (asked to determine whether the email is suspicious rather than to determine the email’s intention). Importantly, this priming did not increase false positive rates, making it a promising strategy for future use. In our analysis of the larger dataset, Claude 3.5 Sonnet achieved a 97.25% detection rate on the 363 phishing emails with no false positives, which we believe could still be improved with better prompt engineering. Lastly, in Section VI , we present an economic analysis of how AI affects the cost-effectiveness of phishing, showing that AI increases phishing profitability by up to 50 times. If attackers can recover the initial development cost, AI automation is almost always more beneficial than traditional phishing, highlighting the need for new defense strategies, policies, and mitigation techniques. We will continue to evaluate frontier AI models’ capability to launch phishing attacks and deceive users. If the current pace of development continues, the deceptive capabilities of language models will soon surpass human experts. Language models can also be used to defend against phishing, but they increase the attackers’ incentives far more than they benefit defenders. Thus, we urge researchers, policymakers, and technical practitioners to understand the severity of AI-enhanced phishing and increase our efforts to counter it via new technical, organizational, and policy-oriented mitigation strategies. II Related work Language models have improved rapidly during the past years, and their proficiency in creating realistic, coherent, and persuasive text makes them excellent tools for phishing. Thus, recent research has extensively explored the intersection of large language models (LLMs) and phishing attacks. Several studies evaluate