Promoting Online Safety by Simulating Unsafe Conversations with LLMs DOI: XXXXXXX.XXXXXXX Conference: ACM Conference on Conversational User Interfaces Workshop on Personas Evolved: Designing Ethical LLM-Based Conversational Agent Personalities; July 8-10; Waterloo, Canada
Owen Hoffman, Kangze Peng, Zehua You, Sajid Kamal, Sukrit Venkatagiri · 2025
Read Promoting Online Safety by Simulating Unsafe Conversations with LLMs DOI: XXXXXXX.XXXXXXX Conference: ACM Conference on Conversational User Interfaces Workshop on Personas Evolved: Designing Ethical LLM-Based Conversational Agent Personalities; July 8-10; Waterloo, Canada by Owen Hoffman, Kangze Peng, Zehua You, Sajid Kamal, Sukrit Venkatagiri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22267.txt.
Introduction
Owen Hoffman and four co-authors teach online safety by letting two chatbots act out a scam while learners coach the victim. Their work appeared in twenty twenty-five. Language models make fraudulent chats cheap and convincing, so the public needs practice spotting them.
One model plays the scammer, another the target. Users watch the exchange unfold and feed advice to the target along the way. Learning research says feedback on hypothetical choices sticks better than lectures. Earlier work had shown models can simulate scams realistically enough to train on. The system turns that unsettling skill defensive. The authors presented the design at a conversational AI workshop on ethical agent personalities.
Catalog record
| CREATOR | Owen Hoffman, Kangze Peng, Zehua You, Sajid Kamal, Sukrit Venkatagiri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Promoting Online Safety by Simulating Unsafe Conversations with LLMs DOI: XXXXXXX.XXXXXXX Conference: ACM Conference on Conversational User Interfaces Workshop on Personas Evolved: Designing Ethical LLM-Based Conversational Agent Personalities; July 8-10; Waterloo, Canada Owen Hoffman, Kangze Peng, Zehua You, Sajid Kamal, Sukrit Venkatagiri Abstract . 1. Introduction Unsafe conversations online range from impersonation, smishing, extortion, and harassment, to financial and sexual grooming, cyberbullying, and doxxing, among others. These unsafe conversations have caused significant harm, from people losing hundreds of thousands of dollars in scams to mental anguish and self-harm. Generative AI, including large language models (LLMs) have the potential — and already are being used — to increase the speed, scale, and types of unsafe conversations online ( Gressel et al. 2024 ; Tan et al. 2024 ) . LLMs lower the barrier for entry for bad actors to create unsafe conversations in particular because of their ability to generate persuasive and human-like text. In our current work, we explore ways to promote online safety by teaching people about unsafe conversations that can occur online with and without LLMs. We build on prior work such as ScamGPT-J ( Tan et al. 2024 ) that show that LLMs can successfully simulate scam conversations. We also leverage research in the learning sciences that shows that providing feedback on one’s hypothetical actions can promote learning ( Ion et al. 2019 ) . In particular, we focus on simulating scam conversations using LLMs . Our work incorporates two LLMs that converse with each other to simulate realistic, unsafe conversations that people may encounter online between a scammer LLM and a target LLM but users of our system are asked provide feedback to the target LLM. 2. Related Work Prior work has shown that roleplaying through simulation games can help people develop new skills, ranging from negotiation and conflict resolution ( Ebner and Efron 2005 ; Jin et al. 2018 ) to trauma counseling ( noa 2022 ) and cybersecurity awareness ( Kumaraguru et al. 2008 ) . Simulation has also been used to increase empathy for targets of cyberbullying ( Wright et al. 2009 ) and resilience to sexual grooming ( Susi et al. 2019 ) . Recent work in HCI has explored the use of LLMs to provide general-purpose advice ( Milana et al. 2023 ) and tailored to specific applications ( Guyre et al. 2024 ; Markel et al. 2023 ; Shaikh et al. 2024 ) . For example, Guyre et al. ( Guyre et al. 2024 ) fine-tuned OpenAI’s GPT Builder to provided advice to perspective managers by providing 23 source material across three phases. While Guyre’s system followed a question-answering structure, more recent work has sought to simulate actual conversation, e.g., Markel et al. ( Markel et al. 2023 ) developed GPTeach to train novice teachers by practicing with simulated students and Desai et al. ( Desai et al. 2023 ) developed an interactive voice UI to help older adults learn about health-related topics. GPTeach enabled teachers to develop their skills without negatively impacting actual students and the simulated environment gave teachers access to students with varying personas, needs, and learning goals. Similarly, Shaikh et al. ( Shaikh et al. 2024 ) leveraged prompt engineering and zero-shot prompting with OpenAI’s GPT-4 model to develop Rehearsal, a system to simulate conflict to teach conflict resolution. Given a conflict premise, Rehearsal not only generated messages for users to respond to, but also scored users’ own responses against a rubric. Shaikh et al.’s between-subjects evaluation found that Rehearsal improved against a baseline conflict resolution teaching strategy by an average of 67%. GPTeach and Rehearsal simulated specific conversational contexts but did not design conversational agent with tailored personas. A parallel body of work has explored ways to infuse personality into conversational agents ( Baik et al. 2025 ; Ahmad et al. 2022 ; noa 2023 ; Kovacevic et al. 2024 ) . For instance, through design workshops, Kim et al. ( Kim et al. 2019 ) identified different dimensions that conversational agents should have such as age, race, and gender, as well as what characteristics should be common or distinctive between different agents. With LLMs, recent studies have shown that personas can be easily created through prompt engineering involving only a few utterances ( Han et al. 2022 ; Gu et al. 2023 ) . In an experimental study involving LLMs with high and low extroversion and agreeableness, Sonlu et al. ( Sonlu et al. 2024 ) found that higher levels of extroversion and agreeableness was associated with longer responses from participants. Although it is now easier to create conversational agents with personas, Pradhan et al. ( Pradhan and Lazar 2021 ) discuss three challenges with designing personas. First, personas may not match user expectations due to incorrect assumptions made by designers. Second, effective personas may lead to users personifying CAs which may lead to misplaced trust and developing harmful emotional connections. Third, personas may continue to reinforce harmful stereotypes — including docile or subservient-seeming CAs modeled after young women that are used as assistants or "race-blind" CAs that do not acknowledge the existence of race may unintentionally perpetuating racial stereotypes. 3. Our Approach: Simulating Scam Conversations to Increase Resilience In our current work, we are developing a system where two LLMs engage in conversation with each other. One LLM is the scammer LLM and the other is the target LLM . The scammer is tasked with extracting information from the target and the target is tasked with not divulging this information. In addition, our system enables users to provide feedback to the target LLM to promote a more interacting learning environment. The two LLMs also have distinct personalities. The scammer LLM is instructed via the system prompt to develop a sense of urgency and to continue asking for more information from the target. The target LLM is instructed to be kind and understanding such that they may more easily fall victim to a scam. The user is tasked with preventing the target from falling for the scam. To evaluate the system fully, we used different models from OpenAI and tried other companies LLMs too. In these evaluations, we found that the OpenAI models are more aggressive, assertive and would sometimes ignore the user’s feedback which is great for the scammer LLM but not ideal for the target LLM. To try and make the models adhere to the system prompt better, we adjusted the temperature. However, this was not too effective for the OpenAI models because the personality traits remained the same. But in comparison, when we used Gemini, we found that the model acted less assertively and seemed to take in the feedback better, which is desirable for the target agent. Thus, we combined these two models and use the OpenAI model as the scammer LLM and Gemini’s model as the target LLM. Finally, we switched from zero to few shot prompting to give our agents examples of conversations that they should model, and this worked better than zero shot prompting. Using these tools we were able to make an effective scam conversation. Another issue that we faced due to model safety constraints was that the word “scam” or similar words lead to the