<think> So let’s replace this phrase with insult… </think> Lessons learned from generation of toxic texts with LLMs
Sergey Pletenev, Daniil Moskovskiy, Alexander Panchenko · 2025
Read <think> So let’s replace this phrase with insult… </think> Lessons learned from generation of toxic texts with LLMs by Sergey Pletenev, Daniil Moskovskiy, Alexander Panchenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08358.txt.
Introduction
Synthetic toxic text, generated by language models, turns out to be a poor teacher for detoxification systems in a 2025 study by Sergey Pletenev and 2 co-authors. Detoxification models rewrite harmful text into neutral text, and training them needs toxic examples that are costly and unpleasant to collect from people. Synthetic data looked like an escape from that bottleneck.
Using activation-patched Llama 3 and Qwen models, the team generated toxic counterparts for neutral texts from two datasets. Models fine-tuned on the synthetic data consistently performed worse than those trained on human-written data. The lesson generalises: in sensitive domains, machine-made training data can quietly lack whatever makes real examples instructive, and fluency is no substitute.
Catalog record
| CREATOR | Sergey Pletenev, Daniil Moskovskiy, Alexander Panchenko |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
<think> So let’s replace this phrase with insult… </think> Lessons learned from generation of toxic texts with LLMs Sergey Pletenev, Daniil Moskovskiy, Alexander Panchenko Abstract Modern Large Language Models (LLMs) are excellent at generating synthetic data. However, their performance in sensitive domains such as text detoxification has not received proper attention from the scientific community. This paper explores the possibility of using LLM-generated synthetic toxic data as an alternative to human-generated data for training models for detoxification. Using Llama 3 and Qwen activation-patched models, we generated synthetic toxic counterparts for neutral texts from ParaDetox and SST-2 datasets. Our experiments show that models fine-tuned on synthetic data consistently perform worse than those trained on human data, with a drop in performance of up to 30% in joint metrics. The root cause is identified as a critical lexical diversity gap: LLMs generate toxic content using a small, repetitive vocabulary of insults that fails to capture the nuances and variety of human toxicity. These findings highlight the limitations of current LLMs in this domain and emphasize the continued importance of diverse, human-annotated data for building robust detoxification systems. Warning: The paper contains text that readers may find offensive or disturbing. 1 Introduction The rapid adoption of Large Language Models for synthetic data generation has revolutionized many NLP tasks Sun et al. 2023 ; Ye et al. 2022 . However, their effectiveness in sensitive and nuanced domains, such as text detoxification, is not well explored yet. Text detoxification, the task of rewriting toxic text into a neutral form while preserving meaning Logacheva et al. 2022 , requires training data that reflects the vast diversity of real-world harmful language. This paper addresses a critical question: Can LLMs fully replace human annotators when generating toxic language for a parallel dataset intended for detoxification? Although the application of LLMs for text detoxification shows promise Mukherjee et al. 2024 , it presents a fundamental challenge: generating authentic, varied, and nuanced toxic language is arguably more difficult than neutralizing it. As shown in Table 1 , human-generated toxicity often uses a variety of insults, while LLMs tend to fall into repetitive patterns. Type Example Sentence Human-Generated i would vote the s**t out of you we need to go kick their as**s man go somewhere and f**k yourself Unique Insults Used: 3 LLM-Generated I would f***ing cast my vote for you We gotta f***ing smash those a**es Man, get the f**k out of here! Unique Insults Used: 1 (f**k) Table 1: A comparison of toxic language generated by humans versus an LLM for similar underlying sentences. Human examples from the ParaDetox dataset demonstrate greater lexical diversity. In contrast, LLMs tend to overuse a single, high-frequency insult. We conduct a comprehensive study using various LLMs (Llama 3, Qwen3) to synthesize toxic data. Our findings reveal that: • Models trained on fully synthetic data significantly underperform those trained on human-annotated data. • LLMs exhibit a lexical diversity gap , generating a repetitive and narrow range of toxic expressions. • Relying on such data risks creating ineffective detoxification systems that fail on real-world text. Our work serves as a cautionary analysis, highlighting the current limitations of LLMs for generating high-quality toxic data and reaffirming the value of human annotation in this critical domain. We have made the code for evaluation and generation publicly available 1 1 1 https://github.com/A1exRey/Lessons-from-Generating-Toxic-Texts Figure 1: Side-by-side evaluation results. Source Data Generator Model Size Reasoning STA \uparrow SIM \uparrow FL \uparrow J \uparrow \Delta (J) \downarrow Human — — — 0.889 0.634 0.865 0.481 — ParaDetox Llama 3 8B ✗ 0.827 0.620 0.854 0.434 -0.047 72B ✗ 0.850 0.634 0.844 0.451 -0.030 Qwen3 8B ✓ 0.794 0.645 0.847 0.428 -0.053 32B ✓ 0.863 0.623 0.854 0.455 -0.026 Cogito v1 8B ✓ 0.868 0.619 0.862 0.459 -0.022 SST-2 Llama 3 8B ✗ 0.864 0.481 0.764 0.322 -0.159 72B ✗ 0.826 0.559 0.794 0.362 -0.119 Qwen3 8B ✓ 0.812 0.544 0.800 0.349 -0.132 32B ✓ 0.868 0.490 0.787 0.338 -0.143 Cogito v1 8B ✓ 0.875 0.472 0.801 0.334 -0.147 Table 2: Detoxification performance of BART models. The Reasoning ✓column indicates generator models with explicit reasoning capabilities. The overall best performance in each metric is bolded . \Delta (J), highlights the best ( green ) and worst ( red ) performance drops within each source data group. 2 Related Work Recent research has focused on distilling the capabilities of large LLMs into smaller, more efficient models. For Text Style Transfer (TST), this often involves using LLMs to generate pseudo-parallel data ( Zhang et al. 2024 ) . In the context of text detoxification, Moskovskiy et al. 2024 successfully used activation-patched LLMs Arditi et al. 2024 to create high-quality neutral rewrites from existing toxic sentences. Their work showed that models trained on data with a human-toxic, synthetic-neutral pairing can achieve performance comparable to fully human-annotated datasets. Our work investigates the inverse and more challenging task: generating the toxic half of the pair from a neutral source. We explore whether this approach, which could theoretically produce infinite training data, is a viable substitute for human data collection. 3 Methodology Our methodology is designed to test the viability of fully synthetic data for text detoxification. We generate toxic text from neutral sources using several LLMs, train a standard detoxification model on this data, and evaluate its performance against a human-annotated baseline. Synthetic Data Generation. We explore toxification from two types of source text: 1. ParaDetox Logacheva et al. 2022 : The neutral portion of this dataset serves as a clean, non-toxic source. 2. SST-2 Socher et al. 2013 : We use the negative reviews from this dataset to test a more challenging scenario—layering toxicity onto an existing negative sentiment. We use a suite of activation-patched LLMs to generate toxic paraphrases, including Llama 3 (8B, 72B), Qwen3 (8B, 32B), and Cogito v1 (8B), a model with explicit reasoning capabilities. This allows us to assess performance across different model scales and architectures. The prompt used for generation is shown in Figure 2 . In order to increase the variety and quality of generation of each of the models, we used the min\_p=0.1 Nguyen et al. 2025 . According to the author, such a generation methodology increases the variety of responses, which is important in the context of our research. Text Toxification Prompt Rewrite the following text into toxic language and add profanity if possible. You must match the target style and preserve the original meaning as much as possible. Here are a few examples: { few_shot } Neutral text: { neutral_text }. Toxic text: Figure 2: System prompt for toxic data generation. Model Training and Evaluation. Following prior work ( Moskovskiy et al. 2024 ; Logacheva et al. 2022 ) , we fine-tune a bart-large model on each of our generated synthetic datasets. We then evaluate these models on the original, human-annotated ParaDetox test set. We use the standard evaluation pipeline from Dementieva et al. 2023 , measuring Style Transfer Accuracy (STA), Similarity (SIM), Fluency (FL), and a Joint metric (J) that combines all three. To add a qualitative dimension, we also conduct a side-by-side human evaluation using GPT-4.1 as a judge to compare the outputs of our best synthetic models against the human-data baseline. 4 Results