When Detection Fails: The Power of Fine-Tuned Models to Generate Human-Like Social Media Text
Hillary Dawkins, Kathleen C. Fraser, Svetlana Kiritchenko · 2025
Read When Detection Fails: The Power of Fine-Tuned Models to Generate Human-Like Social Media Text by Hillary Dawkins, Kathleen C. Fraser, Svetlana Kiritchenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09975.txt.
Introduction
Hillary Dawkins, Kathleen Fraser, and Svetlana Kiritchenko present a 2025 paper on how hard AI made social media posts are to spot when models are tuned for the job. Short, informal posts already strain detectors. The stakes are high, since fake posts can fuel campaigns that sway public choices.
The team wrote as a capable attacker would. They built a set of more than half a million AI posts on eleven hot topics, using open, closed, and fine tuned models. Under standard lab assumptions, detectors could catch the posts. The paper's title warns that this success breaks down once the attacker tunes the model, which makes the posts read as human.
Catalog record
| CREATOR | Hillary Dawkins, Kathleen C. Fraser, Svetlana Kiritchenko |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Detection Fails: The Power of Fine-Tuned Models to Generate Human-Like Social Media Text Hillary Dawkins, Kathleen C. Fraser, Svetlana Kiritchenko Abstract Detecting AI-generated text is a difficult problem to begin with; detecting AI-generated text on social media is made even more difficult due to the short text length and informal, idiosyncratic language of the internet. It is nonetheless important to tackle this problem, as social media represents a significant attack vector in online influence campaigns, which may be bolstered through the use of mass-produced AI-generated posts supporting (or opposing) particular policies, decisions, or events. We approach this problem with the mindset and resources of a reasonably sophisticated threat actor, and create a dataset of 505,159 AI-generated social media posts from a combination of open-source, closed-source, and fine-tuned LLMs, covering 11 different controversial topics. We show that while the posts can be detected under typical “research” assumptions about knowledge of and access to the generating models, under the more realistic assumption that an attacker will not release their fine-tuned model to the public, detectability drops dramatically. This result is confirmed with a human study. Ablation experiments highlight the vulnerability of various detection algorithms to fine-tuned LLMs. This result has implications across all detection domains, since fine-tuning is a generally applicable and realistic LLM use case. 1 Introduction Large language models (LLMs) are able to produce increasingly complex and fluent output, creating an information environment where humans can no longer reliably distinguish between text that was written by other humans, and that which was generated by LLMs. This increases our vulnerability to various tactics of opinion manipulation, particularly in online spaces such as social media. These tactics include disinformation and misinformation campaigns, as well as astroturfing and “flooding the zone” attacks. While growing research effort has focused generally on the development of methods for detecting AI-generated text (AIGT), very little work has focused on the pressing question of AIGT detection in the social media domain. Furthermore, much of the research has relied on assumptions that are highly unlikely to hold in real-world contexts, such as the assumption that the defender has knowledge of (or even access to) the LLM that was used to generate the text they are trying to detect. In the current work, we are particularly motivated by the concept of astroturfing , or the artificial creation of an impression of widespread support for (or opposition to) a product, policy, or concept Chan 2024 . By flooding the information space with posts advocating a particular stance on some issue, AI can be used to influence and change people’s beliefs by exploiting their natural tendency toward “crowd mentality” and desire for social acceptance. Crucially, astroturfing is distinct from disinformation, in that the sentiment being expressed is not necessarily false . This means that traditional methods of disinformation detection are not effective against such an attack. Rather, the deception involved in astroturfing is the inauthenticity of the apparent crowd Chan 2024 . Therefore, the ability to detect whether a social media post on some topic originates from a real human or from AI becomes an important signal for authenticity determination. Our work attempts to make realistic assumptions about the motivations and resources available to both attackers (those spreading the AIGT) and defenders (those detecting the AIGT) in this scenario. In contrast to previous work, which sampled social media posts on a broad variety of everyday topics Macko et al. 2024 , we assume that both attackers and defenders will be more motivated to participate in/monitor online discussions on controversial or political topics. We assume that the attacker will have a particular stance they want to communicate on a given topic, rather than simply summarizing or paraphrasing existing posts. We also assume they will take reasonable, low-cost steps to produce realistic text (such as fine-tuning the model to mimic the intended social media style), and that they will not release any fine-tuned models for the defender to access. While it is reasonable to assume that an actor will attempt to mimic the intended distribution through fine-tuning, we do not make any further assumptions that the attacker is actively trying to evade detection, which may require more sophisticated knowledge of modern detection schemes. Therefore, we examine four related research questions: • RQ1: Can today’s LLMs produce realistic social media style text? We explore prompting base models as well as models fine-tuned on human-written texts, and conduct linguistic analysis to compare the generated text with human text. We find that the output from fine-tuned models is more similar to human text than from the base models (Section 4.1 ). • RQ2: Assuming a more realistic attack scenario , do different generation strategies result in AIGT that is intrinsically harder to detect? We generate text using different prompting conditions, with four base models and four fine-tuned models, and compare the classification accuracies of eight detection methods using idealized in-domain train-test classification methodology. We find that a supervised RoBERTa-based classifier can detect AIGT from the base models with up to 99% accuracy, but this figure drops significantly for AIGT from the fine-tuned models (Section 4.2.1 ). • RQ3: Assuming a more realistic defense scenario , are detection methods still robust if they cannot access the generating model? Many research studies assume at least black-box access to the generating model. Relaxing this assumption, we find that the fine-tuned models are particularly hard to detect using off-the-shelf methods (Section 4.2.2 ). Ablation studies reveal that access to closely related fine-tuned models is not a viable strategy using the tested detectors (Sections 4.2.3 , 4.2.4 ). • RQ4: Can human readers detect AI-generated texts? In particular, we address the potential concern that fine-tuned models might generate text that automated methods cannot easily detect, but that human annotators can. We run a human study with 250 participants and find that texts from fine-tuned models are essentially undetectable to human readers (54% detection accuracy) (Section 4.3 ). In addition to these research contributions, we also make our dataset of 505,159 English-language AI-generated texts available to other researchers. 2 Background Here, we briefly summarize the state-of-the-art NLP approaches to detecting AIGT, along with their strengths and weaknesses; more in-depth background is available in recent survey papers Crothers et al. 2023 ; Ghosal et al. 2023 ; Uchendu et al. 2023 ; Yang et al. 2023 ; Fraser et al. 2025 ; Tang et al. 2024 ; Wu et al. 2025 . One category of detection methods is metric-based approaches that rely on detecting the statistical regularities of AIGT resulting from the generation process. These include measures of the log-likelihood or rank of each token, with respect to the model’s probability distribution over tokens at each step, or measures of the overall entropy or perplexity of the text Gehrmann et al. 2019 ; Su et al. 2023 . The DetectGPT algorithm Mitchell et al. 2023 extends these ideas to estimate the local curvature of the model’s log probability function, laying the groundwork for Fast-DetectGPT Bao et al. 2024 . These are all so-called “zero-shot” methods that do not require training data (although in practice, some in-domain data is needed to calibrate the decision threshold). They also rely on information about the probability distribution