DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large language models
Shantanu Thorat, Andrew Caines · 2025
Read DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large language models by Shantanu Thorat, Andrew Caines (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00619.txt.
Introduction
Shantanu Thorat and Andrew Caines published a 2025 paper on catching text written by AI when the AI had a human example to copy. Detectors of machine-written text often shine in lab tests and falter in the real world. Most test sets use text generated from scratch. Real misuse often starts from a sample: a student feeds an essay in and asks for one like it.
The authors built a dataset, DACTYL, focused on exactly those one-shot and few-shot generations. It also includes text from models further trained on special fields. Many current detectors struggled badly on this harder material. The team then trained their own classifiers two ways. The standard method edged ahead on familiar test text. A newer risk-based method won clearly on unseen text. In a mock student-essay scenario, it beat the standard method by a wide margin.
Catalog record
| CREATOR | Shantanu Thorat, Andrew Caines |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large language models Shantanu Thorat, Andrew Caines Abstract AI-generated (AIG) texts are diffusing rapidly into our daily lives, blurring the lines between human and AI. Some use cases are malicious, such as generating misinformation on social media, producing fake reviews to mislead consumers, or writing false news articles. Thus, there emerges a need to build an AIG text detector. However, existing AIG text detectors struggle in real-world settings despite succeeding in internal testing, suggesting that they may not be robust enough. We rigorously examine the machine-learning procedure to build these detectors to address this. Most current AIG text detection datasets focus on zero-shot generations, but little work has been done on few-shot or one-shot generations, where LLMs are given human texts as an example. In response, we introduce the Diverse Adversarial Corpus of Texts Yielded from Language models (DACTYL), a challenging AIG text detection dataset focusing on one-shot/few-shot generations. We also include texts from domain-specific continued-pre-trained (CPT) language models, where we fully train all parameters using a memory-efficient optimization approach. Many existing AIG text detectors struggle significantly on our dataset, indicating a potential vulnerability to one-shot/few-shot and CPT-generated texts. We also train our own classifiers using two approaches: standard binary cross-entropy (BCE) optimization and a more recent approach, deep X-risk optimization (DXO). While BCE-trained classifiers marginally outperform DXO classifiers on the DACTYL test set, the latter excels on out-of-distribution (OOD) texts. In our mock deployment scenario in student essay detection with an OOD student essay dataset, the best DXO classifier outscored the best BCE-trained classifier by 50.56 macro-F1 score points at the lowest false positive rates for both. Our results indicate that DXO classifiers generalize better without overfitting to the test set. Our experiments highlight several areas of improvement for AIG text detectors, which can aid in the ongoing race between detectors and adversaries. Keywords AI-generated text detection \cdot large language models \cdot deep X-risk optimization 1 Introduction 1.1 Overview AI-generated (AIG) text detection is the latest text classification task, a response to the flood of AIG texts in various domains, such as social media posts, reviews, and student essays. AIG text detection is a variation of authorship attribution — a classifier determines if an LLM or a human wrote a text. We highlight several use cases of such a classifier: • Student essays — Leading plagiarism detector Turnitin quickly deployed an AIG text detector to determine if students were passing off LLM-generated assignments as their own ( turnitinUnderstandingFalse ) . • Fake review detection — Roughly one in five Google Reviews may be AIG ( originalityFrom2019 ) . • Pre-screening LLM training corpora — shumailov2024ai demonstrated that Meta’s OPT-125M language model’s text generation degraded after training on its own texts — a phenomenon known as model collapse. The early generations of LLMs (pre-2022) did not have to worry about this issue, as individuals rarely employed LLMs. However, current LLM developers may have to start filtering out AIG texts in newer corpora, as LLM popularity explodes ( openai400million ) . Unfortunately, AIG text detection has many challenges — primarily that classifiers deployed in real-world settings struggle. For example, several universities have opted not to use Turnitin’s AI text detection tool due to many false positives ( vanderbiltGuidanceDetection ) . Yet, classifiers are capable of achieving high performance on various AIG text datasets: two classifiers recently achieved at least a 99% true positive rate at just a 5% false positive rate on the Robust AI Detection (RAID) dataset ( dugan_raid_2024 ) . Thus, it appears that classifiers have the potential to differentiate between human and AIG texts in controlled test settings, but it does not help them when facing AIG or human texts “in the wild.” This deployment gap inspires us to rigorously inspect traditional AIG text detector training. AIG text detection follows the conventional machine-learning workflow: construct a dataset, train a classifier, and evaluate on a test set (not used in training). Existing AIG datasets tend to include zero-shot (no example) LLM-generated text. We suspect that giving an LLM a human example would create a more human text, as LLMs are few-shot learners ( brown2020language ) . Providing an LLM with a human example is also a reasonable technique an adversary might use to evade detection. Classifiers often train with binary cross-entropy to minimize the difference between the classifier’s predicted scores (typically probability scores from 0 to 1) and the target label distribution ( pytorchBCELossx2014 ) . Evaluation is done with conventional binary classification metrics, which we explain later in Chapter 2. Given the strong performance on test sets but not in real-world deployment, we suspect that classifiers may overfit to the test set (which follows the training distribution more closely than thought). Our main contributions to the AIG text detection workflow are as follows. • We introduce the Diverse Adversarial Corpus of Texts Yielded from Large language models (DACTYL) dataset, an AIG text detection dataset covering six domains and including one-shot/few-shot generated texts from 11 LLMs. Additionally, we provide an adversarial test set consisting of texts generated from 18 small language models (SLMs) which we have continued to pre-train (earlier works refer to this as fine-tuning ( salminen_creating_2022 ) ). • We train our classifiers on the DACTYL dataset using deep X-risk optimization. X-risks refer to a group of evaluation metrics that compare how well a classifier can contrast a positive sample (AIG text) with a negative sample (human text) ( yuan2023libauc ) . Area Under the Curve (AUC) is a standard X-risk metric. • We evaluate our DACTYL-trained classifiers alongside existing AIG text detectors using the two-way partial AUC (Area Under the Curve) metric. We select this metric and three other X-risk metrics, as it is far less optimistic about classifier performance and highlights differences between classifiers better. • Finally, to ensure that our classifiers are generalizing, we evaluate them on an additional test set containing AIG texts from existing datasets to simulate an out-of-distribution (OOD) scenario to reflect real-world conditions better. We observe that pre-trained detectors struggle significantly on the DACTYL dataset, demonstrating a potential weakness in detection. Our continued pre-trained SLMs also degrade performance (compared to vanilla or “out-of-the-box” SLMs) — even for DACTYL-trained classifiers. While deep X-risk optimized (DXO) classifiers do not perform as well as the binary cross-entropy (BCE) trained classifiers on the DACTYL test set, the DXO classifiers outperform the BCE classifiers on our OOD dataset. In our simulated deployment scenario for student essay detection, the BCE classifier’s best false positive rate was 91.41% (85.72% true positive rate), while the DXO classifier’s lowest was just 0.66% (84.16% true positive rate). This performance gap starkly contrasts with the DACTYL student essay subset performance, where the BCE classifier marginally outperformed the DXO classifier. This example emphasizes the limitation of using traditional binary cross-entropy loss. 1.2 Viability of AIG Text Detection We address a common claim about AIG text detectors in this section. AIG text detectors aren’t perfect, and many researchers have strongly cautioned against using AIG text detectors, even suggesting the problem is impossible ( nicks_language_2024 ) . nicks_language_2024 argue that fine-tuning a language model to generate texts to bypass a detector is trivial. When a new