WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia
Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl · 2025
Read WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia by Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03373.txt.
Description
Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, and Elena Simperl build a test set for spotting machine-written text on Wikipedia, in a 2025 paper. Wikipedia's value rests on trust, and text quietly generated by language models threatens that trust. Detection tools exist, but they are usually tested on generic writing tasks, not on what Wikipedia editors actually do with these models.
The benchmark, WETBench, covers three real editing jobs: writing a paragraph, summarizing, and restyling text. It spans three languages and several text generators, with three prompts tried per task. Trained detectors scored about 78 percent accuracy on average, while detectors used cold, with no training on the task, managed only about 58 percent. The authors argue that judges of machine text must be tested on the messy, task-specific writing where they will actually serve, or their real reliability stays unknown.
Catalog record
| CREATOR | Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03373