WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl · 2025

Read WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia by Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03373.txt.

Description

Gerrit Quaremba, Elizabeth Black, Denny Vrandečić, and Elena Simperl build a test set for spotting machine-written text on Wikipedia, in a 2025 paper. Wikipedia's value rests on trust, and text quietly generated by language models threatens that trust. Detection tools exist, but they are usually tested on generic writing tasks, not on what Wikipedia editors actually do with these models.

The benchmark, WETBench, covers three real editing jobs: writing a paragraph, summarizing, and restyling text. It spans three languages and several text generators, with three prompts tried per task. Trained detectors scored about 78 percent accuracy on average, while detectors used cold, with no training on the task, managed only about 58 percent. The authors argue that judges of machine text must be tested on the messy, task-specific writing where they will actually serve, or their real reliability stays unknown.

Catalog record

CREATORGerrit Quaremba, Elizabeth Black, Denny Vrandečić, Elena Simperl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03373

Browse the full library · About the Tower of Babel