Testing of detection tools for AI-generated text

Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Gabriel Guerrero-Dib, Olumide Popoola, Petr Šigut, Lorna Waddington · 2023

Read Testing of detection tools for AI-generated text by Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Gabriel Guerrero-Dib, Olumide Popoola, Petr Šigut, Lorna Waddington (2023) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/openalex-w4390175962.txt.

Description

Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, and colleagues tested tools that claim to detect AI-written text. Their study appeared in the International Journal for Educational Integrity in 2023. New language models made machine text easy to produce, and schools rushed to find detectors. The team asked whether the tools can reliably tell human writing from ChatGPT output. They also tested whether translation or deliberate disguise fools the detectors.

Fourteen systems were examined, twelve public tools plus two commercial ones used widely in education. The verdict was blunt: the tools are neither accurate nor reliable. They lean toward calling text human-written, missing much machine output. Disguise techniques made performance far worse. The authors warn against punishing students on such shaky evidence.

Catalog record

CREATORDebora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Gabriel Guerrero-Dib, Olumide Popoola, Petr Šigut, Lorna Waddington
YEAR2023
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEOpenAlex / International Journal for Educational Integrity

Source: https://doi.org/10.1007/s40979-023-00146-z

Browse the full library · About the Tower of Babel