Humans can learn to detect AI-generated texts, or at least learn when they can’t
Jiří Milička · 2025
Read Humans can learn to detect AI-generated texts, or at least learn when they can’t by Jiří Milička (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01877.txt.
Introduction
Milička and co-authors test, in a 2025 study, whether people can learn to tell machine-written text from human writing. The team generated hundreds of texts with GPT-4o across many genres and paired them with human texts from a matched Czech corpus. Then 254 native Czech speakers judged each pair, saying which text was human and which was machine made.
Half the group got the right answer after every trial; the other half got no feedback until the end. The feedback group grew more accurate and judged their own skill more honestly. Without feedback, people erred most on the very trials where they felt most sure. Many began with wrong hunches about machine style and readability. The authors argue that short training with feedback can teach this skill, a point they flag for schools.
Catalog record
| CREATOR | Jiří Milička |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Humans can learn to detect AI-generated texts, or at least learn when they can’t Jiří Milička Abstract Abstract This study investigates whether individuals can learn to accurately discriminate between human-written and AI-produced texts when provided with immediate feedback, and if they can use this feedback to recalibrate their self-perceived competence. We also explore the specific criteria individuals rely upon when making these decisions, focusing on textual style and perceived readability. We used GPT-4o to generate several hundred texts across various genres and text types comparable to Koditex, a multi-register corpus of human-written texts. We then presented randomized text pairs to 254 Czech native speakers who identified which text was human-written and which was AI-generated. Participants were randomly assigned to two conditions: one receiving immediate feedback after each trial, the other receiving no feedback until experiment completion. We recorded accuracy in identification, confidence levels, response times, and judgments about text readability along with demographic data and participants’ engagement with AI technologies prior to the experiment. Participants receiving immediate feedback showed significant improvement in accuracy and confidence calibration. Participants initially held incorrect assumptions about AI-generated text features, including expectations about stylistic rigidity and readability. Notably, without feedback, participants made the most errors precisely when feeling most confident — an issue largely resolved among the feedback group. The ability to differentiate between human and AI-generated texts can be effectively learned through targeted training with explicit feedback, which helps correct misconceptions about AI stylistic features and readability, as well as potential other variables that were not explored, while facilitating more accurate self-assessment. This finding might be particularly important in educational contexts, since the ability to identify AI-generated content is highly desirable and, more importantly, false confidence in this domain can be harmful. 1 Introduction In most everyday situations, individuals typically have some idea about their abilities, even if these ideas are imprecise and correlate only loosely with actual competence [ 1 , 2 ] . For example, people who have never learned to swim are usually aware of this inability and understand that swimming can be learned. This awareness arises from experiences involving informal or formal testing, or from social interactions where they observe the difficulty of acquiring a skill. Such self-assessment capacities, however, tend to fail when confronted with entirely novel phenomena, such as interacting with artificial intelligence (AI). This is primarily because there are no standardized assessments, institutional oversight, or established instructional methods for these emerging skills, leaving intuition without any reliable anchor. We may reasonably assume that most people have never undertaken a rigorous test to assess their ability to discriminate between human-written texts and texts generated by frontier language models. In the best case, individuals are agnostic about their skill levels; in the worst case, they hold unrealistic expectations that remain uncorrected. Consequently, many individuals neither perceive the need to learn this skill nor, even if they did, would they find clear guidance in the literature about whether or how such learning could be successful. False confidence in this domain can be particularly problematic in educational contexts — for example, when a teacher is overly confident in identifying AI-generated texts and unjustly accuses students of academic dishonesty. Our current study contributes to the extensive body of literature examining the conditions under which people can discern AI-generated texts from human-written ones. However, our central research question shifts focus from mere discriminatory ability to the capacity of individuals to utilize feedback effectively: Can individuals learn to accurately discriminate between human-written and AI-produced texts when provided with immediate feedback? Furthermore, are they capable of using this feedback to recalibrate their self-perceived competence? Beyond these core questions, our research also explores the specific criteria individuals rely upon when distinguishing between human and AI-generated texts, identifying factors that aid or hinder accurate judgments. We also aim to identify the factors that are influenced by the learning process due to the feedback loop. We specifically focus on textual style (register), hypothesizing that people attribute particular stylistic characteristics to AI-generated texts. These assumptions might help if aligned with reality but could also lead to errors if based solely on prejudice. For similar reasons, we investigate perceived readability, analyzing how participants associate readability with text authorship. Additionally, we collected data on demographic variables (age, gender, education) and participants’ engagement with AI technologies, including usage frequency and general attitudes. For our experimental design, GPT-4o generated a corpus comprising 672 texts across various genres and text types, chosen specifically to reflect stylistic diversity comparable to the Koditex corpus, a traditional human-authored collection designed explicitly for genre and register richness [ 3 ] . These texts were presented pairwise to participants (each receiving 17 randomly selected text pairs plus 3 control items). We recorded not only their accuracy in identifying which text was AI-generated versus human-written but also their confidence levels, response times, and judgments about text readability. Participants (n=254) were randomly assigned to two experimental conditions. One group received immediate feedback after each trial, indicating correctness, whereas the second group received no immediate feedback and learned their overall results only upon completing the experiment. All texts and participants were Czech native speakers, reflecting our intention to study a medium-sized language, as English disproportionately dominates the training datasets of contemporary language models, making it unrepresentative of other languages. Nevertheless, all data, experimental software, analytical scripts, and detailed protocols are publicly available, ensuring that the study can be easily replicated in other linguistic contexts. 1.1 State of the art The question of whether humans can detect AI-generated texts has attracted significant scientific attention, but experimental research with human participants has shown mixed results. As the studies are incomparable in almost every aspect, we can only guess whether the differences in findings can perharps be attributed to the purpose of the study, selected text genres, differing populations tested, mode of presentation, language, or AI models used. To date, studies have reported varying overall accuracy in participants’ recognition of AI-generated texts. [ 4 ] found that English-speaking participants performed slightly better than chance and attested substantial variation based on individual abilities or expertise. In other studies [ 5 , 6 , 7 ] , however, human evaluators’ accuracy was rather at chance levels. Studies exploring how succesfully can people detect or distinguish AI-generated texts typically focus on only one genre, which corresponds to the general framing of the study (e.g., concern for a given field, such as cheating in education), or comparisons of distinct genres. Research has been done on poetry [ 8 ] , EFL students’ essays [ 6 , 9 ] , scientific abstracts [ 10 ] , news articles, recipes and short stories [ 5 ] , social media posts [ 11 ] , or job applications, online dating and AirBnB host profiles [ 12 ] . However, no previous studies have systematically addressed text variability as a continuum across registers, despite some research indicating that certain linguistic features related to text variability may influence how accurately people distinguish between AI- and human-generated texts. [ 4 ] applied the Linguistic Inquiry and Word Count 2022 toolbox to measure word-level characteristics of the text but did not find any significant impact. [ 12 ] used both computational methods and humans to annotate