Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News DOI: 10.1145/3795513.3807431 Conference: 18th ACM Web Science Conference; May 26–29, 2026; Braunschweig, Germany 18th ACM Web Science Conference (WebSci Companion ’26), May 26–29, 2026, Braunschweig, Germany ISBN: 979-8-4007-2492-3/2026/05 Note: Accepted at the 18th ACM Web Science Conference (WebSci Companion ’26). DOI: 10.1145/3795513.3807431 CCS: Human-centered computing Human computer interaction (HCI) CCS: Information systems Web mining CCS: Security and privacy Social aspects of security and privacy
Alexander Loth, Martin Kappes, Marc-Oliver Pahl · 2026
Read Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News DOI: 10.1145/3795513.3807431 Conference: 18th ACM Web Science Conference; May 26–29, 2026; Braunschweig, Germany 18th ACM Web Science Conference (WebSci Companion ’26), May 26–29, 2026, Braunschweig, Germany ISBN: 979-8-4007-2492-3/2026/05 Note: Accepted at the 18th ACM Web Science Conference (WebSci Companion ’26). DOI: 10.1145/3795513.3807431 CCS: Human-centered computing Human computer interaction (HCI) CCS: Information systems Web mining CCS: Security and privacy Social aspects of security and privacy by Alexander Loth, Martin Kappes, Marc-Oliver Pahl (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03755.txt.
Introduction
Alexander Loth and colleagues asked whether readers can spot news text written by a language model. Using the JudgeGPT platform, they gathered 2,318 judgments from 1,054 readers across six models, scoring source guesses and authenticity separately. Readers could not reliably separate machine text from human text, and accuracy faded after about 30 judgments in a row.
Catalog record
| CREATOR | Alexander Loth, Martin Kappes, Marc-Oliver Pahl |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15646 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News DOI: 10.1145/3795513.3807431 Conference: 18th ACM Web Science Conference; May 26–29, 2026; Braunschweig, Germany 18th ACM Web Science Conference (WebSci Companion ’26), May 26–29, 2026, Braunschweig, Germany ISBN: 979-8-4007-2492-3/2026/05 Note: Accepted at the 18th ACM Web Science Conference (WebSci Companion ’26). DOI: 10.1145/3795513.3807431 CCS: Human-centered computing Human computer interaction (HCI) CCS: Information systems Web mining CCS: Security and privacy Social aspects of security and privacy Alexander Loth, Martin Kappes, Marc-Oliver Pahl Abstract . Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text ( p > .05 p>.05 , Welch’s t t -test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy ( r = .35 r=.35 , p < .001 p<.001 ) whereas political orientation does not ( r = − .10 r=-.10 , n.s.); (4) clustering reveals distinct response strategies (“Skeptics” vs. “Believers”); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance. † † cc-license: by 1. Introduction Text has long been the web’s default trust carrier, yet LLMs now produce synthetic text that humans rate as no less credible than human-written content ( Kreps et al. 2022 ; Clark et al. 2021 ) . As false and misleading content already spreads faster than corrections online ( Vosoughi et al. 2018 ) , the ability to generate convincing fabrications at marginal cost amplifies what Wardle and Derakhshan term “information disorder” ( Wardle and Derakhshan 2017 ) across an already fragile ecosystem ( Ferrara 2024 ; Lazer et al. 2018 ; Tandoc et al. 2018 ) . Automated detection approaches face a moving-target problem: classifiers trained on one model generation degrade rapidly on the next ( Chen and Shu 2023 ; Su et al. 2024 ) . A complementary line of research therefore focuses on the human side, investigating cognitive inoculation ( Kozyreva et al. 2023 ; Roozenbeek et al. 2022 ) and digital literacy interventions ( Lewandowsky and Van Der Linden 2021 ) . Designing such interventions, however, requires empirical data on how and where human judgment fails. Building on an earlier survey of the research landscape ( Loth et al. 2024 ) , we developed JudgeGPT ( Loth et al. 2026b ) , a web-based study platform with two methodological innovations. First, it decouples authenticity (legitimate vs. fake) from source (human vs. machine) via independent continuous scales, avoiding the conflation inherent in binary real/fake tasks. Second, stimuli are generated by RogueGPT ( Loth et al. 2026c ; Loth et al. 2026b ) , a controlled multi-model framework that enables systematic comparison across LLM families. This poster reports five empirical findings from 2,318 judgments and discusses their implications for web system design. 2. Related Work Research on AI-generated misinformation spans two complementary directions. On the detection side, neural classifiers such as Grover ( Zellers et al. 2019 ) and more recent transformer-based detectors ( Chen and Shu 2023 ) initially achieved high accuracy but degrade as models evolve ( Ferrara 2024 ) ; comprehensive surveys confirm this trend ( Zhou and Zafarani 2020 ) . On the human side, controlled experiments consistently show that participants perform near chance when distinguishing machine-generated from human-written text ( Jakesch et al. 2023 ; Clark et al. 2021 ) . Kreps et al. demonstrated that GPT-generated political messages are rated as credible as journalist-written ones ( Kreps et al. 2022 ) , while Su et al. showed that detection difficulty increases with model scale ( Su et al. 2024 ) . Most human-subject studies, however, employ binary classification tasks (real vs. fake) that conflate source and veracity: a machine-generated paraphrase of a true story is not misinformation, yet binary framing treats it as such. Our dual-axis method addresses this limitation. We additionally build on work in psychological inoculation ( Roozenbeek et al. 2022 ; Kozyreva et al. 2023 ) and analytic thinking as a predictor of misinformation resilience ( Pennycook and Rand 2021 ) , while connecting to emerging provenance standards (C2PA) as system-level alternatives ( Loth et al. 2026e ) . 3. Methodology JudgeGPT is a web-based study platform for collecting human perception data on AI-generated text ( Loth et al. 2026b ) . Its design was informed by a systematic gap analysis of prior work ( Loth et al. 2024 ) . Dual-Axis Assessment. For each news fragment, participants provide three independent ratings on continuous 0–100 sliders (Figure 1 ): (1) Source judgment (0 = certainly machine, 100 = certainly human), (2) Authenticity judgment (0 = certainly fake, 100 = certainly legitimate), and (3) Topic familiarity . Continuous scales were chosen over Likert items to capture degree of certainty and to enable parametric analyses including Pearson correlation and clustering. Figure 1. The JudgeGPT interface. Participants rate each fragment on three continuous axes: source attribution, authenticity, and topic familiarity. Stimulus Design. Stimuli follow a 2\times 2 factorial design crossing origin (human, machine) with veracity (real, fake). Machine-generated fragments are produced by RogueGPT ( Loth et al. 2026c ; Loth et al. 2026b ) , which orchestrates six LLMs (GPT-4 ( Bubeck et al. 2023 ) , GPT-3.5, GPT-4o, LLaMA-2 13B, Gemma 7B, Mistral 7B) with persona-based prompting. Human-sourced fragments are drawn from established news outlets and known misinformation databases. AI-generated texts are grounded in real news topics and verified for factual consistency to avoid conflating perception of style with detection of hallucinations. The stimulus set is intentionally skewed toward machine-origin fragments ( {\sim} 98%), with human-origin items serving as calibration anchors. This design choice reflects the study’s focus on within-AI variation (across models) rather than human-vs.-AI comparison; participants are not informed of the base rate, and the near-chance detection results (Finding 1) hold when analyzed on the human-origin subset alone. Data Collection. Participants provide informed consent and complete a demographic questionnaire (age, education, political orientation, AI familiarity) before evaluating a sequence of fragments. Each participant evaluates between 5 and 87 fragments (median = 12, IQR = 6–22), with presentation order randomized and model assignment balanced across participants. The platform records the three slider ratings, response latency, and an anonymous participant identifier for linking judgments to covariates. The sample skews toward educated European demographics (68% university-educated, 74% European); we discuss generalizability limitations in Section 5 . 4. Findings The dataset comprises 2,318 judgments from 1,054 unique participants. Table 1 provides an overview; each finding is detailed below. Table 1. Summary of empirical findings ( N=2{,}318 judgments from 1{,}054 participants; median 12 items per participant, range 5–87). Finding Statistic Implication F1: Undetectable t(2316)=0.87 , p=.38 Intuition fails as defense F2: Model-agnostic All 6