Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador · 2026

Read Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy by Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23146.txt.

Introduction

Pudasaini and co-authors ask why AI text detectors fail outside their benchmarks. Many detectors score high on test sets yet prove shaky in real use. Their framework blends linguistic features, machine learning, and explainable AI. Trained on 30 language features, it matches top leaderboard scores on two corpora. Cross-domain and cross-generator tests then show sharp drops. Explanations reveal the key features differ by dataset. Detectors lean on dataset style cues, not stable signs of machine writing. The team releases an open tool with per-text explanations.

Catalog record

CREATORShushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58651
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23146

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador Abstract The widespread adoption of Large Language Models (LLMs) has made the detection of AI-generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. Evaluated across two major benchmark corpora—PAN-CLEF 2025 and COLING 2025—models trained on 38 linguistic features achieve leaderboard-competitive performance, attaining an F1 score of 0.9734 without reliance on large-scale language models. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP-based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigating this, we perform an in-depth error analysis, applying SHAP to False Negatives and False Positives from the model. Our findings show that benchmark accuracy is not reliable evidence of authorship detection: strong in-domain performance can coincide with substantial failure under domain and generator shift. Our in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps in building AI detectors that are robust under different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts. Project Page: https://shushantatud.github.io/ExplainAIGeneratedText/ K eywords Large Language Models, AI-Generated Text Detection, Machine Learning, Explainable AI, Academic Integrity 1 Introduction The rapid adoption of Large Language Models (LLMs) has made the distinction between human-written and AI-generated text a central concern in education, publishing, and online communication. Automated detectors are increasingly proposed as a solution, and recent shared tasks report very high accuracy on benchmark datasets. However, despite these results, the reliability of AI-generated text detection in real-world settings remains uncertain. In high-stakes contexts such as academic assessment, the critical question is not whether detectors can perform well on benchmarks, but whether they genuinely identify machine authorship. A key unresolved issue is whether modern detectors learn universal characteristics of AI-generated language or instead exploit dataset-specific artefacts. Benchmark corpora differ substantially in domain, genre, writing conditions, and generator models. Consequently, high in-domain performance may reflect the ability to distinguish particular datasets rather than to detect AI authorship itself. If this is the case, reported accuracy may overestimate real-world capability, especially when detectors are applied to unseen domains or newly released language models. From a measurement perspective, an AI-text detector is an instrument that claims to measure a latent construct: machine authorship . High benchmark scores demonstrate only that a model separates the labels within a specific dataset; they do not, by themselves, establish that the detector is measuring authorship rather than confounded properties of dataset construction (e.g., topic, genre, prompt style, length, or collection pipeline). We therefore treat detector validity as a first-class outcome: a detector is valid only if the evidence it uses remains stable under domain and generator shift and aligns with plausible linguistic signals of authorship rather than corpus-specific cues. Educational settings illustrate the importance of this problem. Surveys indicate that a large proportion of students regularly use generative AI tools such as ChatGPT, Grammarly, and Microsoft Copilot [ 8 ] . While such tools can support learning, they also enable the generation of complete assignments with minimal human effort. Institutions, therefore, face increasing pressure to ensure fair assessment and academic integrity. At the same time, unreliable detection systems risk unjust decisions and reduced trust in automated evaluation. Existing technical approaches to AI-generated text detection include watermarking-based methods, zero-shot detectors, and supervised classifiers [ 10 ] . These approaches are commonly evaluated in shared tasks such as SemEval 2024 [ 7 ] , PAN@CLEF 2025 [ 2 ] , and COLING 2025 [ 1 ] . However, their validity remains contested [ 30 ] . Watermarking methods are vulnerable to paraphrasing and theoretical limitations [ 38 ] , while many high-performing detectors rely on opaque large models that provide little justification for their decisions [ 16 ] . This lack of transparency is particularly problematic in high-stakes applications. Linguistic-feature-based approaches offer an alternative by grounding predictions in interpretable properties of text. Prior work has identified stylometric and linguistic signals that distinguish human and machine writing within individual datasets [ 25 , 33 ] . However, it remains unclear whether these signals generalise across domains and generators, and most studies report aggregate feature importance without explaining individual predictions. To address this gap, we combine linguistic features with machine learning algorithms and explainable AI techniques to investigate two central research questions. First, we ask which linguistic features most effectively distinguish AI-generated text from human-written text across diverse domains and generators. Second, we examine whether machine learning models trained on these features genuinely capture the characteristics of machine authorship or merely learn dataset-specific artefacts, evaluating this through systematic cross-domain and cross-generator testing. This paper investigates the reliability of AI-generated text detection rather than only its accuracy. We test the hypothesis that many detectors rely on dataset-specific stylistic cues instead of stable indicators of machine authorship. To examine this, we introduce an interpretable detection framework combining linguistic feature engineering, classical machine learning, and explainable AI. We evaluate models across two major benchmark datasets (PAN CLEF and COLING) under in-domain, cross-domain, and cross-generator conditions, and analyse their behaviour using SHAP (SHapley Additive exPlanations). Our central claim is that explainability can be used as a diagnostic tool. By analysing feature contributions for individual predictions, we identify the linguistic patterns actually used by the models and determine whether they represent generalisable authorship signals or dataset artefacts. This enables direct examination of why detectors succeed on benchmarks yet fail under domain and generator shift. The primary contribution of this work is an empirical validity analysis of AI-generated text detection. We show that models can achieve near state-of-the-art benchmark performance while failing under cross-domain and cross-generator evaluation, and we use instance-level explanations to diagnose why this happens. Specifically, we demonstrate that the features driving predictions differ markedly across datasets, indicating reliance on corpus-specific stylistic cues rather than stable signatures of machine authorship. As a secondary contribution, we provide an interpretable feature-based framework and an open-source implementation that returns both predictions and instance-level explanations to support reproducible evaluation and responsible use in academic integrity workflows. For reproducibility purposes, we made our code available on GitHub. 2 Literature Review Research on AI-generated text detection has grown rapidly alongside the deployment of large language models. Prior work broadly addresses four aspects of the problem: detection methodologies, linguistic feature-based approaches, robustness and generalisation, and explainability. Taken together, these strands reveal a central unresolved issue: although detectors often achieve high benchmark accuracy, it remains unclear whether they identify machine authorship itself or instead exploit dataset-specific characteristics. 2.1 Approaches

Read the complete text