The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers
H. Kemal İlter · 2026
Read The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers by H. Kemal İlter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17431.txt.
Introduction
H. Kemal İlter audits how AI help can decay the facts in survey papers. Language models make writing faster. They can also add informational entropy, a slow blur of truth. İlter ran a forensic audit of 50 recent AI survey papers. The set held 5,514 citations from late 2024 to early 2026. He tracked how valid citation chains degrade.
The title names a 17 percent gap. That figure sums up the decay he found. Hallucinated papers are a known flaw; broken chains are quieter. They let false claims look sourced and sound. The audit gives editors a scale for the harm. It argues for citation checks as a core step, not an extra.
Catalog record
| CREATOR | H. Kemal İlter |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17431
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers H. Kemal İlter Abstract Background: The adoption of Large Language Models (LLMs) in scientific writing promises efficiency but risks introducing informational entropy. While “hallucinated papers” are a known artifact, the systematic degradation of valid citation chains remains unquantified. Methodology: We conducted a forensic audit of 50 recent survey papers in Artificial Intelligence ( N = 5,514 N=5{,}514 citations) published between September 2024 and January 2026. We utilized a hybrid verification pipeline combining DOI resolution, Crossref metadata analysis, Semantic Scholar queries, and fuzzy text matching to distinguish between formatting errors (“Sloppiness”) and verifiable non-existence (“Phantoms”). Results: We detect a persistent 17.0% Phantom Rate —citations that cannot be resolved to any digital object despite aggressive forensic recovery. Diagnostic categorization reveals three distinct failure modes: pure hallucinations (5.1%), hallucinated identifiers with valid titles (16.4%), and parsing-induced matching failures (78.5%). Longitudinal analysis reveals a flat trend (+0.07 pp/month), suggesting that high-entropy citation practices have stabilized as an endemic feature of the field. Conclusion: The scientific citation graph in AI survey literature exhibits “link rot” at scale. This suggests a mechanism where AI tools act as “lazy research assistants,” retrieving correct titles but hallucinating metadata, thereby severing the digital chain of custody required for reproducible science. Keywords: citation analysis, hallucination, large language models, scientometrics, reproducibility 1 Introduction The architecture of modern science relies entirely on the chain of custody. When a scholar asserts a claim, the citation serves as the forensic link to the evidence, allowing the community to verify, replicate, and build upon prior work ( 12 ) . For centuries, this ledger was maintained manually. However, the production of scientific synthesis—particularly within the hyper-active field of Artificial Intelligence—has accelerated beyond the unassisted human capacity for review ( 4 ) . To cope with the deluge, the community has quietly outsourced the labor of literature review to Large Language Models (LLMs). This transition has introduced a new form of epistemic risk. While earlier critiques of generative AI focused on “hallucinations”—the fabrication of non-existent papers ( 10 ; 1 ) —a more insidious error mode has emerged. We hypothesize that current AI tools function as “lazy research assistants.” They correctly identify real, seminal titles to maintain semantic coherence, but they hallucinate the bureaucratic metadata required to locate them. They guess DOIs. They fabricate volume numbers. They invent page ranges that look statistically plausible but are functionally dead. The result is a scientific graph that looks robust on the surface but is rapidly rotting underneath. Existing literature has treated citation errors as transient “noise” that will vanish as models scale ( 5 ) . We argue the opposite. The error is not transient; it is structural. Without a quantification of this decay, we risk building a discipline on a foundation of broken links, where the appearance of scholarship outpaces the verifiability of truth. This phenomenon echoes Muller’s Ratchet in evolutionary biology—the irreversible accumulation of deleterious mutations in asexual populations ( 13 ) . Once a phantom citation enters the literature and is subsequently cited by others, the error propagates irreversibly through the citation network. In this study, we conduct a forensic audit of the AI survey literature published between September 2024 and January 2026. We analyzed 50 survey papers containing 5,514 distinct citations, subjecting each to a hybrid verification pipeline of DOI resolution, API-based metadata retrieval, and fuzzy text matching. We move beyond simple error counting to distinguish between recoverable “sloppiness” and irrecoverable “phantoms,” and we further categorize phantoms into three diagnostic failure modes. Our analysis reveals a persistent 17.0% Phantom Rate . This is not a random fluctuation. It is an equilibrium of decay. For nearly one in five citations, the digital chain of custody is severed. This paper quantifies the extent of this entropy and argues that without new verification standards, the AI literature risks entering a state of permanent reference rot. 2 Related Work 2.1 LLM Hallucination in Academic Contexts The phenomenon of LLM hallucination—generating plausible but factually incorrect content—has been extensively documented ( 10 ) . In academic contexts, this manifests as fabricated citations, a problem first highlighted by 1 who found that ChatGPT generated non-existent references when asked to produce academic content. Subsequent studies have confirmed this behavior across multiple LLM architectures ( 3 ; 2 ) . 2.2 Citation Analysis and Link Rot The integrity of academic citation networks has long been a concern in scientometrics. 9 documented the prevalence of “citation amnesia”—the tendency for older works to be forgotten despite their foundational importance. More recently, 11 quantified “reference rot” in scholarly literature, finding that significant portions of web-based references become inaccessible over time. Our work extends this analysis to AI-generated citations specifically. 2.3 The DOI System and Metadata Integrity The Digital Object Identifier (DOI) system was designed to provide persistent identification for digital content ( 14 ) . The Crossref infrastructure maintains metadata for over 150 million scholarly works, enabling programmatic verification ( 8 ) . Our methodology leverages this infrastructure to distinguish between valid references and hallucinated identifiers. 3 Methodology To quantify the degradation of the citation graph, we deployed a forensic auditing framework designed to distinguish between benign formatting errors and genuine informational entropy. We did not merely check if a link worked. We attempted to recover the intended target. Our protocol follows a “presumption of existence” approach: a citation was only classified as a Phantom after multiple recovery mechanisms failed. 3.1 Data Selection and Corpus Construction We focused our analysis on the “Survey Paper” genre within Artificial Intelligence. Survey papers are high-density vectors for citation propagation; a single hallucinatory error in a widely cited survey can contaminate the literature for years—a phenomenon we term the “Muller’s Ratchet” of citation decay. We queried the arXiv repository using the search string: ti:"Survey" AND (ti:"Large Language Models" OR ti:"Generative AI") We selected 50 review articles published between September 2024 and January 2026. Selection criteria prioritized high-volume citation lists (mean citations per paper \bar{n}=110.3 , \sigma=89.2 ) from pre-print repositories (arXiv: cs.CL, cs.LG, cs.AI). The final corpus contained N=5{,}514 unique citations. 3.2 The Forensic Verification Pipeline Each citation underwent a multi-stage verification process utilizing the Crossref API and Semantic Scholar API. The pipeline implements a priority queue of verification methods, from high-confidence exact matches to probabilistic fuzzy matching. 3.2.1 Stage 1: Identifier Extraction We extracted Digital Object Identifiers (DOIs) and arXiv IDs from citation strings using regular expression patterns: DOI Pattern: 10\.\d{4,9}/[-._();/:A-Z0-9]+ (1) arXiv Pattern: \d{4}\.\d{4,5}(v\d+)? (2) 3.2.2 Stage 2: Direct Resolution For citations with extracted identifiers, we performed exact-match verification: • DOIs were resolved via https://doi.org/{doi} with HTTP status code verification • arXiv IDs were verified against the arXiv API • A successful HTTP 200 response confirmed the citation as Valid with similarity score s=100\% 3.2.3 Stage 3: Entropy Filter Before fuzzy matching, we applied an entropy filter to detect PDF extraction artifacts. Many citation extraction failures produce corrupted strings where whitespace is lost (e.g., “ProbingClassifiersPromisesShortcomings…”). Definition 1 (Space Ratio Entropy Filter) . Let c be a citation string of length |c| , and let \text{spaces}(c) denote the count of space characters. The space ratio is defined as: \rho(c)=\frac{\text{spaces}(c)}{|c|} (3) A citation passes the entropy filter if and only