AI-Powered Citation Auditing: A Zero-Assumption Protocol for Systematic Reference Verification in Academic Research

L.J. Janse van Rensburg · 2025

Read AI-Powered Citation Auditing: A Zero-Assumption Protocol for Systematic Reference Verification in Academic Research by L.J. Janse van Rensburg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04683.txt.

Introduction

L. J. Janse van Rensburg, writing alone, presents an AI protocol for auditing academic citations. Studies suggest about twenty percent of citations contain errors, and manual checking takes months. The protocol assumes nothing and verifies every reference on its own.

An agentic system with tool use checks each citation against several databases. Semantic Scholar, Google Scholar, and CrossRef supply the comparison records. Validation covered thirty documents and two thousand five hundred eighty-one references, from student projects to journal papers. Published PLOS papers reached a verification rate near ninety-two percent, with real errors caught along the way.

Catalog record

CREATORL.J. Janse van Rensburg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04683

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI-Powered Citation Auditing: A Zero-Assumption Protocol for Systematic Reference Verification in Academic Research L.J. Janse van Rensburg Abstract Academic citation integrity faces persistent challenges, with research indicating 20% of citations contain errors and manual verification requiring months of expert time. This paper presents a novel AI-powered methodology for systematic, comprehensive reference auditing using agentic AI with tool-use capabilities. We develop a zero-assumption verification protocol that independently validates every reference against multiple academic databases (Semantic Scholar, Google Scholar, CrossRef) without assuming any citation is correct. The methodology was validated across 30 academic documents (2,581 references) spanning undergraduate projects to doctoral theses and peer-reviewed publications. Results demonstrate 91.7% average verification rate on published PLOS papers, with successful detection of fabricated references, retracted articles, orphan citations, and predatory journals. Time efficiency improved dramatically: 90-minute audits for 916-reference doctoral theses versus months of manual review. The system achieved <0.5% false positive rate while identifying critical issues manual review might miss. This work establishes the first validated AI-agent methodology for academic citation integrity, demonstrating practical applicability for supervisors, students, and institutional quality assurance. 1 Introduction Academic citation integrity forms the foundation of scholarly knowledge accumulation. When researchers cite sources accurately, ideas build on verified foundations. However, the citation system faces a persistent crisis: research shows approximately 20% of citations in published papers contain errors [ 5 ] , while 80% of authors cite papers without reading them fully [ 6 ] . These errors propagate through citation networks, creating cascading inaccuracies that undermine scholarly reliability. The root cause is scalability. Manual reference verification demands enormous expert time. Verifying 200 citations in a doctoral thesis can consume weeks or months of supervisor attention—time that could be spent on substantive feedback about research contributions. This creates a practical dilemma: supervisors must either invest unsustainable effort in comprehensive citation checking or accept sampling strategies that miss many errors. Neither option is satisfactory. Recent advances in agentic AI—large language models with tool-use capabilities that can autonomously query databases and synthesize information [ 1 , 2 ] —suggest a potential solution. However, applying AI to citation verification raises fundamental questions. Can AI systems verify academic references with sufficient accuracy for practical deployment? What types of errors can they detect reliably? Can they handle the scale and diversity of real academic documents? This paper addresses these questions by developing and empirically validating an AI-powered citation auditing methodology. Our approach applies agentic AI to a task it is well-suited for: systematic database searching, cross-referencing, and structured reporting. Importantly, this represents the inverse of recent concerns about AI-generated citations [ 3 , 4 ] . Rather than studying AI systems that generate potentially fabricated references, we investigate AI systems that verify references in human-authored work. We make four primary contributions. First, we develop a zero-assumption verification protocol that treats every citation as unverified until independently confirmed across multiple academic databases. This rigorous approach prevents systematic biases and ensures comprehensive coverage. Second, we validate the methodology through two empirical phases encompassing 30 documents and 2,581 references, ranging from undergraduate projects to doctoral theses and peer-reviewed publications. Third, we characterize detection performance, demonstrating 91.7% verification rates on published papers with fewer than 0.5% false positives. Finally, we establish practical deployment viability through evidence of dramatic time savings (90 minutes for 916-reference audits versus months of manual review) and successful identification of critical issues including fabricated references and retracted articles. Our work demonstrates that AI-powered citation auditing addresses a genuine bottleneck in academic quality assurance. While prior research has examined AI systems generating citations, we show that AI can serve as an effective verification tool—shifting the technology from a potential integrity threat to a quality assurance asset. 2 Related Work 2.1 AI Agents and Tool Use Recent advances demonstrate that large language models can perform multi-step reasoning while interfacing with external tools. Chain-of-thought prompting and ReAct [ 1 ] enabled systematic problem decomposition and tool orchestration. These capabilities make AI agents viable for complex auditing tasks requiring database searches, verification across multiple sources, and structured reporting. Our work applies agentic AI specifically to academic reference verification—a task requiring systematic database queries, cross-validation, and quality assessment across heterogeneous sources. 2.2 Citation Integrity Challenges Traditional research documents persistent citation quality issues. Studies show 20% error rates in published citations [ 5 ] , with 80% of authors citing without reading full texts [ 6 ] . Manual verification is prohibitively time-consuming: comprehensive thesis audits require months of expert time, creating supervision bottlenecks. Several types of citation errors plague academic literature: • Metadata errors (wrong year, page numbers, author names) • Orphan references (listed but not cited in text) • Orphan citations (cited but not in reference list) • Content misrepresentation (source cited incorrectly) • Fabricated references (non-existent sources) • Citation of retracted articles Our contribution: We develop automated methodology to systematically detect these errors with high accuracy. 2.3 AI Citation Generation (Inverse Problem) Recent work has studied AI systems generating citations. Walters et al. [ 2 ] found ChatGPT fabricated references in medical Q&A. Alkaissi & McFarlane [ 3 ] documented systematic citation errors in AI-generated content. DeepTRACE [ 4 ] audited AI research assistants’ information retrieval accuracy. These studies examine AI as a citation generator . Our work addresses the inverse: AI as a citation auditor —using AI to verify references in human-authored work. 2.4 AI and Academic Integrity Broader work examines AI’s role in education [ 7 ] and student assignments [ 8 ] . However, these focus on AI assistance for writing, not AI-powered quality assurance. Gap addressed: No prior work has developed and validated AI-agent methodology specifically for comprehensive citation auditing in academic manuscripts across diverse contexts (undergraduate to doctoral level, multiple disciplines). 3 Methodology 3.1 Research Questions RQ1: Can agentic AI with tool-use capabilities reliably verify academic references across diverse document types and disciplines? RQ2: What verification accuracy, false positive rates, and detection capabilities can AI-powered auditing achieve compared to manual review? RQ3: What are the practical implications for time efficiency, scalability, and deployment in academic supervision and quality assurance contexts? 3.2 Zero-Assumption Verification Protocol We developed a rigorous audit methodology that assumes nothing about citation validity: Verification Requirements: A citation is marked "VERIFIED" only when: • Title matches exactly (or with minor punctuation differences) • All authors confirmed (names and order) • Publication year matches • Venue (journal/conference/book) matches • Abstract or key content retrievable and relevant Search Strategy: 1. Semantic Scholar API (primary) 2. Google Scholar (secondary) 3. CrossRef DOI lookup 4. Direct publisher website verification Failure Documentation: Every unverifiable citation requires documented evidence of search attempts and specific reasons for failure. Quality Assessment: For verified sources, we assess: • Journal quality (SCImago Journal Rank) • Source type (peer-reviewed, preprint, grey literature) • Citation appropriateness (does source support claim?) 3.3 Audit Execution To validate our methodology, we conducted a two-phase empirical study across 30 academic documents representing diverse contexts and scales. Phase 1 - Initial Validation (6 documents, 1,369 references): • 1 Honours project (19 references) • 3 Masters dissertations (65-196 references each) • 1 Conference paper (46 references) • 1 Doctoral thesis (916 references) • Purpose : Establish scalability, identify failure modes, refine protocol • Results : 76.8% average verification

Read the complete text