Identifying AI Web Scrapers Using Canary Tokens DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Privacy protections CCS: Information systems Web log analysis CCS: Security and privacy Web protocol security
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger · 2026
Read Identifying AI Web Scrapers Using Canary Tokens DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Privacy protections CCS: Information systems Web log analysis CCS: Security and privacy Web protocol security by Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13706.txt.
Introduction
Steven Seiden and five colleagues propose identifying AI web scrapers with canary tokens. Existing scraper finding relies on willing company sharing, one-off lab tests, or crowd-sourced reports, none sound or wide. The team hosts dynamic sites serving unique canary tokens to every scraper that visits, then prompts language models about their sites. If a model often outputs tokens unique to one scraper, that shows contact with that scraper. Across 22 live language model systems, the method names which scrapers feed which models, including several not openly known.
Catalog record
| CREATOR | Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80158 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Identifying AI Web Scrapers Using Canary Tokens DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Privacy protections CCS: Information systems Web log analysis CCS: Security and privacy Web protocol security Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger Abstract . From pre-training to query-time augmentation, web-scraped data helps to improve the quality and contextual relevancy of content generated by large language models (LLMs). However, large-scale web scraping to feed LLMs can affect site stability and raise legal, privacy, or ethics concerns. If website owners wish to limit LLM-related web scraping on their site, due to these or other concerns, they may turn to scraper access control mechanisms like the Robots Exclusion Protocol. To be most effective, such mechanisms require site owners to first identify the scrapers that they wish to restrict (e.g., via User-Agent strings). Existing mechanisms to identify LLM-related scrapers rely on voluntary disclosure by companies, one-off experiments by researchers, or crowd-sourced reports — methods that are neither reliable nor scalable. This paper proposes a novel technique for accurately and automatically inferring LLM-related scrapers. We host dynamic websites that serve unique canary tokens to each visiting scraper, then prompt LLMs for information about our sites. If an LLM consistently generates outputs containing tokens unique to a scraper, it provides evidence of exposure to that scraper. Via experiments across 22 production LLM systems, we demonstrate that our approach can reliably identify which scrapers feed which LLM, including several that are not publicly known or disclosed by the companies. Our approach provides a promising avenue for unprivileged third parties to infer which scrapers serve data to which LLMs, potentially enabling better control over unwanted scraping. AI Artificial Intelligence LLM Large Language Model SEO Search Engine Optimization ASN Autonomous System Number RAG Retrieval Augmented Generation SEO Search Engine Optimization 1. Introduction Large language models (LLMs) power popular AI chatbots such as ChatGPT and Claude, among others. To achieve fluency on language generation tasks, AI chatbots are first pre-trained on massive datasets—typically culled from large-scale web scrapes—then undergo post-training “alignment” procedures to stamp out unwanted behaviors. This training process gives models baseline language generation capabilities. While sometimes impressive, these are often insufficient for the wide range of tasks chatbots perform. Even well-trained AI models struggle with hallucinations ( 26 ) , in which they confidently state incorrect responses, and cannot correctly answer questions requiring knowledge after their training data “cutoff date.” To solve this problem, AI chatbot providers have created live content retrieval strategies such as RAG ( RAG ) ( 33 ) or web search ( 16 ; 10 ) . These additional capabilities allow models to supplement their responses with contemporary information from web sources, enhancing their performance on time-dependent queries (e.g. current weather, breaking news, etc) or complex factual questions. Numerous AI chatbot providers advertise web search capabilities (e.g. ( 16 ; 10 ; 2 ; 30 ; 75 ; 67 ; 72 ; 64 ; 68 ; 53 ; 49 ; 3 ; 41 ; 1 ; 5 ; 25 ; 76 ; 23 ; 4 ; 20 ; 17 ; 42 ) ), and consumers have quickly adopted this new technology. Recent studies show significant increases in consumer use of AI chatbots for search, product discovery, and other functions traditionally performed by search engines ( 6 ; 40 ; 62 ) . Today’s leading AI chatbots have tens or even hundreds of millions of daily active users ( 39 ; 48 ) , implying that chatbot’s web search feature is regularly used. Yet, the implementation details of these AI-powered web searches are poorly understood. To our knowledge, only two prior works ( 35 ; 15 ) have investigated the behaviors of scrapers, and both of them focused on live content retrieval. This leaves open questions of how different AI chatbots implement web scraping and whether their public scraping policies match what happens in practice. These questions are both of academic interest and connect to broader issues of AI-related web scraping. Large-scale collection of web data for AI chatbot pre-training has raised legal, privacy, and ethics concerns. In response to these concerns, numerous lawsuits have been filed over the unwanted use of internet-derived data in AI chatbot training ( 70 ; 61 ; 7 ; 27 ; 69 ; 80 ; 71 ; 9 ) . In this same vein, academic work has proposed various tools to disrupt AI-related scraping ( 56 ; 58 ; 59 ) . However, this work focuses on web data used for model pretraining , leaving the issue of unwanted real-time content retrieval unexplored. It remains unclear whether site owners who do not want their sites appearing in AI web searches—for performance, privacy, or other concerns—can opt out, whether via traditional scraper access controls like robots.txt ( 31 ) or otherwise. Our Contribution. Our study provides a critical first look at the infrastructure underpinning real time content retrieval in AI chatbots by identifying which web scraper User-Agents feed content to different chatbots during content retrieval. To do this, we deploy 20 websites that feed differentiated content —which we term canary tokens —to visiting web scrapers. We then elicit information about our websites from 22 different production AI chatbots, recording which canary tokens from these sites are returned by each chatbot. We compare these tokens to those fed to visiting scraper bots, and when enough match, we know with some certainty that a chatbot was exposed to a specific scraper. Figure 1 . High-level overview of how web data is sourced during real-time content retrieval by AI chatbots. Key findings. We use our canary token infrastructure to first map scrapers to chatbots, then explore whether simple blocking mechanisms like taking sites offline or employing a robots.txt file prevent sites from appearing in AI outputs. This experimental approach surfaces the following findings: • AI chatbots often seem to scrape and relay content only fed to search engine scrapers — even sometimes deviating from the official behaviors they self-report. • Many AI chatbots pretend to be regular browser-based visitors when displaying their User-Agent — some even rotating through many generic User-Agent strings. • AI chatbots can respond with data previously cached from websites — stored by either themselves or third-party search engines. • Blocking AI scrapers through robots.txt is largely ineffective at removing website data from an AI chatbot, although it is unclear whether this is due to caching or disregard for the robots.txt protocol. The rest of the paper is organized as follows: Section 2 provides background on how live content retrieval is performed by AI chatbots and outlines key research questions in this space. Section 3 gives an overview of our canary token methodology for matching AI scrapers to chatbots. Section 4 describes our experimental setup. Section 5 discusses results. Section 6 outlines limitations, implications, and future work. 2. Background & Related Work Web scraping has been studied for decades, and myriad academic papers probe the inner workings of massive AI models, yet little work has studied the mechanisms by which web data feeds AI models in real-time content retrieval settings. This section describes prior work studying web data in AI models, real-time content retrieval, and preventing unwanted AI-related scraping. Along the way,