Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study
Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, Emily Wenger · 2025
Read Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study by Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, Emily Wenger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21733.txt.
Introduction
Taein Kim and co-authors test whether web scrapers obey the robots.txt rules that sites post to limit crawling, in work from 2025. The file sits at a site's root and tells bots which pages they may take. With AI bots now joining older scrapers, the strength of this voluntary system was in doubt.
The team studied logs from their own institution over 40 days, covering 130 self-declared bots and many unnamed ones. They also ran controlled tests that varied the rules. Bots were less likely to obey stricter rules. Some classes of bots, including AI search crawlers, rarely checked the file at all. The authors warn that sites relying on the file alone risk unwanted scraping, and they urge stronger approaches.
Catalog record
| CREATOR | Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, Emily Wenger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, Emily Wenger Abstract . Online data scraping has taken on new dimensions in recent years, as traditional scrapers have been joined by new AI-specific bots. To counteract unwanted scraping, many sites use tools like the Robots Exclusion Protocol (REP), which places a robots.txt file at the site root to dictate scraper behavior. Yet, the efficacy of the REP is not well-understood. Anecdotal evidence suggests some bots comply poorly with it, but no rigorous study exists to support (or refute) this claim. To understand the merits and limits of the REP, we conduct the first large-scale study of web scraper compliance with robots.txt directives using anonymized web logs from our institution. We analyze the behavior of 130 130 self-declared bots (and many anonymous ones) over 40 40 days, using a series of controlled robots.txt experiments. We find that bots are less likely to comply with stricter robots.txt directives, and that certain categories of bots, including AI search crawlers, rarely check robots.txt at all. Our findings suggest that relying on robots.txt to prevent unwanted scraping is risky and highlight the need for alternatives. Conference: Proceedings of the 2025 ACM Internet Measurement Conference; October 28–31, 2025; Madison, WI, USA Proceedings of the 2025 ACM Internet Measurement Conference (IMC ’25), October 28–31, 2025, Madison, WI, USA DOI: 10.1145/3730567.3764471 ISBN: 979-8-4007-1860-1/2025/10 CCS: Information systems Web log analysis CCS: Security and privacy Access control † † cc-license: by 1. Introduction Web scraping, the process of systematically extracting and downloading information from websites, has been a part of the internet ecosystem since its early days ( Gray, 1995 ) . Scraping is now a critical part of many companies’ business models, allowing search engines to optimize results and shopping sites to compare deals, among other purposes. In recent years, scraping has gained new importance as vast troves of internet data are tapped for a novel purpose: training and operating large-scale AI models. Today’s large-scale AI models require terabytes of training data ( Kaplan et al., 2020 ; Lehane, 2025 ) , for which the internet is an obvious and cheap source. Whitepapers documenting many of today’s biggest models, from GPT to Llama and beyond, openly acknowledge the use of web scraping to create training data ( Team et al., 2024 ; Brown, 2020 ; Dubey et al., 2024 ; Touvron et al., 2023 ; Abdin et al., 2024 ; Chowdhery et al., 2023 ) . The AI demand for web scraping extends beyond training data. AI models for text generation have a pesky tendency to hallucinate , confidently stating incorrect information ( Ji et al., 2023 ; Raunak et al., 2021 ) . To make AI-generated text more reliable, AI developers proposed Retrieval-Augmented Generation (RAG), in which large vectorized databases of web content are used to ground AI responses ( Guu et al., 2020 ; Lewis et al., 2020 ; Izacard and Grave, 2020 ) . These databases are collated via large-scale web scraping and, unlike training datasets, are continuously updated. Additionally, AI “agents,” generative models with additional capabilities, can deploy bots to fetch web content as part of their workflow ( Anthropic, 2025 ; Google, 2025a ; AI, 2025 ) . Beyond well-documented copyright and privacy concerns, widespread scraping for any purpose—including for AI training data, RAG, and agent use—can destabilize public, data-rich websites. Numerous writings from the “grey” (non-academic) literature describe sites being taken offline in recent months due to thousands of download requests from scrapers associated with AI companies ( Mehrotra and Marchman, 2024 ; Iaso, 2025 ; Claburn, 2024 ) . These scraping-induced problems have prompted an escalating tactical war, as content-rich websites attempt to moderate or prevent unwanted web scraping ( Belanger, 2025 ; Longpre et al., 2025 ; Longpre, 2025 ) . Among the most widely used anti-scraping solution is the Robots Exclusion Protocol (REP), implemented via a robots.txt file located at the site root. The REP lets site owners set rules that specify which bots can scrape information from the site, which sub-domains those bots can access, and how long bots must wait between successive page accesses ( rep, 2022 ) . Recent work ( Longpre et al., 2025 ) showed a significant uptick in the use of and restrictions in robots.txt files after the rise of generative AI models around 2022, purportedly in reaction to increased scraping for training data. The REP was codified in 2022, but it is not legally binding, so compliance requires scraper goodwill. Although robots.txt is widely used, little hard evidence exists to prove its efficacy in preventing scraping. Some gray literature suggests that many crawlers, including AI-specific bots, do not respect robots.txt ( Claburn, 2024 ) . Other articles claim that certain crawlers ignore robots.txt files altogether ( Mehrotra and Marchman, 2024 ; Belanger, 2025 ) , while still others observe that bots disallowed by robots.txt sometimes pretend to be other, sanctioned bots to circumvent restrictions ( Iaso, 2025 ) . Despite this, a recent work ( Liu et al., 2024 ) conducted a study of 7 crawlers and observed that all AI bots that promised to respect robots.txt did so. Given the dearth of options for deterring unwanted web scraping and widespread belief that robots.txt can help, there is urgent need for a large-scale, controlled experiment evaluating this belief. Such a study will allow content hosts to make informed decisions about whether robots.txt will provide their desired level of protection or if alternative deterrence mechanisms are needed. Our contribution. We conduct the first controlled study of scraper compliance with robots.txt at scale. To do this, we collect anonymized web traffic data from a set of 36 websites we control over 40 days. During this period, we conduct two empirical measurement studies on sites’ robots.txt files to determine if or to what extent web bots, AI and otherwise, comply with robots.txt directives. The first study, conducted on a single site in our dataset with significant web traffic, deployed three versions of a robots.txt file with increasingly strict bot directives, each for two weeks. By analyzing how bot behavior changes in response to different robots.txt directives, we can determine which directives, if any, effectively deter bot behavior and which bots are more or less respectful of robots.txt . The second study, conducted via passive observation of bot behavior on sites with fixed robots.txt files, broadens the perspective on the analysis, determining the frequency with which robots.txt files are accessed. Along the way, we make observations about bot and scraper behavior in general that we believe the community will find interesting. Our key findings are as follows: • Bots are less likely to respect robots.txt that employ strict directives, such as denying access to certain pages. Across our three versions of robots.txt , we see compliance decrease as directives become stricter. • SEO bots are most respectful of robots.txt , while search engine crawlers are among the least. AI-specific bots like AI assistants and AI data scrapers, fall in between. • Observed non-compliance with robots.txt among otherwise respectful bots can sometimes be attributed to spoofing, in which malicious bots present a false user agent to