Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

Dongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag · 2025

Read Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs by Dongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06219.txt.

Introduction

Dongyang Fan and six co-authors ask what it costs language models when websites refuse to be crawled, in a 2025 paper. More publishers now block AI crawlers, so training data that respects those blocks is smaller and different.

The team names the cost the data compliance gap: the score difference between models trained on compliant data and models trained without limits. They trained models of 1.5 billion parameters both from scratch and by continuing from a compliant base. As of January 2025, general knowledge barely suffered from compliance. But in narrow fields such as biomedicine, cutting out major publishers did hurt scores. The study gives hard numbers to a long-running argument about data rights and model quality.

Catalog record

CREATORDongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs Dongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag Abstract The increasing adoption of web crawling opt-outs by copyright holders of online content raises critical questions about the impact of data compliance on large language model (LLM) performance. However, little is known about how these restrictions (and the resultant filtering of pretraining datasets) affect the capabilities of models trained using these corpora. In this work, we conceptualize this effect as the data compliance gap (DCG), which quantifies the performance difference between models trained on datasets that comply with web crawling opt-outs, and those that do not. We measure the data compliance gap in two settings: pretraining models from scratch and continual pretraining from existing compliant models (simulating a setting where copyrighted data could be integrated later in pretraining). Our experiments with 1.5B models show that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition (close to 0% DCG). However, in specialized domains such as biomedical research, excluding major publishers leads to performance declines. These findings suggest that while general-purpose LLMs can be trained to perform equally well using fully open data, performance in specialized domains may benefit from access to high-quality copyrighted sources later in training. Our study provides empirical insights into the long-debated trade-off between data compliance and downstream model performance, informing future discussions on AI training practices and policy decisions. Our website is available at https://data-compliance.github.io/ . † † † \dagger denotes equal supervision 1 Introduction The success of Large Language Models (LLMs) is largely dependent on web-scale data. As data becomes an increasingly valuable business asset, companies are treating it as part of their Intellectual Property (IP). However, whether training LLMs on copyrighted data qualifies as fair use remains an open question. In a statement from Library Copyright Alliance (LCA): the ingestion of copyrighted works to create LLMs or other AI training databases generally is a fair use; however, if a work created by AI is substantially similar in protected expression to a work that was ingested by the AI, then it is considered copyright infringement ( LCA, 2023 ) . One of the most notable cases in this debate is the lawsuit filed by The New York Times against OpenAI and Microsoft 1 1 1 nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dkt-1-68-Ex-J.pdf , alleging the unauthorized use of its articles to train GPT models. The accusation is two-fold: first, ChatGPT can generate verbatim excerpts from its copyrighted articles; second, ChatGPT has also hallucinated articles attributed to the Times. This case brings copyright infringement and AI development to the forefront. Amid ongoing legal and ethical debates, AI companies offer an opt-out mechanism for content publishers. Publishers can modify their Robots Exclusion Protocol (REP), commonly known as robots.txt, to block web crawlers at their discretion. Starting from mid 2023, there is a rapidly increasing trend for content owners to apply restrictions on web crawling, as shown in Figure 1 . This results in 5-7% token loss among the major web crawl corpora – C4 ( Raffel et al., 2020 ) , RefinedWeb ( Penedo et al., 2023 ) , and Dolma ( Soldaini et al., 2024 ) – in April 2024. We refer readers to the work from Longpre et al. (2024) for a more detailed analysis. Figure 1 : Timeline of the number of Top 20 filtered (based on robots.txt compliance) domains (excluding theguardian.com 3 3 3 theguardian.com was not accessible via the Internet Archive API ) that block at least one of the web crawlers listed in Appendix B . Robots.txt files were retrieved via the Internet Archive API for each domain on a monthly basis from January 2016 to March 2025. Images indicate the first instance in which a crawler is explicitly blocked by any of the domains. Ensuring legal compliance across the full LLM pipeline—from data preparation to training and inference—begins most effectively at the data preparation stage, by excluding content from websites that have opted out of web crawling. Although largely believed high-quality sources such as news articles are often found in non-permissible data, the impact of excluding major publishers on LLM performance remains poorly understood. This raises a critical question: Can we quantify the effect of content publisher opt-outs? And if so, can we identify the specific knowledge gaps introduced by adhering to robots.txt restrictions? In this work, we make a first step towards answering this question. We show that, as of January 2025, content publishers opting out does not alter the training corpus data distribution much. This results in minimal performance gaps observed by complying with crawling opt-outs. Our contributions are as follows: • We provide a thorough inspection of the change in FineWeb-Edu corpus by respecting web crawling opt-out (Section 3 ). • We introduce the concept of Data Compliance Gap (DCG), which quantifies the performance gap between models trained with and without respect for web crawling opt-outs and propose two methods for measuring it (Section 4 ). • We demonstrate that DCG is close to 0% for general knowledge acquisition, however, compliance gap exists in knowledge of veracity and in structural formats (Section 5.2 ). A noticeable DCG also exists for non-compliant medical domain data (Section 5.3 ). • We show that pretraining in compliance with robots.txt reduces memorization of copyrighted content, though it also limits the acquisition of knowledge derived from that content (Section 5.4 ). 2 Related Works AI and Fair Use. Fair use allows users to utilize copyrighted material without explicit permission under specific conditions. To prevent copyright infringement—especially given the strong memorization abilities of large language models (LLMs)—a variety of methods have been proposed to support fair use across different stages of model development. Prior to model training, data filtering is the most straightforward strategy to limit the use of non-permissible content. For instance, AlphaCode ( Li et al., 2022 ) excluded GitHub code based on license types, while Apple Foundation Models ( Gunter et al., 2024 ) honor websites’ opt-out requests using standard robots.txt directives to prevent crawling by Applebot. During model training, techniques like differentially private training ( Carlini et al., 2021 ; Mattern et al., 2022 ) help prevent the extraction of personal data. Source-aware training ( Khalifa et al., 2024 ) associates source information with each document, enabling instance-level attribution. More recently, approaches like training with Goldfish loss ( Hans et al., 2024 ) have been introduced to effectively reduce verbatim memorization. For models that have already been trained on copyrighted content, post-training techniques such as Reinforcement Learning from Human Feedback (RLHF) and output filtering ( Ippolito et al., 2023 ) have been studied. Another area that is gaining traction for mitigating the generation of privacy-sensitive or copyrighted material in post-training is model unlearning ( Golatkar et al., 2020 ; Chundawat et al., 2023b ; Chundawat et al., 2023a ; Tarun et al., 2023a ; Tarun et al., 2023b ; Shi et al., 2025 ) . Our work falls into the first category, as we respect robots.txt compliance from the data preparation stage. Data Valuation. In the Machine Learning field, data valuation is to understand the impact of data in downstream tasks. Koh

Read the complete text