Scale Dependent Data Duplication

Joshua Kazdan ∗, Noam Levi ∗, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo †, David Donoho † · 2026

Read Scale Dependent Data Duplication by Joshua Kazdan ∗, Noam Levi ∗, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo †, David Donoho † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06603.txt.

Introduction

Kazdan and team present evidence that data duplication during pretraining harms reach and leads to rote recall, pushing aggressive deduplication. Yet at web scale counts as a duplicate is unclear beyond surface-form matches. In meaning, equivalent documents such as translations can induce repeat training signals once models grow capable, so semantic twins more and more operate like exact twins. Set training on data with reuse sampled from finite unique document pools shows newness mildly hurting small models. Loss costs for larger models rapidly increase, breaking scaling forecast.

Catalog record

CREATORJoshua Kazdan ∗, Noam Levi ∗, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo †, David Donoho †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53384
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06603

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scale Dependent Data Duplication Joshua Kazdan ∗, Noam Levi ∗, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo †, David Donoho † Abstract Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what constitutes a “duplicate”: beyond surface-form matches, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models, by contrast, produce gradients that reflect surface similarity (e.g., shared tokens) rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, the cosine similarity between nearest-neighbors follows an isotropic power law baseline. However, as corpus size grows to hundreds of billions of tokens, the nearest-neighbor similarities deviate sharply, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from pools of finite unique documents shows that limited uniqueness yields mild degradation for small models, but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus. Our results identify and resolve an unstudied source of scale-dependence, allowing for more accurate prediction at scale. 1 1 footnotetext: Equal contribution 2 2 footnotetext: Equal advising 1 Introduction Modern language models scale by increasing parameters, compute, and training tokens. For example, Llama 1 ( Touvron et al. 2023 ) trained on \sim 1T tokens, while the Llama4 herd ( Adcock et al. 2026 ) trained on up to 40T tokens. At these scales, even small fractions of duplicated data can materially reduce the number of distinct training examples and harm downstream performance, emphasizing the importance of deduplication ( Carlini et al. 2021 ; Hernandez et al. 2022 ; Lee et al. 2022 ; Comanici et al. 2025 ) . Deduplication is often framed as a dataset property: to deduplicate, simply remove exact duplicates and near-duplicates using simhashing techniques ( Broder 1997 ; Manku et al. 2007 ; Khan et al. 2025 ; Lee et al. 2022 ) . Yet, what practically counts as a “duplicate” depends on the model as well: two documents that appear distinct may, to a sufficiently capable model, provide redundant training signal, and thus degrade training just as exact duplicates would. This work identifies a previously unknown source of scale dependence: as models become more capable, semantic duplicates induce the same gradients during training. In tandem, capable models are trained on larger corpora, in which the number of semantic collisions rapidly increases. Together, these effects create a recipe for model degradation. Contributions: 1. We quantify the emergence of semantic sensitivity during training by measuring cosine similarity between per-document cross-entropy gradients across a suite of models and semantic-preserving transformations. We find that in more capable models, semantic duplicates induce similar gradients during training. 2. We study semantic collisions by embedding 192M documents from FineWeb-Edu-Dedup ( Penedo et al. 2024 ) documents and analyzing nearest-neighbor (NN) statistics across dataset scales from 10^{4} to 10^{8} documents. We discover that power laws governing scaling for moderate corpus sizes break down for large corpora. This collapse of scaling laws occurs earlier for synthetic corpora, revealing lower semantic diversity. 3. We examine the consequences for predictability by training scaling ladders on streams sampled with replacement from finite pools of K unique documents, showing that limited uniqueness breaks naive scaling extrapolation. We derive more complete scaling laws that explicitly quantify the effects of limited uniqueness, restoring predictability. Furthermore, we show how to estimate an effective K directly from mean nearest-neighbor cosine similarity. We defer related work to Appendix A . Figure 1 : Semantic-preserving transformations yield more aligned gradients for larger/stronger models. We sample N{=}1000 FineWeb-Edu-Dedup documents and compute per-document gradients of normalized next-token cross-entropy (Eq. 2 ) for each model. We report mean cosine similarity between (i) unrelated document pairs (negative baseline) and (ii) each document and its transformed counterpart (positives), including translations and light surface perturbations. Smaller/weaker models exhibit gradient similarity dominated by surface cues (language/casing), often failing to separate positives from negatives. As capability increases, positives become consistently more aligned than the negative baseline. Error bars show per-document standard deviation. Per-model-family results are in Figure 8 . 2 Emergence of Semantics As model capabilities increase, semantically equivalent documents induce similar training signals, as measured by the gradient of the per-document cross-entropy loss. Consequently, if two documents are semantic duplicates (e.g., translations), then a sufficiently capable model will update its parameters in similar directions when trained on both documents. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. 2.1 Experimental Setup We sample N=1000 texts \{x_{i}\}_{i=1}^{N} from FineWeb-Edu-Dedup ( Penedo et al. 2024 ) . To reduce variance due to length, each text is truncated to at most T=2000 tokens using the tokenizer of the model under evaluation. We compute the per-document full-parameter gradient \displaystyle g(x;\theta) \displaystyle=\nabla_{\theta}\ell(x_{i};\theta), (1) where \ell is the mean next-token cross-entropy: \ell(x;\theta)=\frac{1}{|x|}\sum_{u=1}^{|x|}\mathrm{CE}\!\left(f_{\theta}(x)_{u},x_{u+1}\right). (2) To establish a null baseline, we sample unrelated English documents (x_{i},x_{j}),\quad i\neq j and compute cosine similarity \mathrm{sim}(x_{i},x_{j})=\frac{\langle g(x_{i};\theta),g(x_{j};\theta)\rangle}{\|g(x_{i};\theta)\|_{2}\,\|g(x_{j};\theta)\|_{2}}. (3) We repeat this across many random pairings to estimate the baseline mean \mu^{-} and standard deviation \sigma^{-} . Transformations. We construct a set of transformations \mathcal{T}=\{\tau_{1},\dots,\tau_{L}\} intended to preserve semantic content while perturbing surface form: • Swap Characters: with probability 0.05 , randomly replace each ascii character with another. • Drop Words: Randomly delete each word with probability 0.05 . • Capitalize Humps: Capitalize every other character. • Translate to Chinese/French/German. For translations, we use Google’s Translate API ( Google ) . For each document x_{i} and transformation \tau , we compute s_{i}^{+}(\tau)\coloneqq\mathrm{sim}_{\theta}\!\left(x_{i},\;\tau(x_{i})\right). (4) Separability Metrics (Z-scores and AUC). To summarize separation between positives and negatives, we define: \displaystyle z(\tau) \displaystyle\coloneqq\frac{\mu^{+}(\tau)-\mu^{-}}{\sigma^{-}}, (5) \displaystyle\mu^{+}(\tau) \displaystyle\coloneqq\frac{1}{N}\sum_{i=1}^{N}s_{i}^{+}(\tau), (6) \displaystyle\mu^{-} \displaystyle\coloneqq\frac{1}{|\mathcal{S}^{-}|}\sum_{(i,j)\in\mathcal{S}^{-}}\mathrm{sim}_{\theta}\!\left(x_{i},\;x_{j}\right), (7) \displaystyle(\sigma^{-})^{2} \displaystyle\coloneqq\mathrm{Var}_{(i,j)\in\mathcal{S}^{-}}\left[\mathrm{sim}_{\theta}\!\left(x_{i},\;x_{j}\right)\right]. (8) We also report AUC for distinguishing transformed gradients \{g_{\theta}(\tau(x_{i}))\} (positives) from unrelated gradients \{g(x_{j};\theta)\} (negatives) using the score \mathrm{sim}(x_{i},\cdot) . Figure 2 : Semantic sensitivity emerges over training and is accelerated by scale. For a fixed model family, we compute AUC to detect whether a candidate gradient corresponds to a semantic-preserving transformation of the same document versus an unrelated document, with cosine similarity to the original document gradient as the score. Early in training, AUC remains near 0.5 because gradients are dominated by surface-form features (language/casing). With additional optimizer steps, AUC increases, indicating that gradients increasingly reflect semantic content. Larger models reach a given AUC with fewer steps. 2.2 Results Figure 1 reports mean gradient cosine similarities for both unrelated document pairs (negative baseline) and semantic-preserving transformations (positives). For smaller/weaker models, positive similarities for several transformations are comparable to or below the negative baseline, indicating that gradient direction is dominated by superficial features (e.g., language identity or capitalization). As model capability increases, transformed counterparts become consistently more aligned than unrelated pairs. To quantify separability, we compute z(\tau) in Eq. ( 5 ) and AUC for the binary task

Read the complete text