SEDD: Scalable and Efficient Dataset Deduplication with GPUs

Youngjun Son, Chaewon Kim, Jaejin Lee · 2025

Read SEDD: Scalable and Efficient Dataset Deduplication with GPUs by Youngjun Son, Chaewon Kim, Jaejin Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01046.txt.

Introduction

Youngjun Son, Chaewon Kim, and Jaejin Lee present SEDD, in a 2025 paper, a fast tool for removing duplicate texts from huge datasets. Clean data makes language models train better, but the standard hash-based hunt for near duplicates is slow. New graphics-chip tools still waste effort, shuffling data back and forth and leaving the chips underused.

SEDD replaces the shuffle with a streaming approach that flows data through the chips once. A partly reusable hash function cuts the core cost, and optimized chip code plus automatic parameter selection squeeze the rest. On thirty million texts, it beat a CPU-based tool by a huge factor and the best chip tool by close to eight times. Its duplicate calls stayed close to the standard method. A run over more than a trillion words took three hours on a mid-size chip cluster. The code is free online.

Catalog record

CREATORYoungjun Son, Chaewon Kim, Jaejin Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEDD: Scalable and Efficient Dataset Deduplication with GPUs Youngjun Son, Chaewon Kim, Jaejin Lee Abstract . Dataset deduplication is widely recognized as a crucial preprocessing step that enhances data quality and improves the performance of large language models. A commonly used method for this process is the MinHash Locality-Sensitive Hashing (LSH) algorithm. Recently, GPU-accelerated frameworks such as NVIDIA NeMo Curator have been introduced to handle large-scale corpora; however, they remain suboptimal due to high communication overhead from physical data shuffling and underutilization of GPU resources. In this paper, we propose SEDD, a high-performance GPU-accelerated deduplication framework optimized for distributed cluster environments. SEDD introduces a computationally efficient, partially reusable hash function, alongside highly optimized GPU kernels and a hardware-aware automatic parameter selection mechanism. By replacing traditional data shuffling with a streaming-based approach, SEDD significantly mitigates communication bottlenecks. Our framework outperforms the CPU-based deduplication tool in SlimPajama by up to 158 × \times and the GPU-based tool in NVIDIA NeMo Curator by up to 7.8 × \times when processing 30 million documents on a node with four GPUs. Notably, SEDD dramatically accelerates the previously time-consuming MinHash signature generation phase, achieving speedups of up to 375 × \times over the CPU baseline. Despite these gains in efficiency, SEDD maintains high deduplication fidelity, with duplicate document sets achieving Jaccard similarities of over 0.95 compared to those identified by the standard MinHash algorithm. In large-scale experiments, the deduplication of 1.2 trillion tokens is completed in just 3 hours on an 8-node 32-GPU V100 cluster. The related code is publicly available on GitHub (https://github.com/mcrl/SEDD). 1 Conference: 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea † † cc-license: by 1. Introduction Pretrained language models (PLMs) have demonstrated remarkable performance across a wide range of applications ( Vaswani et al., 2017 ; Raffel et al., 2020a ; Brown et al., 2020a ; Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) , with their capabilities continuing to improve as training datasets scale ( Hoffmann et al., 2022 ; Rae et al., 2021a ) . However, web-scale corpora used for modern LLM training often contain a substantial amount of duplicate and near-duplicate content ( Elazar et al., 2024 ; Magnusson et al., 2023 ) . Such redundancy not only wastes significant computational and storage resources during training but also introduces skewed data distributions due to over-represented documents ( Sorscher et al., 2022 ; Lee et al., 2022 ) . Consequently, large-scale dataset deduplication has become a critical preprocessing step in modern LLM data pipelines. However, scalable near-duplicate detection over trillion-token corpora remains prohibitively expensive with existing approaches. Duplicate detection methods can be broadly categorized into two approaches: exact matching , which identifies identical strings or hash values, and approximate matching , which detects documents with high content similarity ( Albalak et al., 2024 ) . While exact matching is computationally efficient, it fails to capture semantically similar documents with minor variations. In contrast, approximate matching is significantly more computationally intensive. A widely adopted technique for approximate matching is MinHash LSH ( Indyk and Motwani, 1998 ) , which provides a scalable approximation of the Jaccard similarity ( Broder, 1997 ) . Despite its popularity, existing MinHash LSH implementations often struggle to efficiently process datasets at the trillion-token scale due to high computational cost and limited hardware utilization. Recently, NVIDIA introduced NeMo-Curator ( Jennings et al., 2024 ) , a GPU accelerated deduplication pipeline that significantly outperforms traditional CPU-based approaches. However, our analysis shows that its performance remains suboptimal in many practical environments, where excessive communication during data shuffling stalls the pipeline and results in low GPU utilization. These limitations prevent current systems from fully exploiting the massive parallelism offered by modern GPUs. To address these challenges, we propose SEDD, a high-performance GPU-based deduplication framework that maximizes end-to-end processing throughput. By combining computationally efficient hash functions with highly optimized GPU kernels and communication-aware pipeline design, SEDD removes major bottlenecks in existing frameworks, including excessive data shuffling overhead and poor GPU utilization. As a result, our system can deduplicate 1.2 trillion tokens in just 3 hours using an 8-node cluster equipped with 32 V100 GPUs. The contributions of this paper are summarized as follows: • End-to-end GPU deduplication framework. We present SEDD, a framework that achieves 7 \times speedup over NeMo-Curator—the fastest existing GPU-based baseline—without compromising deduplication accuracy. Our design enables efficient processing of trillion-token-scale datasets on standard GPU clusters. • Optimized MinHash LSH computation. We accelerate MinHash generation and comparison by adopting lightweight, reusable hash functions and carefully optimized GPU kernels that improve memory efficiency and parallel utilization. • Communication-efficient data pipeline. We eliminate major data shuffling bottlenecks through a streaming-based execution strategy. Combined with double buffering, SEDD reduces total communication overhead to about 20% of that of existing frameworks. Figure 1 . The process of MinHash generation. RealNews Dataset index: 12347 text: Margins matter. The more Synovis Life Technologies (Nasdaq: SYNO) keeps of each buck it earns in revenue, the more money it has to invest in growth, fund new strategic plans, or (gasp!) distribute to shareholders. Healthy margins often separate pretenders from the best stocks in the market. … (more) index: 39083 text: Margins matter. The more Helen of Troy (Nasdaq: HELE) keeps of each buck it earns in revenue, the more money it has to invest in growth, fund new strategic plans, or (gasp!) distribute to shareholders. Healthy margins often separate pretenders from the best stocks in the market. … (more) Figure 2 . Examples of duplicate documents. 2. Background This section describes the background and related work to SEDD. We examine the impact of data deduplication on the language models. Second, we describe the MinHash LSH algorithm, a standard approach for approximate duplicate detection. Finally, we introduce existing implementations, focusing on the architectural bottlenecks in distributed CPU and GPU-based frameworks that motivate our proposed design. 2.1. Effects of Data Deduplication Data deduplication is the process of identifying and removing redundant entries to ensure data uniqueness, thereby improving both training efficiency and data quality for PLMs. Figure 2 illustrates examples of duplicate documents in the RealNews ( Zellers et al., 2019b ) dataset, where document structures are nearly identical. Such redundancy can hinder the effective learning of language models. Prior studies have shown that duplicated or near-duplicate samples negatively affect model evaluation and generalization. Lee et al. (2022) report that deduplication improves validation perplexity, while Allamanis (2019) demonstrate that duplicated code samples degrade performance on code understanding tasks. Moreover, when near-duplicate documents appear in both training and test sets, evaluation metrics can be artificially inflated, leading to an overestimation of the model’s capability. Removing these overlaps enables a more faithful assessment and reduces overfitting caused by memorization of repeated sequences, ultimately improving generalization ( Lee et al., 2022 ; Tirumala et al., 2023 ; Albalak et al., 2024 ) . In addition to improving model quality, deduplication substantially reduces computational overhead during training. By eliminating redundant content, the model processes fewer repeated tokens, resulting in faster training and lower operational costs without sacrificing—and often improving—model performance ( Sorscher et al., 2022 ) . 2.2. MinHash MinHash ( Broder, 1997 ) is a technique approximating

Read the complete text