AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus

Sultan Alrashed, Francesco Orabona · 2025

Read AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus by Sultan Alrashed, Francesco Orabona (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18834.txt.

Introduction

Sultan Alrashed and Francesco Orabona turn duplicated web data from waste into a quality stamp. Groups building multilingual training corpora keep crawling the same pages; over 40 percent of tokens in major Arabic corpora appear in more than one source. Their method, MixMinMatch, posted to arXiv in December 2025, treats that overlap as a free filter: text independent pipelines both kept is likely good text.

The recipe mixes existing corpora, runs cross-dataset MinHash deduplication, and keeps documents multiple sources recovered. Applied to Arabic, Turkish, and Hindi, it matches or beats the best single-source corpus while supplying up to four times more unique tokens. Gains reach 4.5 percent over ArabicWeb24 and 5.5 percent over FineWeb-2 for Turkish. The datasets are released publicly, offering cheaper quality for languages the web serves unevenly.

Catalog record

CREATORSultan Alrashed, Francesco Orabona
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus Sultan Alrashed, Francesco Orabona Abstract We present AraMix 1 1 1 https://huggingface.co/datasets/AdaMLLab/AraMix , a deduplicated Arabic pretraining corpus containing approximately 178 billion tokens across 179 million documents. Rather than scraping the web again, AraMix demonstrates that substantial value lies in systematically reusing and curating existing pretraining datasets: we combine seven publicly available Arabic web datasets, apply quality filtering designed specifically for Arabic text to re-filter some datasets, and perform cross-dataset deduplication, both MinHash and sentence-level. This approach reveals that nearly 60% of tokens across these independently collected corpora are duplicates, redundancy that any new scraping efforts will reproduce. Our work suggests that for lower resource languages, investment in curation pipelines for existing data yields greater returns than additional web crawls, an approach that allowed us to curate the largest heavily filtered publicly available Arabic pretraining corpus. 1 Introduction Building pretraining corpora through web scraping requires substantial computational resources, storage infrastructure, and engineering effort. For languages beyond English, multiple research groups have independently undertaken this expensive process, producing overlapping datasets that duplicate both the crawling work and the crawled content itself. This redundancy represents a significant inefficiency: resources spent re-scraping the same web pages could instead be used to improve data quality through better filtering and deduplication. Arabic exemplifies this pattern. Several large-scale Arabic web corpora now exist publicly: CulturaX 8 , ArabicWeb24 5 , HPLT 2.0 3 , FineWeb-2 13 , C4’s Arabic subset 4 , the 101 Billion Arabic Words dataset 1 , and FinePDFs 7 . These datasets are typically used in isolation, and their quality filters, designed primarily for English, perform poorly on Arabic text. For example, C4’s line-ending filter uses hardcoded English punctuation, rejecting most Arabic lines. Instead, Gopher’s 14 stop word requirements reject most Arabic documents outright. Hence, we believe that opportunities lie not in scraping more data, but in properly combining what already exists. Figure 1: Pairwise token overlap between source datasets (billions). Our AraMix dataset takes this approach. We aggregate all seven public Arabic corpora and process them through a unified pipeline with Arabic-appropriate quality filtering followed by cross-dataset deduplication using datatrove 11 . Of 442 billion input tokens from all datasets, only 178 billion remain after MinHash deduplication, that is, nearly 60% of the content across these independently collected datasets was redundant. Figure 1 shows the pairwise token overlap between sources. C4 and CulturaX share 91.9% of content. Documents appearing in four or more sources account for 8.5 billion tokens, as seen in Table 5 . This concentration reflects shared upstream dependency on the Common Crawl. In fact, the datasets derive from the same web snapshots, processed through different pipelines. This overlap demonstrates that additional scraping efforts would largely recover documents already present in existing corpora. It also demonstrates the value in extracting from novel sources, as we see with FinePDFs data almost completely surviving deduplication in Table 6 . 2 Related Work Arabic Pretraining Corpora. Several large-scale Arabic pretraining corpora have been released in recent years. ArabicWeb24 scraped Arabic web content directly, producing 40 billion tokens after language identification and basic filtering. The 101 Billion Arabic Words dataset aggregated news, Wikipedia, and web sources. CulturaX and HPLT 2.0 extracted Arabic subsets from Common Crawl using multilingual pipelines, while FineWeb-2 extended FineWeb’s 10 filtering approach to non-English languages. These corpora were created independently, each project investing resources in crawling and filtering without knowledge of what others had already collected. Our AraMix dataset takes a different approach: Rather than scraping again, we combine all seven existing corpora and measure their overlap, finding that 60% of tokens are redundant across sources. Corpus Aggregation and Deduplication. Prior work has combined and deduplicated existing corpora, though typically within narrower scope. CulturaX merged mC4 and OSCAR with additional cleaning but did not measure cross-source redundancy. SlimPajama 15 applied MinHash deduplication to RedPajama, removing 49% of content, demonstrating substantial within-corpus duplication. The FineWeb effort 10 analyzed overlap between their corpus and RefinedWeb 12 , finding significant shared content from Common Crawl’s upstream data. 16 showed that deduplication combined with diversity-based selection improves training efficiency by 20%. All these efforts focus on English corpora or single-source deduplication. AraMix extends cross-dataset deduplication to seven independently collected Arabic corpora, quantifying pairwise overlaps and identifying content that appears in four or more sources. Language-Specific Quality Filtering. Quality filters developed for English corpora perform poorly on morphologically rich or non-Latin-script languages. C4’s pipeline rejects lines not ending in English terminal punctuation, filtering out most Arabic text. The Gopher filters require English stop words, rejecting Arabic documents entirely. CCNet addressed multilingual extraction using perplexity-based filtering against Wikipedia language models, enabling corpus construction for over 100 languages without language-specific rules. FineWeb-2 adapted thresholds for non-English text but retained structurally English-centric heuristics. Work on Arabic-specific NLP tools 2 ; 9 has focused on downstream tasks rather than pretraining corpus construction. AraMix introduces filters designed for Arabic text characteristics: accepting documents with zero terminal punctuation (common in Arabic web text), using Arabic punctuation characters for line-level filtering, and tuning repetition thresholds for Arabic’s orthographic patterns. 3 AraMix Corpus Table 1: Token counts across filtering stages: before filtering, after document-level MinHash deduplication, and after sentence-level deduplication. Tokens counted using meta-llama/Llama-3.2-3B ’s tokenizer 6 . Source Tokens (Before) Tokens (MinHash) Tokens (Sent-Dedup) CulturaX 87.4B 42.1B 38.4B ArabicWeb24 40.7B 35.4B 31.6B HPLT 2.0 108.4B 34.7B 30.4B FineWeb-2 67.2B 27.5B 24.2B C4 59.2B 22.5B 20.4B 101B / ClusterLab 49.9B 9.5B 7.7B FinePDFs 29.7B 6.3B 6.1B Total 442.5B 177.8B 158.8B AraMix draws from seven publicly available Arabic datasets on HuggingFace, as seen in table 1 . Figure 2: Number of documents left over after MinHash deduplication and sentece-level deduplication. The sources exhibit different characteristics: FinePDFs contributes fewer documents but with high average length (9,337 tokens/doc), while FineWeb-2 and C4 tend toward shorter documents (857 and 837 tokens/doc respectively). When counting the number of words in the 101 Billion Arabic Words dataset, we only find approximately 22.6B words when downloading the dataset from Huggingface, which leads to our token count. The paper mentions that that they have 116 million rows of data, we only see 33 million. Moreover, in their Huggingface repository we found a commit that indicates that the team might not have uploaded the dataset in full. The pipeline consists of three stages: quality filtering, document-level MinHash deduplication, and sentence-level deduplication. Starting from 394.6 million raw documents, the pipeline produces 167.6 million final documents after sentence-level deduplication, corresponding to a 57.5% overall reduction. In the following, we describe each of these steps. Table 2: Document counts at each pipeline stage. Stage Input Output Reduction Quality Filtering 394,620,199 331,747,344 15.9% MinHash Deduplication 331,747,344 178,883,241 46.1% Sentence-Level Deduplication 178,883,241 167,571,677 6.3% 4 Quality Filtering Standard quality filters designed for English corpora perform poorly on Arabic text. C4’s line-ending filter uses hardcoded English punctuation ( .?!"’ ), rejecting most Arabic lines. Gopher requires English stop words, rejecting all Arabic documents. FineWeb’s 12% punctuation threshold 10 is too strict for Arabic web text, which tends to lack punctuation. We developed a custom Arabic quality filter addressing these issues. The key thresholds we used for filtering: • Terminal punctuation : \geq 5% of lines

Read the complete text