The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora
2026
Read The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11170.txt.
Introduction
Web crawling for South Slavic tongues is the topic of this 2026 preprint. The authors are not named in the record held here. Their team built the CLASSLA-web text sets by crawling whole country domains. That method works well for tongues with few digital texts.
An early round of crawling gave the largest sets yet for this group. Those sets were called CLASSLA-web 1.0 in the first release. The new work looks at a fresh round of crawling, named 2.0. It weighs what repeat crawls add in new text. It also weighs the costs and flaws that come with each new round.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora Abstract Crawling national top-level domains has proven to be highly effective for collecting texts in less-resourced languages. This approach has been recently used for South Slavic languages and resulted in the largest general corpora for this language group: the CLASSLA-web 1.0 corpora. Building on this success, we established a continuous crawling infrastructure for iterative national top-level domain crawling across South Slavic and related webs. We present the first outcome of this crawling infrastructure – the CLASSLA-web 2.0 corpus collection, with substantially larger web corpora containing 17.0 billion words in 38.1 million texts in seven languages: Bosnian, Bulgarian, Croatian, Macedonian, Montenegrin, Serbian, and Slovenian. In addition to genre categories, the new version is also automatically annotated with topic labels. Comparing CLASSLA-web 2.0 with its predecessor reveals that only one-fifth of the texts overlap, showing that re-crawling after just two years yields largely new content. However, while the new web crawls bring growing gains, we also notice growing pains – a manual inspection of top domains reveals a visible degradation of web content, as machine-generated sites now contribute a significant portion of texts. Keywords: web corpora, South Slavic languages, genre corpora, topic classification, web, crawling The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora Taja Kuzman Pungeršek ∗ , Peter Rupnik ∗ , Vít Suchomel § , Nikola Ljubešić ∗ † ‡ ∗ Jožef Stefan Institute; † Faculty of Computer and Information Science, University of Ljubljana; ‡ Institute of Contemporary History; § Natural Language Processing Centre, Masaryk University ∗ † ‡ Ljubljana, Slovenia; § Brno, Czech Republic {taja.kuzman, peter.rupnik, nikola.ljubesic}@ijs.si; § [email protected] Abstract content 1. Introduction Figure 1: The CLASSLA-web construction pipeline (detailed in Section 3 ), consisting of several key steps: web crawling, language identification using multiple tools, post-processing to ensure high-quality corpus content, automatic genre and topic annotation, and linguistic annotation. Building large, high-quality corpora for less-resourced languages remains a central challenge for the natural language processing field. Crawling top-level domains (TLD) has been shown to be a very efficient method to collect large numbers of texts for less-resourced languages. For South Slavic languages, this approach recently resulted in the largest general corpora for this language group – the CLASSLA-web 1.0 corpora ( Ljubešić and Kuzman 2024 ) . The corpus collection has been widely used for pre-training BERT-like and decoder-only large language models ( Ljubešić et al. 2024b ; Vreš et al. 2024 ) , for building datasets for downstream natural language processing (NLP) tasks ( Kuzman and Ljubešić 2025 ; Kuzman et al. 2024 ) , and has been shown to be heavily consulted by linguists ( Erjavec et al. 2024 ; Ljubešić et al. 2024a ) . Motivated by these positive results, we set up an ongoing crawling infrastructure dedicated to iterative national top-level domain (TLD) crawling of South Slavic and other webs. In this paper, we present the first outcome of the crawling infrastructure – the CLASSLA-web 2.0 corpus collection, a new release of web corpora for seven South Slavic languages. As the crawling pipeline methodologically mirrors the crawling pipeline from the MaCoCu project ( Bañón et al. 2022 ) , with which the CLASSLA-web 1.0 South Slavic corpus collection was produced, this enables us a comparison over time and provides insights into the evolution of the web in the two years between the two crawls. Our study makes the following contributions. First, we release a substantially larger web corpus collection containing 17.0 billion words in 38.1 million texts across Bosnian, Bulgarian, Croatian, Macedonian, Montenegrin, Serbian, and Slovenian (presented in Section 4 ). The CLASSLA-web 2.0 corpora can be downloaded in JSONL and VERT formats from the CLARIN.SI repository ( Kuzman Pungeršek et al. 2026 ) 1 1 1 The CLASSLA-web 2.0 corpora are freely accessible on the CLARIN.SI repository at http://hdl.handle.net/11356/2079 . and queried through the CLARIN.SI concordancers 2 2 2 https://www.clarin.si/ske/ , the links to each of the corpora are provided at the website https://clarinsi.github.io/classla-web/ . . Each text is linguistically annotated and labelled with genre and topic categories. Leveraging these annotation layers, we provide a detailed view of genre and news topic distributions across South Slavic webs (Section 4.1 ). Second, we perform a comparison between the CLASSLA-web 1.0 and CLASSLA-web 2.0 corpora (Section 5 ) showing that repeating TLD crawls after only two years yields much larger corpora with predominantly new content: on average, 82% of texts in CLASSLA-web 2.0 do not occur in CLASSLA-web 1.0. Analysing the correlation between the URL overlap and content overlap between two CLASSLA-web versions, we propose an approximate function for estimating content overlap via a weighted linear regression (Section 5.2.1 ). Third, we quantify web evolution in terms of change of sizes (Section 5.1 ), and web content quality (Section 5.3 ). We report a visible degradation of web content between versions, with “bad” (e.g., machine-generated or machine-translated) domains now contributing a much larger share of crawled material, underscoring the necessity of manual inspection of web domains that contribute the largest amounts of text. 2. Related Work The practice of building large web corpora for European languages was established with the WaCky initiative ( Baroni et al. 2009 ) . The first web corpora of South Slavic languages were compiled for Slovenian (slWaC) and Croatian (hrWaC) ( Ljubešić and Erjavec 2011 ; Erjavec et al. 2015 ) , followed by Bosnian (bsWaC) and Serbian (srWaC) ( Ljubešić and Klubička 2014 ) . After a seven-year gap, the MaCoCu ( Bañón et al. 2022 ) project resumed crawling activities for South Slavic languages and other less-resourced European languages. The monolingual web corpora for South Slavic languages that were produced in the MaCoCu project have been later additionally post-processed and enriched with linguistic and genre automatic annotation, and published as CLASSLA-web 1.0 ( Ljubešić and Kuzman 2024 ) corpus collection. The CLASSLA-web corpora have received considerable attention in the South Slavic space, both from the NLP community ( Ljubešić et al. 2024b ; Vreš et al. 2024 ; Kuzman and Ljubešić 2025 ; Kuzman et al. 2024 ) and from linguists and language teachers ( Erjavec et al. 2024 ; Ljubešić et al. 2024a ) . In parallel, numerous other web-based datasets have emerged from the Common Crawl and the Internet Archive data collections, e.g., the cc100 dataset ( Conneau et al. 2020 ) , the mC4 dataset ( Xue et al. 2021 ) , the OSCAR dataset ( Suárez et al. 2019 ) , and most recent, the HPLT ( de Gibert et al. 2024 ) and FineWeb2 datasets ( Penedo et al. 2025 ) . While these multilingual collections are highly useful for multilingual language modelling tasks, they suffer, inter alia, from lower quality of content, namely, cc100 and mc4 datasets ( van Noord et al. 2024 ) ; an unexplainable small amount of content in certain South Slavic languages, namely in OSCAR 3 3 3 https://oscar-project.github.io/documentation/versions/oscar-2301/ , mC4 dataset ( Xue et al. 2021 ) , the HPLT 3.0 datasets ( de Gibert et al. 2024 ) , and the FineWeb2 dataset ( Penedo et al. 2025 ) . More