Dynaword: From One-shot to Continuously Developed Datasets

Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan, Balázs Szabó, Márton Kardos, Kirsten Vad, Johan Heinsen, Andrea Blasi Núñez, Gianluca Barmina, Jacob Nielsen, Rasmus Larsen, Peter Vahlstrup, Per Møldrup Dalum, Desmond Elliott, Lukas Galke, Peter Schneider-Kamp, Kristoffer Nielbo · 2025

Read Dynaword: From One-shot to Continuously Developed Datasets by Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan, Balázs Szabó, Márton Kardos, Kirsten Vad, Johan Heinsen, Andrea Blasi Núñez, Gianluca Barmina, Jacob Nielsen, Rasmus Larsen, Peter Vahlstrup, Per Møldrup Dalum, Desmond Elliott, Lukas Galke, Peter Schneider-Kamp, Kristoffer Nielbo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02271.txt.

Introduction

Kenneth Enevoldsen and a large team set out a new way to build big text sets for language work, and show it in Danish. Their paper, from two thousand twenty-five, calls the way and the set Dynaword.

They name three flaws in how such sets are now made. Sources are used under vague rights, sets are fixed once and never grow, and checks rest with the first team alone. Their plan answers each flaw. Sets stay open, can be updated by the wider group, and draw checks from many hands. The Danish set shows the plan at work. The aim is large, clean text that stays live and shared.

Catalog record

CREATORKenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan, Balázs Szabó, Márton Kardos, Kirsten Vad, Johan Heinsen, Andrea Blasi Núñez, Gianluca Barmina, Jacob Nielsen, Rasmus Larsen, Peter Vahlstrup, Per Møldrup Dalum, Desmond Elliott, Lukas Galke, Peter Schneider-Kamp, Kristoffer Nielbo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02271

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynaword: From One-shot to Continuously Developed Datasets Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan, Balázs Szabó, Márton Kardos, Kirsten Vad, Johan Heinsen, Andrea Blasi Núñez, Gianluca Barmina, Jacob Nielsen, Rasmus Larsen, Peter Vahlstrup, Per Møldrup Dalum, Desmond Elliott, Lukas Galke, Peter Schneider-Kamp, Kristoffer Nielbo Abstract Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise. To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword 1 1 1 Available at: https://huggingface.co/datasets/danish-foundation-models/danish-dynaword . The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution. 1 Introduction Figure 1: Overview of the guiding principles for Dynaword corpora Continuously developed open-source projects are instrumental for contemporary research and lay the foundation for the success of our fields. These projects range from foundational software such as NumPy Harris et al. 2020 , to datasets like the Universal Dependencies Treebank Nivre et al. 2016 , to recent contributions including Flash Attention Dao et al. 2022 and LoRA Hu et al. 2022 . In open-source communities, it is understood that a project is never complete but is continually enhanced and adjusted in response to advances in the field and the software ecosystem. It is similarly recognized that relying on proprietary technology may result in legal ramifications, which, in the worst case, can render projects unusable or lead to their removal, undermining community contributions. We have already seen this impact in several instances: Udio AI Music Generator was shut down citing legal concerns; a state-of-the-art Danish encoder Enevoldsen et al. 2023 was removed following threats of legal action; and the Nordic Pile corpus Öhman et al. 2023 was never released, presumably due to copyright issues. However, the values of open source are notably lacking for (pre-)training datasets in the field. Although we have seen large-scale releases Gao et al. 2020b ; Xue et al. 2020 , these datasets often adhere to the pattern of being released once without updates. Even when we see continual releases Penedo et al. 2024 ; Abadji et al. 2022 , they are frequently based on Common Crawl content. Baack et al. 2025 defines this category as open access and notes several legal risks associated with the use of both datasets and derived models. Baack et al. 2025 also defines openly licensed data, which enables the resharing, reuse, and modification of data, thereby providing a solid foundation for derivative works. There have been a few initiatives Langlais et al. 2025 ; Langlais 2024 ; however, despite being a step forward, these contributions still have notable shortcomings. These datasets Langlais et al. 2025 ; Langlais 2024 constitute a single release without reproducible code for collection. This makes it difficult to reproduce the data collection process and difficult to improve 2 2 2 For instance, the Common Corpus consists of multiple OCR’d documents that could likely be improved with recent advancements. or update 3 3 3 E.g. the number of tokens on Wikipedia is expected to grow linearly Suh et al. 2009 . . Lastly, a vague description of the underlying licenses makes it difficult to validate the license claims. For instance, stating that "Alice in Wonderland" is public domain differs from documenting that the author died in 1898, which renders it within the public domain; we refer to this as a traceable license. Following these limitations, we propose the dynaword approach for curating a corpus: • Traceable and open licensing : All datasets within the corpus must be openly licensed and maintain a traceable license. • Reproducibility : It should be possible to derive a substantially similar dataset 4 4 4 Denoted as a ’substantially equivalent system’ in the Open-Source AI definition: https://opensource.org/ai/drafts/the-open-source-ai-definition-1-0-rc1 . • Documented : The dataset should be well-documented under best practices in the field Gebru et al. 2021 . • Extensible : Extending and improving the corpus should be possible, and methods for doing so should be documented. These guidelines are intended to provide datasets that are compatible with the open-source AI definition 5 5 5 https://opensource.org/ai/drafts/the-open-source-ai-definition-1-0-rc1 , FAIR Wilkinson et al. 2016 , and compliant under a diverse set of legal frameworks, including the European Parliament Artificial Intelligence Act European Union 2024 , and conducive to creating lasting resources for both research and industry. This approach draws upon successful datasets such as the Universal Dependency Treebanks Nivre et al. 2016 , which remain essential building blocks for linguistic analysis, model development Honnibal et al. 2020 , and recently, multilingual benchmarking of LLMs Nielsen 2023 ; Nielsen et al. 2024 . These guidelines were developed jointly with the Danish Dynaword, which acts as a practical implementation and a testbed, showing that the guidelines are implementable and not simply ideals. We believe Danish provides an ideal case, being a low-to-mid-resource 6 6 6 Class 2-4 as defined by Joshi et al. 2021 language, which has enough contributors willing to participate, yet well-contained in scope. High-resource languages 7 7 7 Class 5 as defined by Joshi et al. 2021 could likely sustain multiple dynaword projects targeting specific domains such as code, academia, or healthcare. The following sections outline how we developed a framework around Danish Dynaword that actively ensures traceable and open licensing and reproducibility, while being documented and reproducible. Our final dataset represents more than a fourfold increase in Danish tokens available compared to previous datasets and considerably improves reproducibility and documentation. We expect that this work will continue to expand with future contributions. 2 Related Works 2.1 Continually released pre-training data Recently, dataset releases have been increasingly iterative releases with improvements to extraction, cleaning, and collection procedures. These include OSCAR Abadji et al. 2022 , HPLT Aulamo et al. 2023 ; Arefyev et al. 2024 , and fineweb Penedo et al. 2024 , all which are mostly based on Common Crawl 8 8 8 https://commoncrawl.org . Though Common Crawl and its derivatives represent significant artifacts, the license of the underlying data remains ambiguous, raising a slew of ethical and legal concerns Baack et al. 2025 . Additionally, these sources typically work with web-derived content, overlooking sources such as APIs, television, or radio. 2.2 Openly licensed data Data licensing is often unclear, and it is not uncommon to find datasets on Hugging Face 9 9 9 https://huggingface.co published under permissive licenses containing copyrighted content. In some cases Abadji et al. 2022 ; Penedo et al. 2024 the license applies only to the packaging, metadata, and annotations, but not to the data itself. To avoid such confusion, we will, throughout this work, utilize the three tiers defined by Baack et al. 2025

Read the complete text