Language corpora for the Dutch medical domain
Bram van Es · 2026
Read Language corpora for the Dutch medical domain by Bram van Es (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25374.txt.
Introduction
van Es addresses scarce Dutch medical corpora limiting NLP development by translating English data sets, identifying medical text in generic corpora and extracting open Dutch resources. The corpus comprises about 36 billion tokens across the medical domain in about 105 million documents, freely available on Hugging Face. This work establishes the first large Dutch medical language corpus for pre-training and NLP tasks.
Catalog record
| CREATOR | Bram van Es |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10895 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language corpora for the Dutch medical domain Bram van Es Abstract Background: Dutch medical corpora are scarce, limiting NLP development. Methods: We translated English datasets, identified medical text in generic corpora, and extracted open Dutch medical resources. Results: The resulting corpus comprises ± \pm 35 billion tokens across the medical domain in about 100 million documents, freely available on Hugging Face. Conclusion: This work establishes the first large-scale Dutch medical language corpus for pre-training and downstream NLP tasks. keywords clinical natural language processing, language corpora, open source 1 Background Recent advances in natural language processing (NLP) have been driven by the availability of large, high-quality corpora such as C4, The Pile, and PubMedBERT’s training data 19 ; 5 ; 6 . However, for many minority and medium-resource languages, including Dutch, such domain-specific resources remain scarce. This is particularly true for the medical domain, where privacy concerns, copyright restrictions, and fragmented data sources make corpus creation challenging 14 . Several general-purpose Dutch corpora exist, such as COW 23 , OSCAR 17 , TwNC 16 and SoNaR 15 . More recently large-scale web-crawled resources such as FineWeb2 and Finepdf 18 contain high quality Dutch corpora as does C4. While valuable for training large generic language models (e.g., RobBERT 3 ), these corpora contain little medical content and are not optimized for clinical or biomedical NLP. For clinical Dutch text specifically, resources are often limited to small annotated datasets for named entity recognition or classification tasks, typically derived from electronic health records under strict confidentiality agreements that require de-identification, see e.g. 13 or 4 . As a result, researchers developing Dutch clinical NLP applications must either rely on translations of English corpora or manually curate small datasets. Both approaches have significant drawbacks: machine translations may introduce linguistic artifacts, and small corpora lack the coverage needed for robust model training. To bridge this gap, we present a large-scale Dutch medical corpus that combines multiple strategies: machine translation of existing biomedical corpora, automated identification of medical text in large Dutch web corpora, and targeted extraction of openly available resources such as PhD theses and professional guidelines. The resulting corpus, comprising approximately 35 billion tokens, is freely available and provides a broad, open resource for Dutch medical NLP research and model pre-training. 2 Methodology Translation & Transformation of non-Dutch corpora. We used a variety of decoder and encoder/encoder models to translate existing English corpora. Specifically NLLB by 24 and MariaNMT by 25 for encoder/decoder models, and GPT3.5, GPT4o, Gemini1.5, Gemini2 and Gemini2.5 for the decoder models, and (accidentally) the Google Translate API 1 1 1 pro tip: don’t start an unconstrained translation of GBs of text with a $20/M-tokens model before you do your groceries . We also translated several annotated corpora, including BioASQ and MedQA, using GPT4o-mini. Besides direct translations using LLMs we also transformed PMC Patient cases into structurally formatted discharge letters using Gemini 1.5 and Gemini 2. We translate the Pubmed PMC archives using MariaNMT. For the machine translations we split the texts into non-overlapping, sentence delimited chunks of about half the maximum context length of the model, with half reserved for the decoder. For all the PMC content we have added the specific copyright statement belonging to each individual PMID. Identification of medical texts in generic Dutch corpora. We used an LLM, GPT4.1-nano, to label 100.000 random samples from the OSCAR dataset as medical/non-medical. We then trained a dense layer on top of a frozen RobBERT2023 encoder model on this labeled corpus. This classifier was then used to identify medical texts in FineWeb2 and FinePDFs. Extraction of texts from open resources. Via an Open Archives Initiatives 2 2 2 https://www.openarchives.org/ (OAI)-connection with Dutch academic institutions we extracted Dutch PhD theses from which the Dutch/English content was extracted. For the selection we checked for medical keywords. We specifically selected right-free theses that were not under embargo. These PhD theses were parsed into an English medical corpus, as well as a parallel Dutch/English corpus based on the summaries and abstracts. The summary pairs were checked for multilingual similarity using sentence transformers (we used the paraphrase-multilingual-MiniLM-L12-v2 model). For the pdf-parsing we used PyPDF, Fitz and PDFminer in order of success. We checked whether the PDFs were the result of OCR by checking the producer in the Meta data and as a fallback we checked the number of pages that had text available in the page-objects 3 3 3 Quite arbitrary if more 75% or more had no text, for a minimum of 15 pages, it was scanned, inversely if more than 75% had text, for a minimum of 15 pages, it was not scanned . If we decided that PDF was scanned we used PyTesseract to perform image-to-text on the list of images that we extracted using Fitz. We also share online resources such as NtvG publications and medical protocols from the federation of medical specialists (FMS) and the Dutch GP society (NHG). Cleaning the translations and the thesis-extraction were done using FTFY, and otherwise we applied regular expressions to remove spurious word repetitions and multiplicity of line breaks and spacing. De-identification was not performed on the invididual datasets, note that email and IP-address were already de-identified for the FineWeb2 and Finepdf corpora. We did perform a deidentification using the heuristics-based DEDUCE by 13 on the combined set DutchMedicalTextV3 4 4 4 https://huggingface.co/datasets/UMCU/DutchMedicalTextV3 . For most of these tasks we used the (pre-alpha level) open source library PubScience 5 5 5 https://github.com/bramiozo/PubScience . 3 Dataset characteristics Source Extraction type Words Documents Words/Document FineWeb2 by 18 Identification 11.5B 11.7M \pm 500 Finepdfs by 18 Identification 2.2B 0.5M \pm 2000 Pubmed Abstracts 6 6 6 https://pubmed.ncbi.nlm.nih.gov/download/ Translation 2.6B 15M \pm 200 Pubmed PMC OA Non-commercial 7 7 7 https://pubmed.ncbi.nlm.nih.gov/download/ Translation/Transformation 8.7B 3.4M \pm 2560 Pubmed PMC OA Commercial 8 8 8 https://pubmed.ncbi.nlm.nih.gov/download/ Translation/Transformation 4.4B 1.5M \pm 2933 Apollo corpus by 26 Translation 0.2B 1.2M \pm 167 EMEA 9 9 9 https://opus.nlpl.eu/EMEA/corpus/version/EMEA Translation 34M 1662 \pm 36000 MIMIC III/IV 11 ; 10 Translation 2.9B 9.4M \pm 308 ACGT by 21 Translation 18M 422K \pm 42 NtvG 10 10 10 https://www.ntvg.nl/ N/A 34M 44K \pm 773 FMS 11 11 11 https://demedischspecialist.nl/ , NHG 12 12 12 https://richtlijnen.nhg.org/ Scraping 200M 364K \pm 550 other 13 13 13 PhD summaries, BioLORD 20 ; 22 , Meditron 2 , etc. 2.2B 30M \pm 73.3 total 35B 73.5M \pm 480 Table 1: Overview of (most of the) datasets available with estimates of word counts, as of December 2025 . Note that these are the statistics for the datasets resulting from the translation effort. Source Extraction type Interaction type # MedicalFlashCards 14 14 14 https://huggingface.co/datasets/medalpaca/medical_meadow_medical_flashcards Translation Q/A open 32.9K MediQA by 1 Translation Q/A open 2.11K MedQA by 9 Translation Q/A MC 9.86K MedQA 15 15 15 The multiple choices are written out explicitly Translation Q/A MC written 14.4K MedicalMMLU by 8 Translation Q/A MC 3.65K WikiDoc by 7 Translation Q/A MC 5.76K SymptomDiseaseQA 16 16 16 https://huggingface.co/datasets/prognosis/symptoms_disease_v1 Translation Q/A open 10.1K BioASQ by 12 17 17 17 https://zenodo.org/records/7655130 Translation Q/A 4.7k total 54.28K Table 2: Overview of the interaction finetuning set available, as of December 2025 4 Models We trained several models on (parts of) this data. Specifically, as of writing; • CardioLlama.nl: