The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
Henry He, Johann Frei, Raphael Schmitt · 2026
Read The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP by Henry He, Johann Frei, Raphael Schmitt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03250.txt.
Description
He, Frei, and Schmitt train German language models for clinical text. Digital healthcare makes vast clinical text that could power AI tools. German biomedical models lag, held back by old designs or thin training data. Their family, ChristBERT, is built on RoBERTa and tuned for clinical and healthcare topics.
The training corpus holds 13.5 gigabytes of text: science papers, clinical notes, health web content, and translated clinical sources. The team compares three paths: continued pretraining, training from scratch, and vocabulary adaptation. Tests cover three entity recognition tasks and two classification tasks in medicine. ChristBERT beat past general and medical German models on four of five benchmarks. It set a new state of the art for German clinical language work. The best path split by task. Training from scratch won on the most specialized texts. Continued pretraining won on more common medical writing. All models are released in public.
Catalog record
| CREATOR | Henry He, Johann Frei, Raphael Schmitt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03250