Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug · 2026
Read Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain? by Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19394.txt.
Introduction
Doll and colleagues ask whether continual pre-training can close the gap between small specialized models and much larger general-purpose models in medicine. They build FineMed-de, a high-quality German medical corpus from FineWeb2. The corpus pre-trains and merges three well-known language models from 7 billion to 24 billion parameters, creating the DeFineMed family. Specialization lifts 7-billion-model scores sharply on German medical benchmarks, and win rates against the much larger Mistral-Small-24B-Instruct rise roughly three-and-a-half-fold through domain adaptation.
Catalog record
| CREATOR | Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55788 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19394
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain? Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug Abstract This paper narrows the performance gap between small, specialized models and significantly larger general-purpose models through domain adaptation via continual pre-training and merging. We address the scarcity of specialized non-English data by constructing a high-quality German medical corpus ( FineMed-de ) from FineWeb2 . This corpus is used to continually pre-train and merge three well-known LLMs (ranging from 7B to 24B parameters), creating the DeFineMed model family. A comprehensive evaluation confirms that specialization dramatically enhances 7B model performance on German medical benchmarks. Furthermore, the pairwise win-rate analysis of the Qwen2.5 -based models demonstrates an approximately 3.5 3.5 -fold increase in the win-rate against the much larger Mistral-Small-24B-Instruct through domain adaptation. This evidence positions specialized 7 B 7B models as a competitive, resource-efficient solution for complex medical instruction-following tasks. While model merging successfully restores instruction-following abilities, a subsequent failure mode analysis reveals inherent trade-offs, including the introduction of language mixing and increased verbosity, highlighting the need for more targeted fine-tuning in future work. This research provides a robust, compliant methodology for developing specialized LLMs, serving as the foundation for practical use in German-speaking healthcare contexts. 1 Introduction Large language models (LLMs) have demonstrated transformative potential across various domains, including healthcare ( 42 ; 58 ) . Their ability to process and generate human-like text enables applications such as diagnostics, personalized treatment plans, and efficient information retrieval ( 33 ) . However, significant gaps remain in the integration of LLMs into clinical workflows, where general-purpose models often fail to capture domain-specific knowledge and terminology with sufficient accuracy ( 27 ; 1 ) . The effective application of LLMs in the medical field faces two significant, interconnected challenges. First, stringent data protection regulations necessitate on-premise solutions, making the use of large, API-based LLM services impractical and favoring smaller, computationally efficient models ( 2 ) . Second, these smaller models, while being regulatory compliant, struggle to capture the complex, nuanced medical terminology due to a scarcity of high-quality, domain-specific datasets, a problem acutely felt in non-English languages ( 13 ) . This creates a critical trade-off: regulatory constraints demand small models, underscoring the necessity of a targeted, specialized knowledge base to achieve competitive clinical performance. This work directly addresses this challenge by investigating the core research question: Can domain-adaptation of 7B language models, achieved via continual pre-training and merging, close the performance gap sufficiently to compete with significantly larger, general-purpose models on complex medical tasks? Beyond technical constraints, this study aims to quantify how effectively domain knowledge, gained without massive retraining, can be internalized to transform resource-efficient models into viable clinical tools. This paper directly addresses these challenges by developing a specialized German medical LLM through a systematic approach to dataset creation and model adaptation, as illustrated in Figure 1 . Our key contributions are as follows: • A robust and scalable filtering methodology for creating high-quality, domain-specific datasets from general-purpose corpora. This methodology combines LLM-based annotation with machine learning techniques and is applicable to various domains and languages. • A comprehensive evaluation, utilizing knowledge-intensive benchmarks ( MMLU-de and MedQA-de ) and a pairwise win-rate analysis, demonstrating that specialized 7B models drastically close the performance gap toward 24B models. This validates the competitive viability of smaller, resource-efficient LLMs, while a simultaneous failure mode analysis highlights the critical trade-offs inherent to the specialization methodology. The remainder of this paper is structured as follows: Section 2 provides an overview of the state of the art. Section 3 outlines the construction of the German medical pre-training dataset. Further, Section 4 describes the model adaptation techniques employed. The evaluation methodology and results are presented in Section 5 , followed by a discussion of the observed results in Section 6 . Finally, Section 7 concludes the paper with a summary of our findings and potential future directions. Figure 1: High-level illustration of the Data Filtering and Model Adaptation workflow: a subset of the German FineWeb2 dataset is annotated into medical and non-medical documents. The resulting annotated dataset, FineMed-de-260k , is used to train a classifier, which is then applied to the full German FineWeb2 split, producing the FineMed-de dataset. This dataset is subsequently used for continual pre-training - resulting in a Medical LLM - which is then merged with the initial instruction-tuned checkpoint. 2 Related Work This work intersects several key areas of research in medical LLM development, including model adaptation through continual pre-training and model merging techniques, as well as dataset curation through document filtering. To provide a comprehensive context, we review relevant literature in each of these domains, highlighting advancements and methodologies that align with our approach. 2.1 Model Adaptation through Continual Pre-training Recent studies have explored continual pre-training as a means to adapt general-purpose LLMs to specialized domains, but results remain inconsistent. 40 showed that additional pre-training can fail, or even harm performance, when domain data diverges from the model’s original distribution, underscoring the importance of corpus quality and alignment. In contrast, research in the legal and financial domains ( 37 ; 49 ) suggests that combining continual pre-training with model merging can yield competitive improvements while preserving general reasoning capabilities. Similarly, 56 showed that targeted continual pre-training with efficient adaptation strategies enhances domain fluency, suggesting that success depends less on scale and more on data selection and adaptation design. 2.2 Medical Large Language Models Recent progress in Medical LLMs has been achieved through pre-training ( 32 ; 44 ) , fine-tuning ( 48 ; 50 ; 18 ; 60 ) , and prompting ( 39 ; 29 ) , yielding models that match or surpass human experts on medical QA tasks ( 48 ; 39 ) . 28 advanced this line with BioMistral , a continually pre-trained biomedical model based on Mistral ( 21 ) and PubMed-Central data ( 36 ) , outperforming open-source baselines. Our work fundamentally differs from BioMistral not only in targeting the German language but in its primary research objective. Where BioMistral aims for performance improvements on established, knowledge-intensive benchmarks, our study investigates the competitive viability of specialized small LMs against significantly larger general-purpose models and extends the evaluation beyond knowledge-intensive benchmarks. More recently, 60 introduced Apollo-2 , a multilingual medical LLM family covering 50 languages, focusing on instruction-tuning and modular routing rather than continual pre-training for domain adaptation. 2.3 Merging Large Language Models Model merging provides an efficient alternative to computationally expensive fine-tuning and ensembling by combining pre-trained LLMs fine-tuned on specialized tasks. The survey by 55 presents a taxonomy of model merging approaches, their applications in various domain subfields, and their challenges. Additionally, 38 propose Activation-Informed Merging , leveraging activation-space information to improve the robustness and efficiency of merging, showing up to a 40\% performance gain across benchmarks. Alternative strategies further explore computational efficiency and generalization. 16 introduce a task-vector-based approach using LoRA-derived representations and geometric median aggregation, enabling effective merging with reduced computational costs. Meanwhile, 54 conducted large-scale experiments, merging models up to 64B parameters, showing that merging enhances zero-shot generalization and is particularly effective when starting from strong base models. Their findings suggest that larger