BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP
Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, \ANDtwo Matthew McDermott, Tristan Naumann, Charlotta Lindvall · 2025
Read BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP by Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, \ANDtwo Matthew McDermott, Tristan Naumann, Charlotta Lindvall (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10896.txt.
Introduction
Software that mines hospital notes and research text needs models trained on medical language. Encoder models have lagged behind the flashier chat models in that training. Sounack and co-authors introduce BioClinical ModernBERT, an encoder adapted to biomedical and clinical text. It builds on the recent ModernBERT design, adding long-context handling and speed gains. It was further trained on the largest such corpus yet, over 53.5 billion tokens drawn from twenty datasets across many institutions and regions.
On four downstream tasks spanning a range of uses, the new encoder beat the existing biomedical and clinical encoders. The team releases two sizes, a base model of 150 million parameters and a large one of 396 million, along with training checkpoints. They aim the release at researchers who need fast, accurate extraction of structured facts from unstructured clinical text.
Catalog record
| CREATOR | Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, \ANDtwo Matthew McDermott, Tristan Naumann, Charlotta Lindvall |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10896
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, \ANDtwo Matthew McDermott, Tristan Naumann, Charlotta Lindvall Abstract Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pre-training on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research. † † https://github.com/lindvalllab/BioClinical-ModernBERT 1 Introduction As clinical information databases have become more comprehensive, there has been a surge in the application of Natural Language Processing (NLP) methods to support downstream healthcare tasks, ranging from clinical decision support and care delivery analyses to patient cohort selection for clinical trials ( Cho et al. 2024 ; Abdel-Jaber et al. 2022 ) . This trend has coincided with the rapid evolution of deep learning models, particularly transformer-based architectures, which have shown strong performance across both generative and discriminative NLP tasks ( Min et al. 2024 ) . Encoder-based architectures, which process entire input sequences using bidirectional attention, have traditionally been preferred for non-generative tasks such as classification and span extraction, due to their ability to capture rich contextual representations. In contrast, decoder-based models rely on autoregressive decoding, generating one token at a time while attending only to previously generated tokens. This structure makes them well-suited for open-ended text generation ( Nielsen et al. 2025 ) . Although encoder-based approaches were once dominant, recent research has increasingly centered on decoder-based models, and encoder-based architectures have received comparatively less interest ( Shool et al. 2025 ) . This has led decoders to be used for non-generative tasks where encoders would be better suited, using workarounds like structured output generation ( Geng et al. 2025 ) . The community’s preference for decoder-based models may at least in part be attributed to their capacity to support longer input lengths and have better generalization as well as zero-shot capabilities; advantages largely driven by greater engineering investment and scaling efforts. Nonetheless, encoders are still necessary in many applications and are used in conjunction with LLMs, such as in Retrieval-Augmented Generation (RAG) frameworks ( Lewis et al. 2020 ) . The release of ModernBERT ( Warner et al. 2024 ) represents a significant advancement in encoder architectures, introducing a longer context window, improved efficiency, an expanded vocabulary, and a substantially larger volume of training data. These enhancements are particularly impactful in the clinical domain. With a context window of up to 8,192 tokens, extendable via rotary positional embeddings (RoPE) ( Su et al. 2023 ) , ModernBERT enables the processing of entire clinical notes and documents in a single pass, eliminating the need for fragmentation into smaller chunks. The increased vocabulary size of 50,368, compared to BERT’s 30,000, supports the learning of more precise token embeddings, which is especially beneficial for capturing the diversity and complexity of clinical and biomedical terminology. In this work, we perform a two-step continued pretraining of the ModernBERT models on biomedical and clinical corpora, offering both base and large versions, and publicly release the trained models along with their training checkpoints. We demonstrate significant improvements, achieving state-of-the-art performance across multiple benchmarks compared to existing encoders, while offering support for long-context inputs and maintaining the highest overall efficiency across various input distributions. Our approach leverages the largest training dataset ever used for a biomedical or clinical encoder and incorporates a diverse range of clinical text sources to supplement the traditionally relied-upon MIMIC data ( Johnson et al. 2016 ; Johnson et al. 2023 ) . Notably, while MIMIC replaces protected health information (PHI) with generic de-identification tags, limiting the model’s ability to learn meaningful representations of entity-specific context, several of the additional datasets we include use realistic surrogate identifiers which allows BioClinical ModernBERT to learn more natural representations around these entities. 2 Related Work General domain encoders Although decoder-only models have gained significant popularity, encoder-only architectures remain widely used due to their favorable trade-off between performance and efficiency, especially in Information Retrieval (IR) and RAG pipelines. Their low inference cost and ability to process documents efficiently make them attractive for large-scale applications. While many existing pipelines still rely on older models like BERT, which are limited by short context windows, outdated training corpora, and less efficient design choices ( Wang et al. 2018 ; Xiao et al. 2024 ) , recent efforts such as MosaicBERT ( Portes et al. 2024 ) , CrammingBERT ( Geiping and Goldstein 2022 ) , and GTE-en-MLM ( Li et al. 2023 ) have introduced updated architectures and longer context support. However, these models have largely focused on retrieval tasks or efficient training rather than broad downstream performance. ModernBERT ( Warner et al. 2024 ) addresses many of these limitations by scaling up training data to two trillion tokens, introducing improved architectural choices, and offering competitive performance across a wider range of tasks. Biomedical and clinical encoders Recent advancements in NLP have led to the development of domain-adapted transformer models that significantly enhance the extraction and interpretation of clinical text. Among these, BioBERT ( Lee et al. 2019 ) and Clinical BERT ( Alsentzer et al. 2019 ) have emerged as pivotal tools in biomedical and clinical informatics, respectively. BioBERT extends the original BERT architecture through pretraining on large-scale biomedical literature, enabling more accurate semantic understanding of domain-specific language. Clinical BERT builds upon this by incorporating additional pretraining on clinical notes from the MIMIC-III dataset ( Johnson et al. 2016 ) , allowing it to better capture the nuances of clinical documentation. BioMed-RoBERTa ( Gururangan et al. 2020 ) , relying on the RoBERTa architecture ( Liu et al. 2019 ) , takes a different approach by training on Semantic Scholar ( Kinney et al. 2025 ) as its source of biomedical text. These models have outperformed general-domain encoders in tasks such as concept extraction, temporal relation identification, and outcome prediction, making them essential components in the clinical NLP pipeline. Long context Long-context understanding is especially important in the clinical NLP domain, as clinical documentation tends to be highly variable in structure, terminology, and length, often exceeding the standard 512-token input limit of many encoder-based models ( Rule et al. 2021 ) . The ability to process longer sequences enables models to capture and integrate relevant information dispersed across entire clinical notes or even multiple documents. This extended context facilitates the identification of patterns and