MedCL-Bench: Benchmarking stability–efficiency trade-offs and scaling in biomedical continual learning
Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang · 2026
Read MedCL-Bench: Benchmarking stability–efficiency trade-offs and scaling in biomedical continual learning by Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16738.txt.
Introduction
Zeng, Zhou, Zhan, and Zhang benchmark continual learning for medical language models. Models must absorb new evidence and terms without forgetting old skills. No shared, task-diverse benchmark existed for that audit. Their MedCL-Bench streams ten datasets across five task families. Eleven strategies run under eight task orders, with GPU-hour costs reported. Plain fine-tuning forgot catastrophically across the board. Parameter isolation gave the best retention per GPU-hour. Replay protected more at higher cost; regularization helped least.
Catalog record
| CREATOR | Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43550 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16738
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedCL-Bench: Benchmarking stability–efficiency trade-offs and scaling in biomedical continual learning Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang Abstract Medical language models must be updated as evidence and terminology evolve, yet sequential updating can trigger catastrophic forgetting. Although biomedical NLP has many static benchmarks, no unified, task-diverse benchmark exists for evaluating continual learning under standardized protocols, robustness to task order and compute-aware reporting. We introduce MedCL-Bench , which streams ten biomedical NLP datasets spanning five task families and evaluates eleven continual learning strategies across eight task orders, reporting retention, transfer, and GPU-hour cost. Across backbones and task orders, direct sequential fine-tuning on incoming tasks induces catastrophic forgetting, causing update-induced performance regressions on prior tasks. Continual learning methods occupy distinct retention–compute frontiers: parameter-isolation provides the best retention per GPU-hour, replay offers strong protection at higher cost, and regularization yields limited benefit. Forgetting is task-dependent, with multi-label topic classification most vulnerable and constrained-output tasks more robust. MedCL-Bench provides a reproducible framework for auditing model updates before deployment. Large language models (LLMs) are increasingly used to support biomedical question answering, evidence retrieval, relation extraction, and document-level classification. Yet biomedical knowledge is not static: new findings, revised clinical evidence, and evolving therapeutic guidance continually change what models should know. However, updating large models by full retraining is often computationally impractical, while repeatedly fine-tuning on new datasets can erode previously acquired capabilities through catastrophic forgetting ( 18 ; 9 ) . The result is a tension between plasticity (incorporating new knowledge) and stability (preserving prior competencies) that is especially consequential in biomedical settings, where outdated or inconsistent behavior can directly undermine downstream research and decision-support. Continual learning (CL) has therefore emerged as a promising paradigm for enabling models to acquire new knowledge over time while mitigating degradation of prior competencies ( 20 ) . In the clinical domain, these challenges are sharpened by three practical constraints ( 12 ; 15 ) . First, privacy and data-governance constraints often limit the pooling of raw patient records across institutions, creating persistent “data walls” between sites ( 8 ; 1 ) . This motivates model-to-data workflows in which models are updated sequentially across hospitals (e.g., Hospital A \rightarrow Hospital B) by transferring parameters rather than sensitive primary data ( 4 ; 19 ) . Second, clinical data are inherently sequential and longitudinal ( 2 ; 10 ) : patient information accrues over time through repeated encounters, tests, treatments, and evolving diagnoses, while labeled datasets and task definitions often emerge incrementally rather than as a single static corpus. Third, clinical practice exhibits continual knowledge drift : diagnostic standards, treatment guidelines, and pathogen profiles can change over time ( 23 ; 7 ; 14 ) , requiring models to incorporate new evidence without repeated full retraining. These constraints translate directly into deployment risk. Sequential updates can introduce silent regressions ( 7 ; 13 ) : a model may improve on newly introduced data while degrading on previously validated capabilities. In safety-critical workflows, such regressions increase the verification burden for clinicians and can undermine the reliability of downstream decision-support, including tasks such as adverse-event monitoring or drug–drug interaction detection. This creates a practical need for rigorous evaluation of continual learning behavior under repeated updates. Despite growing interest in both biomedical Natural Language Processing (NLP) and continual learning, several questions remain unresolved in realistic biomedical deployment settings. In particular, it remains unclear how severe catastrophic forgetting is in biomedical NLP, whether CL methods reliably mitigate forgetting and deliver meaningful benefits in biomedical NLP, which classes of CL strategies are most effective for biomedical tasks, whether findings from general-domain CL transfer to settings with specialized biomedical language and heterogeneous data distributions, whether larger backbones consistently improve performance or instead exhibit non-monotonic, backbone-dependent trade-offs, and how these methods compare in terms of training cost, parameter efficiency, and update-time overhead. Biomedical NLP still lacks a unified, task-diverse benchmark for CL under a standardized training and evaluation protocol to answer these questions rigorously. To address these gaps, we introduce MedCL-Bench (Fig. 1 ), a unified continual learning benchmark for biomedical NLP. MedCL-Bench is designed to answer three practical questions: (i) how severe catastrophic forgetting is in biomedical NLP, (ii) which continual learning strategies offer the best trade-offs between end-of-stream performance and training cost, and (iii) how sensitive conclusions are to task order and backbone choice under realistic resource constraints. Specifically, MedCL-Bench provides a standardized benchmark suite and evaluation pipeline for controlled comparison of continual learning strategies in biomedical NLP. First, we curate and standardize ten public biomedical NLP datasets into a standardized continual learning benchmark suite spanning five task families: biomedical question answering ( PubMedQA , BioASQ ), scientific fact checking ( SciFact , PubHealth ), relation extraction ( GAD , ChemProt , DDI ), document-level classification ( Pubmed_RCT , DRUGLIB ), and multi-label topic classification of biomedical literature ( LitCovid ). Second, we evaluate sequential updates across eight pre-specified task orders with a unified preprocessing and evaluation protocol, enabling matched comparisons across continual learning strategies and backbone architectures. Third, we benchmark representative continual learning strategies—including naive fine-tuning, multi-task learning ( 5 ) , regularization ( 11 ; 29 ; 3 ) , rehearsal/gradient projection ( 16 ; 6 ; 21 ) , generative replay ( 31 ; 28 ; 24 ) , and parameter-efficient adaptation ( 17 ; 27 ) —and report end-of-stream performance together with compute- and parameter-efficiency (including GPU-hour cost) to characterize stability–efficiency trade-offs. Finally, we release the full benchmark codebase—preprocessing, training, and evaluation—to enable reproducible comparisons and further research. Using this benchmark, we further find that retention is not uniform across biomedical tasks. Tasks with more constrained output structure, such as multiple-choice question answering and multi-class relation extraction, are comparatively robust to sequential updates, whereas multi-label topic classification with overlapping label sets is markedly more vulnerable to forgetting. These patterns suggest that forgetting in biomedical continual learning depends not only on task difficulty, but also on task formulation and output structure. Figure 1 : Overview of MedCL-Bench. (a) Biomedical knowledge and datasets evolve continuously (e.g., new literature and drug–disease relations), creating realistic sequential update streams—both across institutions (where data cannot be pooled) and within an institution over time. (b) Sequential sequential fine-tuning can overwrite previously acquired capabilities (catastrophic forgetting), whereas CL aims to retain prior knowledge while learning new tasks. (c) MedCL-Bench comprises ten biomedical NLP datasets grouped into five task families (QA, fact checking, relation extraction, document classification, and multi-label topic classification). (d) Benchmark workflow: a pretrained backbone is updated sequentially on a task stream under multiple task orders, and evaluated on all previously seen tasks after each stage. (e) CL metrics reported in this work: overall task performance (AP), backward transfer (BWT), and forward transfer (FWT). (f) Key questions addressed: forgetting severity, method comparison, order sensitivity, compute efficiency, and scaling/backbone dependence. Icons are sourced from Flaticon.com (full attributions in Supplementary Note ). Results We evaluate continual learning on MedCL-Bench, a stream of ten public biomedical NLP datasets spanning five task types: biomedical question answering, scientific fact checking, relation extraction, document-level classification, and multi-label topic classification. A detailed overview of all datasets is provided in Extended Data Table . Tasks are presented sequentially under eight randomized task orders (Extended Data Table