MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
Junzhi Ning, Jiashi Lin ⋆ \star, Yingying Fang, Wei Li, Jiyao Liu, Cheng Tang, Chenglong Ma, Wenhao Tang, Tianbin Li, Ziyan Huang, Guang Yang ✉ Junjun He ✉ · 2026
Read MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark by Junzhi Ning, Jiashi Lin ⋆ \star, Yingying Fang, Wei Li, Jiyao Liu, Cheng Tang, Chenglong Ma, Wenhao Tang, Tianbin Li, Ziyan Huang, Guang Yang ✉ Junjun He ✉ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10755.txt.
Introduction
Ning and eleven colleagues introduce MMRareBench, the first benchmark testing multimodal large language models on rare diseases across multiple images. Existing benchmarks focus on common conditions and single images, leaving multi-image evidence integration under data scarcity unevaluated. The benchmark spans four workflow tracks, diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion, with 1,756 question-answer pairs and 7,958 images from PMC case reports. Evaluating 23 models reveals fragmented skills and universally weak treatment planning, with medical models trailing general models on multi-image tracks despite competitive diagnosis scores.
Catalog record
| CREATOR | Junzhi Ning, Jiashi Lin ⋆ \star, Yingying Fang, Wei Li, Jiyao Liu, Cheng Tang, Chenglong Ma, Wenhao Tang, Tianbin Li, Ziyan Huang, Guang Yang ✉ Junjun He ✉ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21428 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark Junzhi Ning, Jiashi Lin ⋆ \star, Yingying Fang, Wei Li, Jiyao Liu, Cheng Tang, Chenglong Ma, Wenhao Tang, Tianbin Li, Ziyan Huang, Guang Yang ✉ Junjun He ✉ Abstract Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinical knowledge, forcing them to rely strictly on case-level evidence for clinical judgments. Existing benchmarks predominantly evaluate common-condition, single-image settings, leaving multimodal and multi-image evidence integration under rare-disease data scarcity systematically unevaluated. We introduce MMRareBench , to our knowledge the first rare-disease benchmark jointly evaluating multimodal and multi-image clinical capability across four workflow-aligned tracks: diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion. The benchmark comprises 1,756 question-answer pairs with 7,958 associated medical images curated from PMC case reports, with Orphanet-anchored ontology alignment, track-specific leakage control, evidence-grounded annotations, and a two-level evaluation protocol. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. These patterns are consistent with a capacity dilution effect: medical fine-tuning can narrow the diagnostic gap but may erode the compositional multi-image capability that rare-disease evidence integration demands. The benchmark will be available at https://anonymous.4open.science/r/MMRareBench-C80E/ . 1 Introduction Rare diseases collectively affect over 300 million people worldwide [ 8 ] . However, each individual condition occurs so infrequently that the resulting long-tail distribution creates severe class imbalance and overlapping phenotypes [ 18 ] . In this regime, reliable disease-specific priors are sparse, and clinical judgments cannot rely on memorized textbook patterns; they must be strictly grounded in the specific evidence available in the current case. To navigate this complexity and reach an accurate diagnosis or treatment decision, clinicians often jointly interpret narrative text, structured signals such as laboratory panels, and multiple medical images spanning modalities and time points. Therefore, successfully diagnosing rare diseases fundamentally depends on two interconnected capabilities: (1) multimodal integration , which requires fusing text, visual, and tabular data to extract complementary signals [ 12 ] , and (2) multi-image analysis , which demands selecting, aligning, and comparing across several images within one encounter, such as reconciling radiology with pathology or tracking longitudinal change. Despite the critical need for these capabilities in clinical practice, they are rarely measured together in the development and evaluation of current medical AI. Current multimodal large language models (MLLMs) are typically assessed on common-condition settings or exam-style, single-image tasks [ 16 , 9 ] . While some benchmarks focus on rare diseases [ 5 ] , they primarily evaluate text-based diagnostic accuracy rather than visual evidence integration. Furthermore, many existing settings are textbook-like and internally consistent, which may lead to overestimated performance when clinical evidence is atypical or partially contradictory [ 30 , 20 , 15 ] . Table 1: Benchmark comparison. Rare: rare-disease focus; M.mod: multi-modality inputs (text, image, table); M.img: multi-image per case (multiple images); Clinical: workflow-aligned clinical tracks. Benchmark Rare M.mod M.img Clinical MedR-Bench [ 21 ] ✗ ✗ ✗ ✓ MTBBench [ 25 ] ✗ ✓ ✓ ✓ RareBench [ 5 ] ✓ ✗ ✗ ✗ GMAI-MMBench [ 30 ] ✗ ✓ ✗ ✓ Ours ✓ ✓ ✓ ✓ To address these gaps, we present MMRareBench , a multimodal rare-disease benchmark that stress-tests whether MLLMs truly integrate complex case evidence under rare-disease data scarcity, rather than rely on memorized common-condition patterns. Constructed from thousands of real-world clinical case reports and mapped to the Orphanet rare-disease nomenclature, MMRareBench ensures medical authenticity while enabling standardized diagnosis normalization. Because case-report-based evaluation can be inflated by shortcut cues when answers leak into the context, we build leakage-resistant masked views with per-item leakage_audit records to support auditing and evidence-grounded assessment. The benchmark spans four workflow-aligned clinical tracks: Diagnosis track T1, Treatment Planning track T2, Cross-Image Evidence Alignment track T3, and Examination Suggestion track T4. Performance is measured using a two-level protocol that combines deterministic verification with model-graded rubric scoring. Table 1 summarizes key differences between MMRareBench and existing works. Contributions: (1) We introduce MMRareBench, to our knowledge the first Orphanet-aligned multimodal, multi-image benchmark for rare diseases, constructed from real-world clinical case reports. (2) We define a workflow-aligned evaluation tasks spanning four tracks, with structured targets and track-specific rubrics under rare-disease data scarcity. (3) We benchmark 23 MLLMs and uncover systematic capability gaps under the rare-disease workflow, including a non-trivial treatment-planning bottleneck and a capacity dilution pattern where medical fine-tuning narrows diagnosis error yet degrades cross-image evidence alignment. 2 Related Work Medical benchmarks. Early medical benchmarks focus on text-only, exam-style QA [ 14 ] , but lack grounding in clinical workflows. Recent work introduces structured evaluation via case reports, with MedR-Bench [ 21 ] modeling multi-stage workflows and MedCaseReasoning [ 27 ] emphasizing rationale-based diagnostic inference. Multimodal extensions further expand this landscape: GMAI-MMBench [ 30 ] provides a comprehensive evaluation for general medical AI, while MTBBench [ 25 ] targets sequential clinical decision-making, and MedFrameQA [ 31 ] explores multi-image VQA. However, none of these benchmarks jointly evaluates multimodal and multi-image evidence integration under the rare-disease long tail. Rare-disease benchmarks. Rare-disease evaluation is complicated by long-tail distributions, phenotypic overlap, and limited per-condition data. Existing benchmarks are predominantly text-driven and diagnosis-oriented. RareBench [ 5 ] formulates phenotype-to-code mapping, while ReDis-QA [ 26 ] frames rare-disease diagnosis as question answering over structured and unstructured clinical descriptions. CaseReportBench [ 32 ] emphasizes narrative understanding and diagnostic inference over real-world rare case reports. NOVA [ 4 ] introduces multimodal MRI anomaly detection, but focuses on lesion localization rather than end-to-end diagnostic workflows. None of these benchmarks jointly evaluates multimodal and multi-image evidence integration under the data-scarce rare-disease long tail. MMRareBench fills this gap with workflow-aligned tracks anchored to ORPHAcodes. 3 Method 3.1 Data Source and Curation Pipeline We collect 14,700 PMC case reports published between 2005 and 2025, using the PMCID list provided by [ 27 ] . These reports were processed into 1,756 benchmark items covering 403 rare diseases through a five-stage pipeline. Rare-disease case reports pose distinctive curation challenges because long-tail distributions and overlapping phenotypes demand ontology-anchored filtering that generic clinical NLP pipelines cannot provide. Each report is parsed into typed document blocks, each assigned a persistent identifier that serves as an evidence anchor throughout the pipeline. An LLM-assisted extraction step then identifies imaging modalities, disease mentions, and clinical summaries from these blocks. Disease candidates are matched to the Orphanet rare-disease nomenclature; only cases whose primary diagnosis maps to a confirmed ORPHA code are retained. Documents are then assigned to track-specific buckets, retaining only single-patient reports. Figure 1: MMRareBench overview. (a) Track and modality distribution across 1,756 items. (b) Images per sample (mean = 4.5). (c) Two-level evaluation protocol: L1 model-graded rubric scoring and L2 token-level F1. 3.2 Leakage Control and QA Generation Clinical benchmarks for rare diseases risk becoming trivially solvable when diagnostic conclusions leak into the input context. To enforce genuine, evidence-grounded reasoning, we construct track-specific masked views implemented via three cascading layers. Layer 1 removes metadata and summary sections (title, abstract, conclusion) across all tracks to isolate the clinical narrative. Layer 2 replaces the primary diagnosis and