MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge
Perry E. Radau Department of Radiology, University of Calgary Calgary, AB T2N 1N4, Canada Child and Adolescent Imaging Research (CAIR) Program Alberta Children’s Hospital Research Institute University of Calgary Calgary, AB T3B 6A9, Canada [email protected] · 2026
Read MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge by Perry E. Radau Department of Radiology, University of Calgary Calgary, AB T2N 1N4, Canada Child and Adolescent Imaging Research (CAIR) Program Alberta Children’s Hospital Research Institute University of Calgary Calgary, AB T3B 6A9, Canada [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05175.txt.
Introduction
Perry Radau introduces MRI-Eval, a benchmark testing language models on MRI physics and on operating GE scanners, knowledge central to research scanning practice. Existing benchmarks lean on review-book multiple-choice questions where top models already score near the ceiling. His benchmark instead uses primary questions across nine categories and three difficulty levels, drawn from textbooks, scanner manuals, course materials, and expert-written items.
Five model families were tested, all scoring high on multiple choice, with scanner operations the weakest category for every model. When answer options were removed, accuracy fell sharply, and scanner operations collapsed furthest. Primed questions also tested whether models accept incorrect claims from users, adding a diagnostic angle.
Catalog record
| CREATOR | Perry E. Radau Department of Radiology, University of Calgary Calgary, AB T2N 1N4, Canada Child and Adolescent Imaging Research (CAIR) Program Alberta Children’s Hospital Research Institute University of Calgary Calgary, AB T3B 6A9, Canada [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05175
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge Perry E. Radau Department of Radiology, University of Calgary Calgary, AB T2N 1N4, Canada Child and Adolescent Imaging Research (CAIR) Program Alberta Children’s Hospital Research Institute University of Calgary Calgary, AB T3B 6A9, Canada [email protected] Abstract Abstract Background: The prior LLM benchmark for MRI knowledge uses review-book questions on which a subset of top proprietary models scored very well, offering limited discriminative power. No systematic evaluation has tested LLM performance on vendor-specific scanner operational content central to research MRI work. Purpose: To develop and validate MRI-Eval, a tiered benchmark for relative model comparison on MRI physics and GE scanner operations knowledge through primary multiple-choice questions (MCQ), with stem-only (free-text, options removed) and primed diagnostic conditions as complementary analyses. Methods: MRI-Eval comprises 1365 scored multiple-choice items across nine categories and three difficulty tiers, drawn from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five models from distinct families (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B) were evaluated. MCQ was the primary evaluation format. A complementary stem-only condition removed MCQ options and scored free-text responses by an independent LLM judge (four models, with Gemini excluded due to mandatory reasoning tokens). A primed stem-only diagnostic condition tested responses to incorrect user claims. Results: Overall MCQ accuracy ranged from 93.2% to 97.1% (3.9 percentage point [pp] spread). GE scanner operations was the lowest-scoring category for every model (88.2–94.6%). Removing options reduced accuracy for the four proprietary frontier models to 58.4–61.1% (35.5–36.8 pp difference). Llama 3.3 70B dropped to 37.1% (56.1 pp difference). GE scanner operations stem-only accuracy fell to 13.8–29.8%, showing that near-ceiling MCQ performance did not translate into strong free-text recall for this vendor-specific content. Run-to-run repeatability was near-perfect for all four tested models (Cohen’s kappa [ 2 ] 0.983–0.998). Under the primed diagnostic condition, incorrect answer suggestions increased overall accuracy for all models, consistent mainly with answer-cueing effects. Conclusion: MRI-Eval shows that strong MCQ performance on MRI content can mask weak generative recall, especially for GE-specific operational knowledge. The benchmark is most informative as a relative model-comparison tool rather than as an absolute competency measure, particularly in the absence of human calibration. These findings support caution against relying on raw LLM outputs for GE-specific protocol guidance and motivate testing retrieval-augmented or document-grounded systems as a next step. Keywords: large language models, magnetic resonance imaging, MRI physics, benchmark evaluation, vendor-specific scanner operations, clinical workflow support Introduction Large language models (LLMs) are increasingly consulted as reference tools in MRI research settings, for protocol design, sequence parameter selection, artifact troubleshooting, and physics education. Whether these models possess sufficient domain-specific knowledge to be reliable in such roles has not been systematically evaluated. The only MRI-specific LLM benchmark to date [ 7 ] assessed several models on 570 multiple-choice questions drawn from a 1995 technologist review book. Within that benchmark, the top-performing model (o1 Preview) scored 94%, with GPT-4o and o1 Mini at 88% and Claude 3.5 Haiku at 84%, indicating that some top proprietary models have approached the ceiling of review-book content. That benchmark used a single source targeting technologist board-exam preparation, included no vendor-specific operational content or tiered difficulty, and did not evaluate whether high MCQ scores reflect knowledge or option recognition. General-purpose benchmarks such as MMLU-Pro [ 12 ] and GPQA [ 10 ] employ similar tiered difficulty and expert-crafted question designs. MRI-Eval applies the same principles to a specialized domain where current frontier models have not previously been tested. General biomedical benchmarks such as MedQA, MedMCQA, and PubMedQA [ 5 , 6 , 9 ] assess clinical reasoning but do not cover MRI physics, image formation, or scanner-specific operational knowledge. No existing benchmark evaluates the content central to day-to-day research MRI work: vendor-specific protocol parameters, pulse sequence behavior, and scanner console operations. This gap is consequential because MRI protocol decisions directly affect image quality, scan efficiency, and patient safety. This study introduces MRI-Eval, a tiered benchmark comprising 1365 scored multiple-choice items across nine categories and three difficulty tiers, drawn from textbooks, GE scanner application manuals, programming course materials, and expert-generated questions. The benchmark was designed to locate knowledge boundaries (particularly the boundary between general MRI physics and vendor-specific operational content) rather than confirm competence, and to identify domains where LLMs lack the knowledge that MCQ scores suggest they possess. The present study is intended as a controlled model-to-model benchmark of relative performance across evaluation formats, not as a calibration against human expert performance or a direct estimate of deployment readiness. Five models from distinct provider families (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B) were evaluated. In this paper, ‘‘frontier models’’ refers to GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, and Gemini 2.5 Pro. Llama 3.3 70B serves as an open-weight comparison model. MCQ was the primary evaluation format. A complementary stem-only evaluation removed MCQ options and adjudicated free-text responses by an independent LLM (four models with Gemini 2.5 Pro excluded due to mandatory reasoning tokens), directly measuring whether MCQ performance reflects generative recall or option recognition. A primed stem-only diagnostic condition tested model responses to incorrect user claims that might elicit recall or sycophancy 1 1 1 We use the field’s term for “yes man” but note our preference for Seneca’s assentator , while the true friend amicus tells us uncomfortable truths. . The principal contributions are: (1) a tiered benchmark incorporating GE vendor-specific scanner content underrepresented in LLM training corpora; (2) a complementary stem-only evaluation alongside the primary MCQ benchmark, quantifying the gap between MCQ performance and knowledge, plus a primed stem-only diagnostic condition probing responses to framed user claims; (3) a five-model comparison with pairwise statistical testing and repeatability assessment; and (4) public release of the evaluation harness and reproducibility artifacts. The following sections describe benchmark construction and evaluation methodology, present results across all evaluation conditions, and discuss implications for the use of LLMs in research MRI settings. Methods The MRI-Eval benchmark comprised 1365 scored items (1355 multiple-choice questions and 10 misconception questions) drawn from six source types covering nine clinical and technical categories. Sources were selected for authoritative grounding: three textbooks [ 13 , 3 , 11 ] provided answer-keyed physics and applied content. Two vendor-specific sources (the GE SIGNA Works scanner application manual [ 4 ] and GE EPIC Programming Course materials) supplied operational content underrepresented in general training corpora. Thirty-five author-generated questions addressed expert-knowledge gaps not covered by existing sources. Questions were authored and validated by the author, a medical physicist with active MRI research experience on GE systems. AI-assisted structural screening was applied across the full bank as a partial external check on question quality; flagged items were reviewed and resolved by the author against primary sources. (See Appendix F.) Each item was assigned to exactly one of nine categories: GE scanner operations and protocol (n = 406), pulse sequences (n = 330), GE domain knowledge (n = 155), safety (n = 142), k-space and image formation (n = 113), artifacts (n = 80), T1/T2 relaxation and contrast (n = 67), SNR and image quality (n = 52), and parallel imaging (n = 20). Items were also