MedCalc-Bench Doesn’t Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluation

Artus Krohn-Grimberghe · 2026

Read MedCalc-Bench Doesn’t Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluation by Artus Krohn-Grimberghe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02222.txt.

Introduction

Artus Krohn-Grimberghe audits a medical math benchmark in this 2026 paper. The benchmark, MedCalc-Bench, tests language models on clinical calculator tasks. Scores for direct answers have stalled near a third correct on its checked split. The author asks whether the test really measures clinical reasoning at all.

A full audit found over twenty faults in the benchmark's own calculators. Faults ranged from wrong formulas to code bugs that broke runs. Giving models the formula sheet at test time lifted scores sharply, past all prior results. No extra training was needed for that gain. A top model with the sheet nears the ceiling, and left-over errors trace mostly to flawed answer keys. The test, he argues, mostly measures formula recall and arithmetic.

Catalog record

CREATORArtus Krohn-Grimberghe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32294
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MedCalc-Bench Doesn’t Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluation Artus Krohn-Grimberghe Abstract MedCalc-Bench is a widely used benchmark for evaluating LLM performance on clinical calculator tasks, with state-of-the-art direct prompting scores plateauing around 35% on the Verified split (HELM MedHELM leaderboard) and the best published approach—RL with verifiable rewards—reaching 74%. We present three contributions that challenge the benchmark’s current framing. First, we conduct a systematic audit of the benchmark’s calculator implementations, identifying and fixing over 20 errors ranging from critical formula inaccuracies to runtime bugs in a NeurIPS-published dataset. Second, we show that a simple intervention—providing the model with the calculator specification at inference time (“open-book” prompting)—raises accuracy from ∼ 52 {\sim}52 % to 81–85% on GLM-4.6V and GLM-4.7, surpassing all published results including RL-trained systems, without any fine-tuning. Third, we establish an upper bound of 95–97% using GPT-5.2-Thinking, with residual errors attributable primarily to ground-truth issues and dataset ambiguities. Our findings suggest that MedCalc-Bench predominantly measures formula memorization and arithmetic precision rather than clinical reasoning, and would be better framed as a tool-use evaluation. 1 Introduction Clinical calculators—standardized scoring systems such as CKD-EPI, APACHE-II, MELD, and the Glasgow Coma Scale—are routine instruments in hospital medicine. They transform patient observations into risk scores, dosage adjustments, or staging classifications. MedCalc-Bench (NeurIPS 2024 Datasets Track) operationalizes 55 such calculators as an LLM evaluation task: given a clinical vignette, the model must identify the relevant calculator, extract the required parameters, and compute the correct result. On the Stanford HELM MedHELM leaderboard, the best direct-prompting results reach only 34.8% accuracy on MedCalc-Bench Verified. Specialized approaches—RL with verifiable rewards (MedCalc-R1), agentic tool frameworks (RiskAgent), and extended thinking with code execution (Claude Opus 4.5)—push this to 61–74%, but the task remains among the hardest in clinical NLP benchmarking. We start from a simple observation: in clinical practice, no physician memorizes the APACHE-II formula. They use a calculator. The benchmark’s difficulty may therefore stem less from clinical reasoning and more from an artificial constraint—requiring the model to recall, from parametric memory, the exact formula and execute multi-step arithmetic on real-valued inputs (logarithms, exponents) where even minor precision errors cascade outside the benchmark’s acceptance band. This paper makes three contributions: 1. Benchmark audit. We identify and document over 20 errors in the official calculator implementations, including incorrect formula coefficients (CKD-EPI, MELD-Na, Child-Pugh), missing scoring criteria (SIRS, SOFA), key typos that silently zero out score components (CCI, Cardiac Risk Index), and broken file paths. Several of these errors affect ground-truth labels in the evaluation set. 2. Open-book prompting. We propose a minimal intervention: appending the calculator’s specification (formula, parameter definitions, scoring rules) to the prompt. With no fine-tuning, no retrieval augmentation, and optional code execution, this raises GLM-4.6V from 52% to 81.5% and GLM-4.7 from 36% (vanilla) to 85.5% on a 275-row subsample—exceeding all published results. 3. Upper-bound analysis with GPT-5.2-Thinking. We run GPT-5.2-Thinking (xhigh reasoning effort, with search and tools) on the 198 rows that GLM-4.6V fails under open-book prompting. It resolves 140/198, with a subsequent audit attributing 19 of the remaining 58 errors to ground-truth or implementation issues and 10 to genuine ambiguity. This yields a conservative upper-bound accuracy of 94.7% and an optimistic bound of 97.4%. Taken together, these results suggest that MedCalc-Bench, as currently constituted, is primarily a test of formula recall and decimal arithmetic under tight tolerance conditions ( \pm 5% for equation-based; exact match for rule-based)—not of clinical reasoning. We argue it should be repositioned as a tool-use benchmark, where the evaluation focuses on parameter extraction and appropriate calculator and unit converter selection rather than formula memorization. 2 Background and Related Work 2.1 MedCalc-Bench MedCalc-Bench was introduced at NeurIPS 2024 as a dataset of clinical calculator tasks spanning 55 calculators in two categories: equation-based (continuous-valued outputs evaluated within \pm 5% tolerance) and rule-based (exact categorical match). MedCalc-Bench Verified is the maintained successor with corrected labels. The benchmark provides three evaluation settings: zero-shot, one-shot, and code-augmented. 2.2 Consolidated Leaderboard Table 1 synthesizes all published MedCalc-Bench results we could identify through a structured literature survey using seven different research-capable LLM systems (OpenAI Deep Research, Gemini Deep Research, Kimi K2 Deep Research, Qwen 3, Grok 4 Expert, GPT-5.2-Pro). Results were consolidated and cross-validated by GPT-5.2-Pro with reference verification by GPT-5.2-Thinking and Claude Opus 4.5. Table 1: Published MedCalc-Bench results, sorted by accuracy descending. Results span different dataset variants (Orig, Verified, Cleaned) and are not directly comparable; see Section 2.1 for variant descriptions. Rank Approach Model/System Dataset Setting Accuracy Src 1 RL + verifiable rewards DeepSeek-R1 (MedCalc-R1) Orig (train/test 9,765/1,048) SFT + GRPO 73.95% [ 1 ] 2 RL on recomputed labels Qwen3-8B Orig (recomputed; test n =1,047) GRPO 71.4% [ 2 ] 3 RL + verifiable rewards o1-mini Orig (train/test 9,765/1,048) SFT + GRPO 67.84% [ 1 ] 4 Agentic tool framework RiskAgent-GPT-4o Orig (external eval) Zero-shot CoT + tools 67.71% [ 3 ] 5 Extended thinking + tools Claude Opus 4.5 Bench (company eval) 64k thinking + Python 61.3% [ 4 ] 6 Step-wise eval diagnostic GPT-4o Cleaned & restructured Step-wise final-answer 62.7% \to 43.6% [ 5 ] 7 Agentic + step-wise eval MedRaC Cleaned & restructured Retrieval + Python up to 53.19% [ 5 ] 8 RL + verifiable rewards MedCalc-R1 (3B) Orig (train/test 9,765/1,048) SFT + GRPO 51.34% [ 1 ] 9 Prompting baseline GPT-4 Orig (test 1,047) One-shot CoT 50.9% [ 6 ] 10 Supervised fine-tuning Mistral-7B Orig (train split) SFT 49.19% [ 7 ] 11 Code-exec prompting GPT-4 Orig Zero-shot CoT + code 48.51% [ 7 ] 12 Supervised fine-tuning Llama-2-7B Orig (train split) SFT 45.75% [ 7 ] 13 Direct prompting (HELM) DeepSeek-R1 / Gemini 2.5 Pro Verified Zero-shot 34.8% [ 8 ] 14 Code-exec prompting GPT-3.5 Orig Zero-shot CoT + code 30.29% [ 7 ] 2.3 Why Is This Benchmark Hard? The low scores invite decomposition. The task requires three capabilities in sequence: 1. Parameter extraction —identifying the correct clinical values from a vignette, anchored to the right admission date when multiple encounters are described. 2. Formula recall —retrieving the exact scoring rules or equations from parametric memory, including version-specific details (e.g., MELD vs. MELD 3.0, CKD-EPI 2009 vs. 2021). 3. Arithmetic execution —computing results involving unit conversions, logarithms, exponents, and real-valued coefficients to within the benchmark’s \pm 5% tolerance band. Failure at any step cascades to an incorrect final answer. Prior work has focused on improving steps 2 and 3 through RL training or code execution. We focus on the simpler question: what happens if you eliminate step 2 entirely? 3 Benchmark Audit 3.1 Methodology We conducted a systematic review of all 55 calculator implementations in the MedCalc-Bench repository. Each implementation was cross-referenced against its primary clinical source (original publication, MDCalc documentation, or clinical guideline). Test suites were written for each calculator. Errors were identified through a combination of automated testing, manual review, and LLM-assisted code analysis (GPT-5.2-Thinking, GPT-5.2-Pro, Claude Opus 4.5). 3.2 Error Taxonomy We identified and fixed over 20 errors across four categories. Table 2 summarizes the findings. Table 2: Calculator implementation errors by category. Category Count Examples Logic & formula errors 9 CKD-EPI 2021 male coefficient ( 0.7\to 0.9 ); MELD-Na updated to MELD 3.0; HOMA-IR unit conversion (

Read the complete text