How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts
Minh-Vuong Nguyen Fatemeh Shiri Zhuang Li Karin Verspoor · 2026
Read How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts by Minh-Vuong Nguyen Fatemeh Shiri Zhuang Li Karin Verspoor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11133.txt.
Introduction
Nguyen and three colleagues introduce ClinicNumRobBench, a benchmark for robust clinical numeracy. Safe deployment requires reliable handling of patient measurements in varied notes. Existing evaluations cover mainly math and rarely assess robustness across note formats. The benchmark holds 1,624 instances testing retrieval, math, and totals. It presents longitudinal vital-sign records in three equivalent representations, including a note-style variant. Tests on 17 models show finding values is strong, while comparison and totals remain hard. Medical fine-tuning can cut scores by over 30 percent.
Catalog record
| CREATOR | Minh-Vuong Nguyen Fatemeh Shiri Zhuang Li Karin Verspoor |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54979 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts Minh-Vuong Nguyen Fatemeh Shiri Zhuang Li Karin Verspoor Abstract Large Language Models (LLMs) are increasingly being explored for clinical question answering and decision support, yet safe deployment critically requires reliable handling of patient measurements in heterogeneous clinical notes. Existing evaluations of LLMs for clinical numerical reasoning provide limited operation-level coverage, restricted primarily to arithmetic computation, and rarely assess the robustness of numerical understanding across clinical note formats. We introduce ClinicNumRobBench , a benchmark of 1,624 context–question instances with ground-truth answers that evaluates four main types of clinical numeracy: value retrieval , arithmetic computation , relational comparison , and aggregation . To stress-test robustness, ClinicNumRobBench presents longitudinal MIMIC-IV vital-sign records in three semantically equivalent representations, including a real-world note-style variant derived from the Open Patients dataset, and instantiates queries using 42 question templates. Experiments on 17 LLMs show that value retrieval is generally strong, with most models exceeding 85% accuracy, while relational comparison and aggregation remain challenging, with some models scoring below 15%. Fine-tuning on medical data can reduce numeracy relative to base models by over 30%, and performance drops under note-style variation indicate LLM sensitivity to format. ClinicNumRobBench offers a rigorous testbed for clinically reliable numerical reasoning 1 1 1 Code and data URL are available on https://github.com/MinhVuong2000/ClinicNumRobBench . 1 Introduction Large Language Models (LLMs) have exhibited rapid exploration of medical applications such as clinical question answering and decision support ( 4 ) . Yet safe clinical use critically requires LLMs to handle patient measurements reliably, which depends on both clinical numeracy , executing basic numerical operations over clinical measurements, and clinical robustness , preserving numerical correctness under semantically equivalent variations in how the same measurements are documented across clinical notes. First, despite substantial progress on mathematical reasoning ( 37 ) , prior numeracy evaluations show that LLMs still exhibit persistent weaknesses in fundamental operations, including value retrieval , arithmetic computation , relational comparison , and aggregation ( 22 ; 26 ) . Meanwhile, clinical text is numerically rich, with numbers appearing in most documents across widely used clinical datasets ( 25 ) , and these same operations are repeatedly required in clinical workflows over patient measurements, making numerical reliability a prerequisite for safe deployment. Second, clinical numeracy differs from canonical mathematical word problems with systematic symbolic structure. Clinical measurements are embedded in diverse and noisy contexts, and the same value may appear in structured fields, semi-structured templates, or free-text notes with varied surface forms (e.g., “BP: 120/80” vs. “blood pressure of 120 over 80”) ( 35 ) . As a result, even when an LLM succeeds in one presentation, its numerical behavior may be brittle under semantically equivalent documentation variants. A clinically meaningful evaluation of LLMs must therefore test both fine-grained numerical operation correctness and robustness under representational shifts where the underlying measurements remain unchanged. However, existing datasets do not jointly address these requirements. General numeracy benchmarks 25 ; 26 ; 22 are mostly non-clinical, while clinical benchmarks often cover only a subset of operations and omit robustness testing. For instance, MedCalc-Bench ( 21 ) is derived from patient notes but mainly evaluates arithmetic computation and Electromyogram Table Mart (ETM) ( 24 ) focuses on table-to-text diagnosis generation rather than operation-level clinical numeracy benchmarking. Neither consider robustness under note-style variation. A recent study ( 13 ) examined LLMs’ sensitivity to non-clinical input perturbation, with altered patient attributes causing inconsistent treatment recommendations and demographic disparities. While the work focuses on the fairness of LLMs with the change of non-clinical input, the robustness of LLMs on diverse clinical documentation formats remains unexplored, especially numerical reasoning being fundamental to clinical decision-making. To address this gap, we introduce ClinicNumRobBench , a benchmark of 1,624 context–question instances for fine-grained clinical numeracy evaluation and robustness testing, together with a scalable synthetic construction pipeline. Grounded in real-world MIMIC-IV vital-sign records and demographics ( 19 ; 20 ; 18 ) , our pipeline automatically constructs longitudinal patient contexts by sampling and assembling records, then renders the same underlying measurements in three semantically equivalent context representations, from structured formats to realistic natural-language variants. We instantiate 42 question templates to generate queries probing value retrieval , arithmetic computation , relational comparison , and aggregation operations, and compute ground-truth answers programmatically. Using ClinicNumRobBench, we evaluate 17 LLMs across three categories and report three findings in task accuracy, medical fine-tuning negative impact, and context robustness. Our main contributions are: • We introduce ClinicNumRobBench , a benchmark of 1,624 instances for evaluating clinical numeracy across 4 operations and robustness across 3 context representation formats. • We propose a scalable construction pipeline grounded in MIMIC-IV longitudinal records that automatically builds the benchmark with minimal human effort. • Experiments on 17 LLMs reveal persistent weaknesses in comparison and aggregation, potential numeracy degradation from medical fine-tuning, and strong sensitivity to clinical note representational shifts. 2 Related Work Clinical numeracy evaluation. Most numeracy benchmarks evaluate LLMs using general mathematics questions, ranging from grade school arithmetic to Olympiad level problems ( 9 ; 15 ; 22 ) . Despite strong performance on difficult problems, LLMs can still fail simple operations such as comparison or multiplication ( 22 ; 26 ) . Clinical text is numerically rich, yet widely used medical benchmarks such as MedQA, MedXpertQA, and MedBullets ( 16 ; 43 ; 5 ) assess broad clinical knowledge and reasoning, making it hard to isolate numeracy. 25 identified prevalent types of numerical information in clinical documents, highlighting the lack of systematic resources in the clinical domain for numerical understanding and reasoning. Recently targeted studies on medical calculations and table interpretation ( 21 ; 24 ) focus on limited formulas or modalities and do not systematically test value retrieval, arithmetic computation, relational comparison, and aggregation. Robustness to documentation variation. LLMs are sensitive to small changes in prompt wording or formatting ( 41 ; 42 ; 3 ) . This has motivated robustness benchmarks such as GSM Plus ( 23 ) , FinBias ( 29 ) , and CARES ( 7 ) . In the medical domain, robustness studies mainly target knowledge-oriented adversarial prompts ( 30 ) , or non-clinical input perturbations affecting clinical recommendations ( 13 ) , and rarely examine numerical robustness, even though clinical measurements appear in structured EHR fields, semi-structured templates, and free text notes. Reliable deployment therefore requires testing whether numerical accuracy holds under semantically equivalent documentation variants. 3 ClinicNumRobBench Figure 1: Overview of benchmark construction. In this section, we present the Clinic al Num eracy Rob ustness Bench mark ( ClinicNumRobBench ) and its construction pipeline, which is used to evaluate numerical reasoning of LLMs in clinical settings and their robustness to semantically equivalent documentation variants. Each instance consists of a patient context , a question , and an exact ground-truth answer. ClinicNumRobBench integrates (i) multiple clinical context representations and (ii) template-based tasks that target core numerical operations. We then detail context construction, task and question generation, and dataset statistics. A detailed comparison with existing benchmarks, particularly the arithmetic-focused MedCalc-Bench, appears in Table 2 and Appendix B . 3.1 Context Construction Real world clinical measurements appear in heterogeneous formats, from structured representations (e.g., JSON,