CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field

2025

Read CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03441.txt.

Introduction

Doria Bonzi and colleagues introduce CareMedEval, a test set for judging how well language models critique biomedical studies. Critical reading is a core medical skill, and model help there is promising yet still shaky in specialist areas. Questions come from real exams taken by French medical students, not from synthetic prompts.

Five hundred thirty-four questions rest on thirty-seven genuine articles. Unlike older benchmarks, every item demands appraisal grounded in the study at hand. Top general and medical models were tried under varied context setups, and none cleared a modest exact-match bar. The gap shows that fluent summary is not the same as sound critical judgment in medicine.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field Abstract Critical appraisal of scientific literature is an essential skill in the biomedical field. While large language models (LLMs) can offer promising support in this task, their reliability remains limited, particularly for critical reasoning in specialized domains. We introduce CareMedEval, an original dataset designed to evaluate LLMs on biomedical critical appraisal and reasoning tasks. Derived from authentic exams taken by French medical students, the dataset contains 534 questions based on 37 scientific articles. Unlike existing benchmarks, CareMedEval explicitly evaluates critical reading and reasoning grounded in scientific papers. Benchmarking state-of-the-art generalist and biomedical-specialized LLMs under various context conditions reveals the difficulty of the task: open and commercial models fail to exceed an Exact Match Rate of 0.5 even though generating intermediate reasoning tokens considerably improves the results. Yet, models remain challenged especially on questions about study limitations and statistical analysis. CareMedEval provides a challenging benchmark for grounded reasoning, exposing current LLM limitations and paving the way for future development of automated support for critical appraisal. Keywords: critical appraisal, reasoning, evaluation, medical, domain specific dataset, LLM CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field Doria Bonzi 1 , Alexandre Guiggi 2 , Frédéric Béchet 3 , Carlos Ramisch 3 , Benoit Favre 3,4 1 University of Lorraine, LORIA, France 2 University Grenoble-Alpes, France 3 Aix-Marseille University, LIS, France 4 CNRS, Grenoble INP, LIG, France [email protected], [email protected], {frederic.bechet, carlos.ramisch, benoit.favre}@lis-lab.fr Abstract content 1. Introduction Medical professionals must engage in continuous learning to stay up to date with evolving medical knowledge. Even though they can gather knowledge from trusted sources such as Cochrane, they also engage with latest research published in scientific papers often available before peer review in dedicated archives. Critically appraising scientific publications is a complex cognitive task, even for trained physicians. As highlighted by previous studies, proper interpretation of biomedical literature requires not only familiarity with medical content, but also awareness of methodology and statistics ( 7 ) . Moreover, studies have shown that medical research can suffer from major methodological flaws, raising concerns about the reliability and overall trustworthiness of scientific evidence ( 13 ; 3 ). These issues often relate to study design quality and various forms of bias ( 4 ; 5 ). These challenges show why teaching and assessing critical appraisal skills remains a major and ongoing issue in the biomedical field. Natural language processing (NLP), and in particular large language models (LLMs), represents a promising technological solution for supporting this lifelong learning process. Recent development of LLMs has put focus on their "reasoning capabilities", opening new possibilities for supporting medical professionals in the critical reading, analysis, and synthesis of scientific literature. It is important to evaluate the reliability of such technologies. Several evaluation benchmarks already exist in the biomedical domain, but these resources do not explicitly target the evaluation of research methodology or a system’s ability to identify limitations and biases in a study. As a result, they are not well-suited for measuring the specific skills involved in critical appraisal. Despite the growing interest in applying LLMs to these tasks, there remain significant challenges related to hallucinations, bias, and keeping scientific accuracy intact ( 32 ; 35 ; 23 ). Recent work has explored long-context processing in LLMs ( 24 ; 19 ) and their application to the medical field ( 2 ), including retrieval-augmented generation (RAG) systems, where external knowledge sources are integrated into the model’s reasoning process ( 22 ). In these studies, RAG methods make biomedical QA more accurate and robust, yet they mainly support information access and synthesis. The task of critically assessing study design and validity is not well represented in benchmarks, especially when grounded in a given study described in a scientific article. In this work, we introduce (1) a dataset for evaluating critical appraisal of medical studies described in scientific articles 1 1 1 Dataset and code available at https://github.com/bonzid/CareMedEval . , derived from multiple choice question-answers (MCQA) medical education exams in French. With this dataset release, we present (2) a comprehensive evaluation of diverse state-of-the-art language models on this challenging task, providing baseline performance results and insights into model capabilities and limitations. Sample question with context and answer choices taken from the dataset Article, available in PDF or plain text: You are a physician capable of rigorously interpreting data from medical studies. Based on the given article, answer the following multiple-choice question. Provide only the letter(s) corresponding to the correct answer(s) among: A, B, C, D, E. Your answer must strictly follow the exact format: one or more letters, separated by commas (e.g., B, E). Question: What is the main limitation of this study? Answers: a) The 2:1 randomization b) The choice of the primary endpoint c) The number of subjects included d) The amount of antibodies contained in the plasma e) The fact that the study was conducted in Argentina Figure 1: Example from the dataset showing an excerpt from a scientific article, the given instruction prompt with a corresponding question, and answer choices. Unlike existing biomedical question answering datasets which are typically not grounded in research articles, our resource is directly contextualized with authentic scientific publications, aiming to evaluate the information gathering and reasoning capabilities of models. We argue that this dataset is complementary both to factual and RAG-oriented medical benchmarks, by addressing the underexplored dimension of critical appraisal in the medical field. The dataset developed in this work could also support future technologies beyond LLMs, providing a foundation for the development of tools aimed at enhancing medical reasoning and evidence-based decision-making. 2. Related Work Systematic reviewing is a time - consuming process that has recently attracted interest for automation ( 26 , 10 ). However, it has been observed that automation still faces limitations, with human reviewers outperforming current automated methods on such tasks ( 34 ). The main goal of our study is to reflect the challenge of critical appraisal of medical articles, providing a benchmark for evaluating how well models can support both retrieval and critical reasoning in biomedical contexts. To our knowledge, no existing dataset specifically targets the task of critical appraisal and analysis of scientific articles. However, similar resources for the general evaluation of biomedical NLP models are available: for instance, PubMedQA ( 15 ) is an English-language dataset where questions are derived from the abstracts of biomedical research articles; MedQA ( 14 ) offers an open-domain QA benchmark in English, simplified Chinese, and traditional Chinese, based on medical textbooks; SciDQA ( 30 ) provides a collection of questions grounded in full-length scientific articles with figures and images. However, none of these datasets explicitly focus on the critical evaluation of research methodology. They primarily assess factual comprehension or domain knowledge, rather than the ability to review a scientific article in terms of study design, methodology or limitations. In contrast, our dataset is specifically designed to capture and evaluate these skills through multiple-choice questions grounded in medical literature. While the French biomedical NLP landscape still lacks a dataset that aims to explore critical appraisal of scientific articles, other resources exist for adjacent tasks: MedFrenchmark ( 29 ) and FrenchMedMCQA ( 17 ) focus on MCQA in the medical

Read the complete text