MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
Adrien Bazoge · 2025
Read MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation by Adrien Bazoge (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20917.txt.
Introduction
Adrien Bazoge wrote this paper, published in two thousand twenty-five, to present MediQAl, a French data set for testing medical question answering. Most such test sets are in English, which leaves a gap for other languages.
MediQAl holds more than thirty-two thousand questions drawn from French medical exams, spread over forty-one subjects. It comes in three forms: single-answer choice, multi-answer choice, and open questions with short answers. Each question is also tagged as testing plain recall or true reasoning. Bazoge ran fourteen large language models on the set. Scores dropped sharply on the reasoning questions compared with recall. The paper gives builders a hard, public yardstick for medical AI in French. It also shows how far even strong models sit from exam-level clinical reasoning.
Catalog record
| CREATOR | Adrien Bazoge |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20917
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation Adrien Bazoge Abstract This work introduces MediQAl, a French medical question answering dataset designed to evaluate the capabilities of language models in factual medical recall and reasoning over real-world clinical scenarios. MediQAl contains 32,603 questions sourced from French medical examinations across 41 medical subjects. The dataset includes three tasks: (i) Multiple-Choice Question with Unique answer, (ii) Multiple-Choice Question with Multiple answer, and (iii) Open-Ended Question with Short-Answer. Each question is labeled as Understanding or Reasoning , enabling a detailed analysis of models’ cognitive capabilities. We validate the MediQAl dataset through extensive evaluation with 14 large language models, including recent reasoning-augmented models, and observe a significant performance gap between factual recall and reasoning tasks. Our evaluation provides a comprehensive benchmark for assessing language models’ performance on French medical question answering, addressing a crucial gap in multilingual resources for the medical domain. 1 Introduction Medical licensing examinations, originally designed to assess students’ knowledge and reasoning, are increasingly repurposed as benchmarks for evaluating large language models (LLMs) medical capabilities Yan et al. 2024 . Benchmarks of question-answering tasks predominantly rely on multiple-choice questions (MCQs) with a single correct answer Hendrycks et al. 2021 ; Wang et al. 2024 . This format is widely used due to the availability of automatic evaluation metrics that provide consistent and objective assessment of LLMs at scale. While MCQ-based datasets and metrics provide a valuable initial insight into LLM performance, they are often limited in several key aspects: the number of examples, the diversity of medical subjects covered, their representation of real-world clinical scenarios Shi et al. 2024 , and the range of languages represented. Indeed, most of existing benchmarks are heavily centered around English, which restricts their applicability to multilingual or non-English contexts Yan et al. 2024 . Furthermore, medical benchmarks inherently reflect cultural, educational, and regulatory contexts in which they are developed. The format of questions and answers mirrors how medicine is taught and assessed in their respective countries, which differs in structure, emphasis, and evaluative expectations across regions. In addition, treatment guidelines, clinical protocols, and legal standards are often country-specific, meaning that identical questions translated across languages can pose entirely different challenges. Recent efforts have focused on increasing diversity in question difficulty and covering a wider variety of medical subjects Zuo et al. 2025 , yet these benchmarks remain predominantly limited to English-language and rely on MCQs with a single correct answer. This limitation is particularly problematic, as several studies have demonstrated significant performance disparities between languages, with LLMs performing considerably better in English compared to less-resourced languages Jin et al. 2024 ; Dey et al. 2024 ; Alonso et al. 2024 . Therefore, it is crucial to develop more inclusive benchmarks that cover a broader range of languages and are more reflective of real-world clinical scenarios, ensuring a fair and comprehensive evaluation of LLMs in the medical domain. In this work, we present MediQAl, a medical question answering dataset for French. This dataset contains questions sourced from French medical licensing examinations. These are manually created by academic and hospital faculty members to reflect real-world clinical scenarios and cover a broad range of medical subjects. MCQU MCQM OEQ Understanding Reasoning Total Understanding Reasoning Total Understanding Reasoning Total Total Number of Questions 11,336 5,681 17,017 7,742 2,875 10,617 1,842 3,125 4,969 # Isolated Questions 9,126 961 10,087 6,200 343 6,543 836 179 1,015 # In-context Questions 2,210 4,720 6,930 1,542 2,532 4,074 1,006 2,946 3,954 Avg Question Length 18.95 21.57 19.82 13.20 16.12 13.99 16.79 20.95 19.40 Avg Clinical Scenarios Length 83.50 107.67 99.97 94.87 114.77 107.24 109.71 141.28 132.19 Avg Answer Length - - - - - - 25.26 40.24 34.68 Table 1: Characteristics of the MediQAl dataset. In-context questions refers to questions including a clinical scenario. Lengths are measured in words. This paper makes the following contributions: 1. We introduce MediQAl, a French medical question answering (QA) dataset that includes three tasks : (i) Multiple-Choice Question with Unique Answer (MCQU), (ii) Multiple-Choice Question with Multiple Answers (MCQM) and (iii) Open-Ended Question with Short-Answer (OEQ). 2. MediQAl covers a total of 41 medical subjects and each question is categorized as either Understanding or Reasoning , enabling detailed analysis of LLMs’ capabilities across different cognitive tasks. 3. We present an extensive evaluation of 14 large language models (LLMs) on MediQAl, including latest reasoning-based models, providing a comprehensive benchmark for assessing their performance over real-world clinical scenarios. We compare different groups of models, focusing on the performance gap between vanilla models (non-reasoning) and their reasoning-enhanced counterparts. The dataset is available on HuggingFace 1 1 1 https://huggingface.co/datasets/ANR-MALADES/MediQAl under CC-BY-4.0 license and all evaluation scripts are available on Github 2 2 2 https://github.com/abazoge/MediQAl . 2 Related Work Multiple-choice question answering with a unique correct answer (MCQU) is a well-established task, frequently used to benchmark language models. This task is particularly prominent in the medical domain, where several datasets have been developed in various languages. In English, multiple high-quality datasets exist, including HEAD-QA Vilares and Gómez-Rodríguez 2019 , MedQA Jin et al. 2021 , MedMCQA Pal et al. 2022 , MMLU (Medical) Hendrycks et al. 2021 ; Wang et al. 2024 , and more recently, MedXpertQA Zuo et al. 2025 , an expert-level benchmark for medical MCQU tasks. In other languages, efforts have been made to extend the task to non-English settings. Notable examples include datasets for Chinese Li et al. 2021 , Polish Bean et al. 2024 and Spanish Alonso et al. 2024 . However, for French, the resources remain scarce. FrenchMedMCQA Labrak et al. 2022 is a dataset containing 3,105 multiple-choice questions, with both unique and multiple answers, but is limited to pharmacy topics. Another MCQU dataset, MedExpQA Alonso et al. 2024 , includes a French subset that is translated from Spanish. In addition to multiple-choice datasets, open-ended question answering in the medical domain is less common, as evaluating free-text responses is more challenging and often requires manual human validation. Some open-ended QA datasets are derived from existing multiple-choice corpora. For instance, MEDQA-OPEN Nachane et al. 2024 reformulates MedQA questions into an open-ended format. For French, there is only a single small-scale open-ended QA dataset, MedFrenchmark Quercia et al. 2024 , containing only 114 examples. 3 MediQAl We introduce MediQAl, a French medical dataset consisting of questions sourced from French medical examinations. MediQAl is designed to evaluate medical knowledge and reasoning on both isolated and in-context questions reflecting real-world clinical scenarios. The dataset includes three subsets, corresponding to distinct question answering tasks: (1) Multiple-Choice Question with unique answer (MCQU), (2) Multiple-Choice Question with multiple answers (MCQM) and (3) Open-Ended Question with a short answer (OEQ). MediQAl contains a total of 32,603 questions, of which 17,017 are MCQU, 10,617 are MCQM and 4,969 are OEQ. These questions span 41 medical subjects and are categorized as Understanding or Reasoning , offering a diverse and reliable benchmark for medical question answering tasks in French. Table 1 summarizes the main characteristics of the dataset. Figure 1: Distribution of medical subjects across MediQAl dataset. 3.1 Tasks Definition Multiple-Choice Question with Unique Answer (MCQU) This task can be formulated as X=\{C,Q,(O_{1},...,O_{5}),A\} where C is an optional clinical scenario, Q is the question,