EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation
2026
Read EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14306.txt.
Introduction
Causio and nineteen colleagues describe EuropeMedQA, the first full multilingual and multimodal medical exam dataset drawn from official exams in Italy, France, Spain, and Portugal. Built on FAIR rules and SPIRIT-AI guide with strict care and an automated translation flow, the dataset supports side-by-side analysis. Current multimodal models face zero-shot prompting under strict constraints to test cross-lingual transfer and visual reasoning. The test aims to resist leaks, reflect clinical complexity in Europe, and foster more general medical AI.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27072 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14306
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation Abstract While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI. Francesco Andrea Causio 1,2 , Vittorio De Vita 1,2 , Olivia Riccomi 2 , Michele Ferramola 8 , Federico Felizzi 2 , Alessandro Tosi 2 , Antonio Cristiano 1,2 , Lorenzo De Mori 2,3 , Chiara Battipaglia 2 , Melissa Sawaya 2 , Luigi De Angelis 2,4 , Marcello Di Pumpo 1,2 , Alessandra Piscitelli 2,9 , Pietro Eric Risuleo 1,2 , Alessia Longo 7 , Giulia Vojvodic 2,5 , Mariapia Vassalli 2,11 , Bianca Destro Castaniti 2,10 , Nicolò Scarsi 1,2,6 , Manuel Del Medico 1,2 * Affiliations 1. Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy 2. Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy 3. Department of Mental Health, Addiction Treatment Unit, ASL RM 4, Bracciano, Italy 4. Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy 5. Università Cattolica del Sacro Cuore, Rome, Italy 6. Department of Medicine and Surgery, University of Perugia 7. Université Paris Cité, Paris, France 8. NSBproject, Mantova, Italy 9. Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy 10. Department of Diagnostic Imaging, Oncological Radiotherapy and Hematology, Università Cattolica del Sacro Cuore, Rome, Italy 11. Department of Experimental Medicine, Section of Medical Pathophysiology, Food Science and Endocrinology, Sapienza University, Rome, Italy *Corresponding author: [email protected] 1. INTRODUCTION The increasing interest in applying large language models to medicine is driven in part by their impressive performance on medical exam questions, with models such as Med-PaLM 2 and GPT-4 achieving passing scores on the United States Medical Licensing Examination [ 1 , 4 ] . However, these examinations may not fully capture the complexity of real patient–doctor interactions, the synthesis of diverse medical literature, or the nuanced clinical decision-making required in practice [ 2 , 7 ] . Furthermore, LLMs produce varying outcomes when evaluated on items from different countries and contexts, owing to disparities in disease prevalence, clinical guidelines, terminologies, and question formats across regions [ 1 , 9 , 13 ] . For instance, while GPT-4 excels on USMLE-style questions, performance drops notably on non-English exams like Polish medical licensing tests or Spanish benchmarks, revealing biases toward English-centric training data and limited generalizability [ 1 , 9 ] . The proliferation of foundation models, both open-source and proprietary, further complicates the selection of models best suited for specific clinical applications [ 12 ] . Although numerous studies have assessed model accuracy on established datasets like MedQA and MedMCQA, the widespread online availability of these resources poses a significant risk of training data contamination, artificially inflating performance metrics for LLMs [ 2 , 12 , 3 ] . Synthetic question generation has been proposed to circumvent this issue, yet it often fails to replicate the real-world complexity, cultural nuances, and clinical appropriateness of official exams [ 10 ] . To address these challenges—including data contamination, English bias, lack of multimodality, and insufficient diversity in exam contexts—we intend to develop the EuropeMedQA dataset. EuropeMedQA positions itself distinctly in the literature as the first comprehensive European-focused, multilingual, multimodal dataset from real regulatory exams, bridging gaps left by predominantly English or non-European benchmarks like USMLE-derived MedQA, Indian MedMCQA, or global efforts like MultiMedQA [ 2 , 5 ] . While prior works highlight LLMs’ strengths on text-only US exams, they underscore limitations in cross-lingual transfer, visual reasoning, and handling country-specific practices—issues EuropeMedQA directly tackles [ 1 , 8 ] . This benchmarking analysis study aims to investigate: 1. The distribution of medical question topics and their intrinsic characteristics across diverse European national contexts. 2. The performance of contemporary multimodal LLMs in accurately answering questions from the EuropeMedQA dataset. Furthermore, the study will comparatively evaluate the performance of both text-based and multimodal models on this novel dataset, providing insights into the added value of multimodal capabilities for complex medical question answering. 2. MATERIALS AND METHODS The EuropeMedQA dataset will be developed in accordance with the FAIR data principles, ensuring transparency, reproducibility, and reusability in artificial intelligence research [ 11 ] . Consistent with the SPIRIT-AI guidelines [ 6 ] , the dataset architecture, data provenance, and annotation procedures will be systematically documented to enhance transparency in model development and evaluation. Comprehensive metadata enabling full reproducibility of the dataset construction process will be made publicly available alongside the dataset (e.g., via a dedicated GitHub repository). 2.1 Study Design The EuropeMedQA study is designed as a prospective dataset collection and validation project, enabling both supervised and semi-supervised model training. The primary aim is to create a robust dataset that includes diverse multimodal data types and languages, facilitating comprehensive model evaluation in medical knowledge-related questions and answering. 2.2 Search Strategy and Eligibility Criteria We will search online databases and Google for medical licensing examinations and residency admission exams in Italy (Scuola di Specializzazione Medica, SSM), France (Examen Classant National, ECN), Spain (Médico Interno Residente), and Portugal (Prova Nacional de Acesso, PNA). 2.3 Inclusion Criteria • Questions and examinations issued by a central government institution (e.g., Ministry of Education or Ministry of Health) or by officially recognized regulatory authorities responsible for the evaluation of medical professionals upon entering the medical profession (medical licensing) or before enrolling in a medical residency (medical residency admission). • Questions available in the original language and in unedited form. • Text-based questions and image-based questions where the related images are available. • Image-based questions where corresponding images can be retrieved from official sources (ministerial documents and residency preparation materials containing authentic test questions). 2.4 Exclusion Criteria • Questions issued from unofficial sources (e.g., schools preparing doctors for medical licensing examinations). • Image-based questions where corresponding images are not retrievable from official sources or not available. The study flow is summarized in Figure 1 . Internet search for medical licensing and residency admission tests Retrieve original test Data extraction Extraction revision Dataset cleaning Dataset processing English translation (GPT-4o mini), validated Final dataset in original languages Further analysis (additional subsets TBD) LLM API evaluates each question (English dataset) LLM API evaluates each question (original languages) COMPARISON Figure 1: Study flow diagram. After dataset processing, three parallel tracks are produced: an English-translated version, the original-language dataset, and any additional subsets. LLM evaluation is performed on both the English and original-language tracks, and results are compared. 2.5 Data Collection and Extraction The dataset will consist of multimodal data, including structured information from various medical domains, ensuring multilingual diversity by