HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning

2025

Read HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15355.txt.

Introduction

Alexis Correa-Guillén, Carlos Gómez-Rodríguez, and David Vilares release HEAD-QA v2, a greatly expanded benchmark of healthcare exam questions for testing language models. The data set draws on ten years of Spanish professional licensing exams, in biology, medicine, nursing, and allied fields, with English versions alongside. It now holds well over twelve thousand multiple-choice questions that demand real reasoning, not recall alone.

The team benchmarks several open-source models under plain prompting, retrieval-augmented reading, and probability-based answer choice. Scale and raw reasoning ability drive the scores, while elaborate inference strategies add little. A dependable bilingual benchmark matters because clinical question answering is exactly where confident wrong answers carry real-world cost.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15355

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning Abstract We introduce HEAD-QA v2, an expanded and updated version of a Spanish/English healthcare multiple-choice reasoning dataset originally released by 41 . The update responds to the growing need for high-quality datasets that capture the linguistic and conceptual complexity of healthcare reasoning. We extend the dataset to over 12,000 questions from ten years of Spanish professional exams, benchmark several open-source LLMs using prompting, RAG, and probability-based answer selection, and provide additional multilingual versions to support future work. Results indicate that performance is mainly driven by model scale and intrinsic reasoning ability, with complex inference strategies obtaining limited gains. Together, these results establish HEAD-QA v2 as a reliable resource for advancing research on biomedical reasoning and model improvement. Keywords: Multi-choice question answering, LLMs, Healthcare HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning Alexis Correa-Guillén, Carlos Gómez-Rodríguez, David Vilares Universidade da Coruña, CITIC Departamento de Ciencias de la Computación y Tecnologías de la Información Campus de Elviña s/n 15071, A Coruña, Spain {[email protected], carlos.gomez, david.vilares}@udc.es Abstract content 1. Introduction HEAD-QA (v1) 41 is a Spanish/English multiple-choice healthcare dataset designed to evaluate model reasoning abilities. It comprises 6,765 questions from official exams issued between 2013 and 2017. It was conceived as a step toward more demanding benchmarks, following the rise of early reading comprehension datasets such as SQuAD 35 , SNLI 5 , and the AI2 Reasoning Challenge 9 , among others, as well as the neural architectures developed for them 22 ; 6 . Notably, experimental results revealed that these architectures lacked the capacity to reason effectively about diagnostic knowledge and failed to capture definitions and domain-specific concepts essential for accurate inference, often performing worse than simple information retrieval baselines. More specifically, HEAD-QA consists of multiple-choice questions modeled after Spain’s competitive specialization exams 28 , which are used to evaluate and rank graduates in fields such as medicine (MIR), nursing (EIR), biology (BIR), chemistry (QIR), psychology (PIR), and pharmacy (FIR). These highly demanding exams require months or even years of preparation, as their results determine both the specialization and the training location where candidates complete the final 3–5 years of residency before becoming fully qualified professionals. The dataset has since gained notable adoption, having been used to evaluate influential architectures and models such as RMKV 32 , Falcon 31 and OLMo 16 , to investigate data reliability in both open-source and proprietary systems 14 , and to develop and assess specialized solutions in the medical domain 45 ; 42 . It has also served as a precursor to similar medical QA datasets in other languages, including Chinese 24 and French 23 , extending its influence on healthcare question answering research. In the current context, the landscape of question answering and reasoning has changed profoundly with the rise of large language models (LLMs) 29 ; 17 ; 13 ; 25 ; 15 ; 44 . These models have advanced substantially in reasoning, knowledge integration, and domain adaptation through instruction tuning and retrieval-augmented generation (RAG). This shift has redefined what constitutes a challenging benchmark—spanning domains such as coding 47 , Ph.D.-level knowledge 34 , machine translation 2 and multimodal reasoning 30 —and has led to an explosion of datasets 36 ; 26 . Contribution We present HEAD-QA v2, an expanded and updated version designed to better reflect the era of large-scale reasoning models. The new release addresses the limited size and temporal coverage of its predecessor by incorporating 12,751 multiple-choice questions from Spanish professional medical qualification exams—more than doubling the dataset and extending its time span. We expect this expansion to enable future research on model generalization, knowledge retention, and temporal effects to a greater extent than its predecessor. We further establish new baselines through a systematic evaluation of open-source LLMs, exploring multiple inference strategies, including prompting, retrieval-augmented generation, and a probability-based approach. Together, we expect that these contributions offer a practical benchmark for studying how LLMs adapt to domain evolution, balance accuracy with efficiency, and perform complex reasoning in specialized contexts. The dataset is available at https://huggingface.co/datasets/alesi12/head_qa_v2 . 2. Dataset Construction This section outlines the construction of HEAD-QA v2, which, like its predecessor, is based on official, publicly available exams from the Ministerio de Sanidad de España. Each exam includes: (i) a two-column PDF containing the text, (ii) a CSV file listing the correct answers, and (iii) when applicable, a folder with referenced images indexed numerically (e.g., 1, 2, 3, 4, …), enabling text–image alignment. 1 1 1 Questions containing images are relatively rare, and visual processing is therefore excluded from this work. 2.1. Preprocessing The preprocessing pipeline involves converting, cleaning, and standardizing the exam data. 1. PDF to text conversion. Exams were converted from PDF to plain text using pdftotext , preserving the two-column layout. 2. Image mapping. Images were automatically linked, as related questions begin with “Question linked to image no. X,” where X is the image identifier. 3. Question filtering. Questions without an official answer from the Spanish Ministry of Health were removed, as they correspond to disputed or withdrawn items. 4. Manual corrections. Minor edits to fix errors and standardize content. Chemical formulas were converted to SMILES notation (see Figure 1 ) using Mathpix , facilitating processing by text-based ML models 37 ; 7 . The few affected questions were processed manually. Figure 1 : Example of chemical formula converted to SMILES notation for text processing. 5. Storage. Files are stored in Parquet format 38 for efficient compression and fast download. 2.2. Format Each question includes eight fields (Figure 2 ). A unique identifier requires both name (exam name) and qid (question ID). • qid (int): Question number within the exam. • qtext (str): Question text. • ra (int): Correct answer identifier. • answers (list): Answer options, each with: – aid (int): Option ID. – atext (str): Option text. • image (Image): Associated image in PIL format 8 , or null if none. • year (int): Exam year. • category (str): Discipline (e.g., Medicine, Nursing). • name (str): Exam identifier combining year, discipline, and version (e.g., Cuaderno_2013_0_B). ⬇ {’ qid ’: 1, ’ qtext ’: ’ Excitatory postsynaptic potentials :’, ’ ra ’: 3, ’ answers ’: [{’ aid ’: 1, ’ atext ’: ’ Are all - or - none responses .’}, {’ aid ’: 2, ’ atext ’: ’ Are hyperpolarizing .’}, {’ aid ’: 3, ’ atext ’: ’ Can be summed .’}, {’ aid ’: 4, ’ atext ’: ’ Propagate over long distances .’}, {’ aid ’: 5, ’ atext ’: ’ Exhibit a refractory period .’}], ’ image ’: None , ’ year ’: 2013, ’ category ’: ’ biology ’, ’ name ’: ’ Cuaderno_2013_1_B ’} Figure 2 : A HEAD-QA v2 question in JSON format. 2.3. Dataset statistics The dataset contains a total of 12,751 questions distributed across six disciplines and ten years (see Table 1 ). Among them, 334 questions include images. Of these, 36 correspond to the four most recent nursing exams (2019–2022), while the rest belong to the medical exams—with over 30 image-based questions per test until 2018, and around 25 per test in subsequent years. ’13 ’14 ’15 ’16 ’17 ’18 ’19 ’20 ’21 ’22 Total BIR 227 225

Read the complete text