mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
Ren Yi · 2026
Read mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health by Ren Yi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29467.txt.
Introduction
Yi releases two benchmarks for medical retrieval-augmented generation, systems that fetch guideline text before answering health questions. The focus is maternal, neonatal, and reproductive health, an area that general medical question benchmarks barely cover. The questions reflect what a nurse-midwife actually asks in practice. No public benchmark previously graded how relevant each small text chunk is for such questions.
The first benchmark, mamabench, holds nearly twenty-six thousand questions filtered from seven expert-authored sources. Formats include multiple choice, short answer, and rubric-graded tracks. The second, mamaretrieval, pairs over three thousand clinical queries with graded relevance labels. Labels run from zero to six across a corpus of more than sixty thousand guideline chunks. A decomposed rubric separates chunks that truly answer a query from chunks that are merely on topic. Yi also measures and discloses the limits of the labels themselves.
Catalog record
| CREATOR | Ren Yi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Ren Yi Abstract . Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench’s physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded ( 0 0 – 6 6 ) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels—scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit—rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research. 1. Introduction Medical question-answering benchmarks have proliferated, but maternal, neonatal, child, and reproductive health—and the sub-Saharan, midwife-led primary-care setting in particular—are a thin slice of them. The widely used sets draw on US and Indian licensing exams (MedQA ( Jin et al. 2021 ) , MedMCQA ( Pal et al. 2022 ) ) or broad clinical chat (HealthBench ( Arora et al. 2025 ) ); AfriMed-QA ( Nimo et al. 2025 ) is pan-African but multi-specialty. None is scoped to the maternal, neonatal, child, and reproductive-health questions a nurse-midwife actually asks at the point of care. Retrieval is worse served. Medical retrieval-augmented-generation benchmarks such as MIRAGE ( Xiong et al. 2024 ) score the end-to-end answer drawn from a corpus, not whether an individual retrieved passage is relevant; the classical medical-IR relevance collections (the TREC Clinical Decision Support and Precision Medicine tracks) judge full PubMed articles against case descriptions, not the short guideline chunks a deployed RAG system retrieves and injects into a prompt. To our knowledge there is no public chunk-level relevance benchmark for maternal-health guideline retrieval. We release two benchmarks 1 1 1 Datasets: https://huggingface.co/datasets/nmrenyi/mamabench , https://huggingface.co/datasets/nmrenyi/mamaretrieval . Construction code: https://github.com/nmrenyi/mamabench , https://github.com/nmrenyi/mamaretrieval . Released for research use; some sources are non-commercial (e.g. AfriMed-QA, CC BY-NC-SA 4.0), with each item’s license recorded in its per-source manifest. that close these gaps for the domain. Both were built to supply the test data for a companion system paper ( Ren Yi 2026 ) —a deployed, fully on-device RAG assistant for nurse-midwives in Zanzibar—but each is usable as a standalone instrument. • mamabench (§ 2 ) is a scope-filtered question-answering benchmark: 25,949 items assembled from seven expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks, plus a 6,853-triple judge-calibration side-file—not labels of our own, but HealthBench’s physician annotations re-scoped to this domain—for vetting an LLM judge before it is trusted to score the open-ended items. Its scope classifier is cross-checked by agreement with a larger model and parity against prior labels. • mamaretrieval (§ 3 ) pairs 3,185 clinical queries with graded ( 0 – 6 ) relevance labels over a 63,650-chunk maternal-health guideline corpus—a chunk-level relevance benchmark at the granularity a RAG system actually retrieves, built around a graded clinical-relevance rubric grounded in IR relevance theory, with its judge checked against a frontier reference model. Three decisions shape both benchmarks, and they are the through-line of this paper: (1) Assemble and filter from expert-authored sources rather than author new questions. The clinical curation already exists in licensing exams, physician panels, and published guidelines, so the benchmark’s job is reliable scoping and uniform packaging, not de-novo clinical authoring. (2) Decompose clinical relevance into a graded, multi-dimensional rubric rather than a binary relevant/not-relevant label. To a midwife, a chunk that names a drug and a chunk that gives its dose, route, and threshold are not equally useful, and a benchmark that cannot tell them apart cannot separate strong retrieval from adequate retrieval. (3) Measure and disclose the limits of the labels —scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit—rather than present the benchmark as an oracle. The companion system paper uses these benchmarks to evaluate the deployed system and reports those results; this paper documents how the benchmarks were built and validated , and is what the system paper cites for their provenance. We do not reproduce the deployed-system evaluation here. mamabench is assembled from external QA sources and is corpus-independent, whereas mamaretrieval is coupled to a specific corpus chunk set (§ 3.1 ). Section 4 is candid about what each benchmark does and does not establish; the datasets and construction code are linked in the title footnote. 2. mamabench: A Scope-Filtered QA Benchmark mamabench evaluates how well a model answers the questions a nurse-midwife asks—those spanning maternal, neonatal, child, and reproductive health—in formats from multiple-choice to free-text clinical scenarios. We build it by assembling such questions from seven existing, expert-authored sources and filtering each to that scope. This poses three construction problems: assembling and normalising the sources into one scorable schema, defining and applying the scope reliably, and giving consumers a way to trust the LLM judge that scores the open-ended items. We take each in turn. 2.1. Source assembly The first decision is to assemble, not author: writing and validating maternal-health questions from scratch needs clinicians, whereas the seven sources already carry expert curation—medical-licensing boards, a pan-African physician panel, Kenyan clinicians, and OpenAI’s physician annotators—so the benchmark’s contribution is scoping and uniform packaging, not de-novo clinical authoring. Each source is normalised by a dedicated adapter into one canonical schema and sorted into one of three tracks (Table 1 ): • Multiple-choice (23,241 items), for breadth of clinical knowledge: the OBGYN and paediatrics subset of MedMCQA ( Pal et al. 2022 ) , the in-scope portion of MedQA-USMLE ( Jin et al. 2021 ) , and the multiple-choice questions of AfriMed-QA ( Nimo et al. 2025 ) . This track serves breadth; the open-ended tracks below carry the weight. • Short-answer (369 items), closest to the deployment setting: 312 nurse-written primary-care cases ( Mwaniki et al. 2025 ) , the short-answer questions of AfriMed-QA ( Nimo et al. 2025 ) , and a small set of expert-crafted items designed to expose model errors ( The Lumos AI Labs 2025 ) . • Rubric-graded (2,339 items): the in-scope portion of HealthBench ( Arora et al. 2025 ) , whose items are scored against physician-written rubric criteria rather than a single gold answer. Table 1. The seven sources of mamabench, by track. Yield is the share of a source’s upstream pool that our scope classifier (§ 2.2 ) keeps as in-scope; a dash marks sources that arrive already scoped to the domain—MedMCQA and AfriMed-QA by their subject or specialty labels, the Women’s Health Benchmark by expert authoring—and so bypass the classifier entirely. Track Source Items Yield Multiple-choice MedMCQA (OBGYN & paediatrics) 18,508 – MedQA-USMLE 4,199 29.2% AfriMed-QA 534 – Short-answer Kenya Clinical Vignettes 312 61.5% AfriMed-QA (short-answer) 37 – Women’s Health Benchmark 20 – Rubric-graded HealthBench