Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations

Jean-Philippe Corbeil, Asma Ben Abacha, Jérôme Tremblay, Phillip Swazinna · 2025

Read Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations by Jean-Philippe Corbeil, Asma Ben Abacha, Jérôme Tremblay, Phillip Swazinna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26974.txt.

Introduction

Jean-Philippe Corbeil leads six co-authors in reporting a shared task on pulling medical orders from doctor and patient talks. Clinics now record and summarize visits with software, yet turning talk into clear orders for health records is still new ground. The MEDIQA-OE 2025 task is the first open challenge built for that step.

Six teams took part and tested a wide range of methods, using both closed and open large language models. The report lays out the task design, the dataset, the final ranking, and what each team built. For readers in medical language technology, it gives a first shared benchmark and a clear view of which approaches handled spoken orders best.

Catalog record

CREATORJean-Philippe Corbeil, Asma Ben Abacha, Jérôme Tremblay, Phillip Swazinna
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations Jean-Philippe Corbeil, Asma Ben Abacha, Jérôme Tremblay, Phillip Swazinna Abstract Clinical documentation increasingly uses automatic speech recognition and summarization, yet converting conversations into actionable medical orders for Electronic Health Records remains unexplored. A solution to this problem can significantly reduce the documentation burden of clinicians and directly impact downstream patient care. We introduce the MEDIQA-OE 2025 shared task, the first challenge on extracting medical orders from doctor-patient conversations. Six teams participated in the shared task and experimented with a broad range of approaches, and both closed- and open-weight large language models (LLMs). In this paper, we describe the MEDIQA-OE task, dataset, final leaderboard ranking, and participants’ solutions. Table 1: Final Ranking of MEDIQA-OE competition on the test set (100 samples) in which 6 teams participated. Our two baselines (excluded from the ranking) are a simple one-shot prompt for MediPhi-Instruct (1) and GPT-4o-0806 (2) with one example from the training set. Rank Team Name Method Match Desc. Reason Type Prov. AVG 1 WangLab GPT-4 constrained dec. Detailed instructions 81.8 66.8 29.5 81.5 63.0 60.2 2 silver_shaw Gemini 2.5 Pro w/ thinking Detailed plan & instructions 76.4 64.1 41.3 74.7 60.4 60.1 3 MISo KeaneBeanz Qwen3 32B Q4_K_M w/o thinking Instructions w/ 2 shots 73.4 58.0 35.6 71.6 48.4 53.4 4 EXL Health AI Lab MedGemma 27B One shot (short format) 67.7 54.5 30.5 66.2 52.5 50.9 5 MasonNLP Llama4 17B 16E Instruct One shot w/o orders 55.5 39.1 19.8 50.9 41.3 37.8 - Baseline 2 GPT-4o Simple prompt w/ one shot 63.6 39.5 20.4 59.3 1.0 30.1 - Baseline 1 MediPhi-Instruct 3.8B Simple prompt w/ one shot 43.3 25.8 19.5 39.6 13.8 24.7 6 HerTrials Llama3.2 3.2B Instructions w/ one shot 31.2 19.6 9.0 29.6 5.6 15.9 1 Introduction In recent years, the burden of clinical documentation has reduced the time clinicians can devote to direct patient care, and ultimately limited the number of patients physicians can help. To mitigate this, many hospitals and clinics now deploy automatic speech recognition and note summarization tools during consultations. A natural next step in this pipeline is medical order extraction (e.g., medications, labs, imaging, follow-ups) from conversation transcripts to directly populate Electronic Health Records (EHRs). Figure 1: The medical order extraction task takes a doctor-patient dialog and extracts a JSON list of orders containing four keys (description, order_type, reason, and provenance). Orders that were previously prescribed but not explicitly renewed should be excluded (e.g. omeprazole in this example). While named entity recognition (NER) and relation extraction (RE) have been extensively studied in clinical NLP 1 1 1 Natural Language Processing. Xu et al. 2010 ; Doan and Xu 2010 ; Yang et al. 2020 ; Fabacher et al. 2025 ; Henry et al. 2020 ; Lybarger et al. 2023 , extracting actionable, structured orders from full-length consultations remains underexplored despite its potential impact. The task is challenging: inputs are long, dialogues contain interruptions as well as revisions, and outputs combine schema-constrained fields (e.g., order type) with free-text attributes (e.g., description, reason). These challenges are compounded by distributional shifts, as clinicians adapt their language to patients without medical training during consultations. In this era of LLMs Brown et al. ; Achiam et al. 2023 , new approaches have become feasible for the medical order extraction task --- combining long-context reasoning with schema-aware generation --- yet limitations in context length, controllability, and calibration persist. The MEDIQA-OE shared task 2 2 2 https://sites.google.com/view/mediqa-2025 investigates these challenges and benchmarks solutions to improve EHR clinical documentation, which we believe can reduce the burden on providers while ensuring the accurate capture of critical patient orders. 2 Previous Work Tasks similar to order extraction in clinical NLP are commonly formulated as NER and RE. Early systems were rule-based (e.g., MedEx by Xu et al. 2010 ) or used classical machine learning such as support vector machines Doan and Xu 2010 . With pretrained contextual encoders, fine-tuned transformer models (e.g., BERT Devlin et al. 2019 , ClinicalBERT Alsentzer et al. 2019 ) became the standard for NER/RE and delivered consistent gains on clinical benchmarks Yang et al. 2020 ; Fabacher et al. 2025 . More recently, LLMs enable span-free formulations that cast extraction as reading-comprehension style generation. Prompting methods Peng et al. 2023 ; Cui et al. 2023 ; Peng et al. 2024 have shown strong results on several clinical information extraction tasks, including adverse drug events Henry et al. 2020 and social determinants of health Lybarger et al. 2023 . However, order extraction from full patient–doctor dialogues remains underexplored, particularly when models must (i) handle long, multi-speaker inputs and (ii) generate outputs that mix schema-constrained fields (e.g., order type, provenance) with free-text attributes (e.g., description, reason). 3 Methodology 3.1 Source Datasets The long-form doctor-patient conversations used for the order-extraction task are primarily drawn from two datasets: ACI-Bench Yim et al. 2023 and PriMock57 Papadopoulos Korfiatis et al. 2022 . The ACI-Bench corpus comprises 207 naturalistic conversations between physicians and patients, curated by domain experts to reflect real-world clinical interactions. Similarly, the PriMock57 dataset contains 57 mock doctor-patient dialogues, designed to simulate clinical scenarios in a controlled setting. Recent works such as Notechat Wang et al. 2024 has introduced large-scale synthetic dialogue datasets. While this corpus is the largest, we excluded it due to the prevalence of low-quality dialogues we observed. 3.2 Annotations We asked medically trained annotators to produce the gold-standard medical orders for the high-quality conversations of Primock57 and ACI-Bench. Annotation guidelines instructed to assess every medical order of type medication, imaging, lab, or follow-up within the conversation the way a doctor would create them in the EHR. This was intended to replicate doctors’ current process executed at the end of a patient encounter. We measured an inter-annotator agreement of 0.768 . We sampled 100 examples containing 255 medical orders across both data sources as a test set and kept the others as training set (64 samples) used for few-shot prompting, and development set (100 samples) Corbeil et al. 2025 . 3.3 Evaluation We evaluate model performance across four key metrics: description, reason, type, and provenance. Results are reported after performing a matching between reference and hypothesis orders based on description field’s word overlap 3 3 3 Necessary to compare orders with each other. . An intermediary metric, the match score, is computed from this alignment process as the F1 between reference and predicted orders without looking at the content, thus specifically accounting for the amount of fabricated or omitted orders. It represents an upper bound for other metrics that are penalized for empty values for fair comparison. For description and reason metrics, we compute F1 scores of the rouge metric Lin 2004 over unigrams. Type is evaluated using accuracy due to its limited label space, and provenance is assessed via F1 score over provenance labels 4 4 4 Turn numbers where the order originates in the transcript. . Finally, the leaderboard ranking is assessed via the average of all four key metrics: description, reason, type, and provenance. 4 Results 4.1 Leaderboard Ranking We provided in Table 1 the final leaderboard of the MEDIQA-OE along participants’ approaches and our two baselines, which were used as reference

Read the complete text