A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks
Heriberto Cuayáhuitl, Grace Jang · 2026
Read A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks by Heriberto Cuayáhuitl, Grace Jang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26747.txt.
Introduction
Cuayahuitl and Jang release MeDial-Speech, a spoken dataset of robot-patient and doctor-patient consultations covering Lewy body dementia, heart failure, shoulder pain, and angina. Recorded in realistic settings, it holds over 111 hours of speech for training and evaluating medical dialogue systems. A sentence-selection benchmark finds Claude Sonnet 4 most accurate, yet shows all tested models highly overconfident in their predictions whether their chosen sentences are right or wrong.
Catalog record
| CREATOR | Heriberto Cuayáhuitl, Grace Jang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19228 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26747
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks Heriberto Cuayáhuitl, Grace Jang Abstract Large Language Models (LLMs) have brought huge improvements to Artificial Intelligence (AI), which can be applied to general-purpose tasks. However, their application to textual or spoken medical consultations is still an open research problem. This paper proposes MeDial-Speech , a novel speech dataset for training and evaluating Med-AIs that can carry out consultations with patients. It was collected in realistic environments from robot-patient and doctor-patient dialogues, contains 111+ hours of speech data (without data augmentation), and covers four health conditions: Lewy body dementia, heart failure, shoulder pain, and angina. In addition, we propose a dialogue benchmark via sentence selection (with 20 options) to evaluate three state-of-the-art LLMs: GPT-5 mini, DeepSeek-V3, and Claude Sonnet 4. Experimental results reveal that Claude Sonnet 4 is the best in sentence selection, with 71.1% accuracy using manual transcriptions and 74.7% using automatic transcriptions, and that all LLMs are highly overconfident in their probabilistic predictions, regardless of selecting correct or incorrect sentences in medical dialogues. This dataset is free of charge for non-commercial purposes. † † address: Author Affiliation(s) 1 Motivation Effective communication between doctors and patients is an essential aspect of the UK’s NHS core value, ‘commitment to quality of care’ [ 1 ] . Exposure to medical consultations is crucial for trainees—whether human or machine—in the medical field to enhance their competence in communication, clinical reasoning, and professionalism. This paper introduces a new dataset of doctor-patient and robot-patient medical dialogues, with the intention of aiding both medical AIs and students in clinical skills training outside of clinical placement. In addition, this dataset aims to contribute to the development of automated clinicians, as in Fig. 1 , with the ability to manage non-urgent conditions. These types of efforts have the potential to alleviate the workload of healthcare professionals since human clinicians can direct their attention towards patients with more critical needs or other attention-demanding tasks. Figure 1 : An actor patient interacting with a teleoperated robot. 2 Previous Works Speech-based medical datasets publicly available are scarce. Some exceptions include [ 23 ] , who used four participants for a total of 9 consultations with audio and video recorded, [ 10 ] , who collected and annotated 272 human-human consultations, [ 20 ] , who reported a dataset of 57 primary care consultations, [ 4 ] , who collected and annotated 327 human-human consultations, and [ 8 ] , who reported 2,000 hours of speech. Whilst only the latter two involve real patients, they are not publicly available and require purchase. Automatic Speech Recognition (ASR) efforts have been made in this realm, but the speech data is either closed source [ 6 ] or only from human-human conversations with data/synthetic augmentation [ 2 , 16 ] . Their performance ranges between 12% and 30% in word error rate, and these ASR models have not been evaluated using interactive systems and human participants. To our knowledge, there is no publicly accessible, free-of-charge dataset of spoken medical consultations that includes both human–machine and human–human interactions (and with multiple levels of annotations), both of which are arguably valuable for developing an AI doctor. Medical LLMs, on the other hand, are trained from large text-based datasets such as [ 14 , 13 , 11 ] . Whilst they have shown remarkable abilities in classification and discriminative tasks [ 26 , 25 ] , they are not specialised for interactive medical consultations. Nonetheless, medical LLMs and speech-based medical data (as proposed in this work) will play a crucial role in the development of future conversational AIs, including robots for healthcare delivery [ 24 , 22 , 15 , 9 ] . 3 The Data Collection System The Pepper robot [ 19 ] was equipped with a teleoperated system using in-person and remote embodied telepresence. In this setup, the teleoperator was embodied in the body of the robot by hearing and seeing what the robot perceives and by saying what the robot conveys to the person in front of it. This Wizard-of-Oz (WOZ) setting is illustrated in Figure 1 , where the human doctor (teleoperator) used a separate room from the robot and the patient. We avoided the use of GUIs as in [ 21 , 5 ] to prevent predefined responses and to encourage personalised, human-like communication. Instead of using the teleoperator’s actual speech, we used continuous speech recognition via Vosk (model: vosk-model-en-us-0.22, 2.6GB), together with automatic punctuation of FastPunct and speech synthesis via Acapela . This means that as the teleoperator uttered a sentence, automatic speech recognition generated words actively and indefinitely. Whenever a two-second pause was detected, the recognised and generated sentence was automatically punctuated. The latter was useful for producing more intelligible speech than without punctuation. After punctuation, a motor command was generated to execute robot speech concurrently with body movements (gestures: chosen randomly from a set of 18 pre-recorded motions of robot arms and head related to greeting, explain, yes, no, and please). Other non-verbal behaviours included face tracking (to follow patient gaze) and head nodding (for acknowledgements) during human speech. The medical doctors—all medical students—were able to hear and see at all times during a dialogue via continuous audio and video streaming on the laptop in front of them. Additional technical features in this robot system are: (1) audio recorded with sampling rate at 16 kHz, 16-bit, and 1 channel; (2) audio produced at 44 kHz with 2 channels, TTS system with speed=100, pitch shift=1.5, and volume=60%; (3) video at 10 images per second, 640x480 pixels; (4) robot and laptop connected via Ethernet. The dynamics of our conversational robot system are briefly described as follows. The teleoperator’s speech was automatically transcribed via ASR and punctuation, then used to produce robot speech via speech synthesis. Robot speech is what the patients were able to hear, i.e., they never heard the actual speech of the human doctor. The patient’s speech and video were continuously captured by the robot without interruption during a consultation. These two streams of data were perceived by the human doctor in near real time through headphones and the laptop screen. The latter displayed the patient’s video and the transcriptions of the doctor’s speech. This procedure is illustrated in Figure 1 , which began by specifying the participant ID and then pressing Enter to indicate the start of a consultation. A consultation terminated once the teleoperator said the phrase “let’s move on.” Finally, human-human consultations were recorded using MS Teams . 4 The Speech Dataset 4.1 Data Collection We collected data from 325 recruited and unpaid participants, mostly university students and from different schools (but mainly the Schools of Medicine and Computer Science at UoL ) with age categories 18-24 (87.1%), 25-34 (8.9%), and 35+ (4.0%). All participants were fluent speakers of English, including native & non-native speakers of both genders. Each participant read and signed a consent form and reviewed a task description and a simulated patient profile prior to the consultation. Patient profiles included the following health conditions: Lewy-body dementia, heart failure, shoulder pain, and angina. Since our participants were simulated patients instead of real ones, they were allowed to keep a printed sheet describing