Automatic Replication of LLM Mistakes in Medical Conversations
Oleksii Proniakin, Diego Fajardo, Ruslan NazarenkoRazvan Marinescu[0.2cm] Lumos AI · 2025
Read Automatic Replication of LLM Mistakes in Medical Conversations by Oleksii Proniakin, Diego Fajardo, Ruslan NazarenkoRazvan Marinescu[0.2cm] Lumos AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20983.txt.
Introduction
Oleksii Proniakin, Razvan Marinescu, and two co-authors automate the hunt for medical mistakes by language models. Checking whether a model's specific error recurs in other models normally takes manual work. Their pipeline, MedMistake, stages conversations between a model patient and a model doctor. A two-judge committee grades the exchanges across reasoning, safety, and patient-centeredness. Each failure is then distilled into a single-shot question.
The full release holds 3,390 question pairs that two frontier models currently fail, and medical experts validated a 211-question subset as the benchmark proper. Twelve leading models were run against it, with the GPT, Claude, and Grok families performing best. Both datasets are public, giving clinical-AI evaluators a ready-made stress test built from real conversational failures rather than exam questions.
Catalog record
| CREATOR | Oleksii Proniakin, Diego Fajardo, Ruslan NazarenkoRazvan Marinescu[0.2cm] Lumos AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 151 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automatic Replication of LLM Mistakes in Medical Conversations Oleksii Proniakin, Diego Fajardo, Ruslan NazarenkoRazvan Marinescu[0.2cm] Lumos AI Abstract Large language models (LLMs) are increasingly evaluated in clinical settings using multi-dimensional rubrics which quantify reasoning quality, safety, and patient-centeredness. Yet, replicating specific mistakes in other LLM models is not straightforward and often requires manual effort. We introduce MedMistake , an automatic pipeline that extracts mistakes LLMs make in patient-doctor conversations and converts them into a benchmark of single-shot QA pairs. Our pipeline (1) creates complex, conversational data between an LLM patient and LLM doctor, (2) runs an evaluation with a committee of 2 LLM judges across a variety of dimensions and (3) creates simplified single-shot QA scenarios from those mistakes. We release MedMistake-All , a dataset of 3,390 single-shot QA pairs where GPT-5 and Gemini 2.5 Pro are currently failing to answer correctly, as judged by two LLM judges. We used medical experts to validate a subset of 211/3390 questions ( MedMistake-Bench ), which we used to run a final evaluation of 12 frontier LLMs: Claude Opus 4.5, Claude Sonnet 4.5, DeepSeek-Chat, Gemini 2.5 Pro, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, Grok 4, Grok 4.1, Mistral Large. We found that GPT models, Claude and Grok obtained the best performance on MedMistake-Bench . We release both the doctor-validated benchmark MedMistake-Bench , as well as the full MedMistake-All dataset at https://huggingface.co/datasets/TheLumos/MedicalMistakeBenchmark . Figure 1: Overview of the MedMistake-Bench pipeline. We first synthesize conversations using MedPI [ 3 ] (blue), then extract mistakes from those conversations which we distill into single-shot QA pairs (green), and finally we run a medical expert validation (yellow). 1 Introduction Evaluating LLM models in conversational settings [ 11 , 2 ] typically focuses on holistic outcomes — assigning a single score at the end of a multi-turn dialogue. This mirrors how we might evaluate a physician by the overall quality of care, not by every utterance made during an encounter. Such conversation-level assessments, as used in frameworks like HealthBench [ 14 ] , can provide a fair and clinically aligned measure of medical competence. However, this outcome-based evaluation creates a critical gap when moving from assessment to improvement. Post-training refinement and safety tuning require pinpointing where in a conversation reasoning broke down — which turn, decision, or assumption led to a low overall score. Without per-turn granularity, evaluators and model developers face a “black box” problem: we know that an interaction failed, but not how or why. A variety of recent works [ 4 , 5 , 1 , 16 , 22 ] evaluated LLMs on various tasks involving multi-turn clinician-patient conversations. Among these, [ 10 , 16 , 5 ] used LLM patients, all except [ 4 ] used LLM doctors, and [ 10 , 16 ] used LLM judges. However, most of these works stopped at evaluating conversations, without providing a way to distill those mistakes into single-shot QA pairs towards the creation of a mistake benchmark. In addition, the number of dimensions evaluated in these studies was limited: [ 4 ] evaluated dialogue summarization quality on 4 dimensions (coherence, consistency, fluency and relevance), [ 5 ] evaluated synthetic dialogue realism on 7 dimensions (medical accuracy, realism, persona consistency, fidelity to prompt, empathy, relevancy and usability), [ 10 ] evaluated conversational diagnostic accuracy on 3 dimensions (accuracy, history-taking completeness, conversation adequacy), [ 16 ] evaluated patient consultation quality and broke this down into inquiry quality, response quality and safety. However, for significantly improving LLMs and related foundation models, one needs not only to evaluate conversations and identify mistakes, but further distill these mistakes into single-shot QA pairs, which could be used as a benchmark or for fine-tuning the next-generation models. Figure 2: Example snippet from a generated medical conversation between an LLM patient and LLM doctor, where the LLM doctor makes a mistake in a drug prescription. The mistake is identified by a medical committee of 2 LLM judges, and a single-shot clinical scenario is generated that is used to score LLMs. We propose MedMistake , an agentic LLM pipeline that automatically creates single-shot QA pairs on medical knowledge from LLM mistakes detected during complex medical conversations. To identify mistakes, we first run a series of LLM conversations between an LLM patient and LLM doctor, then create a medical committee of 2 LLM judges to evaluate the conversation on 40 dimensions to point out specific mistakes and their location in the conversation. From the committee’s feedback, we extract structured medical reasoning mistakes and create single-shot QA pairs on medical knowledge that can be used to test the model’s understanding of the medical domain. Our contributions are as follows: 1. We synthesize 3390 mistakes from LLM-based patient-doctor conversations spanning a total of 40 dimensions 2. We used medical experts to manually validate a subset of 299/3390 mistakes, of which 211/299 unique mistakes were confirmed to be valid ( MedMistake-Bench ) 3. We evaluated 12 frontier LLMs on MedMistake-Bench – Claude Opus 4.5, Claude Sonnet 4.5, DeepSeek-Chat, Gemini 2.5 Pro, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, Grok 4, Grok 4.1, Mistral Large – finding that GPT models, Claude and Grok obtain the best performance. 4. We release MedMistake-All – a dataset of 3390 single-shot QA pairs where GPT-5 and Gemini 2.5 Pro are currently failing to answer correctly, as judged by two LLM judges. We also release MedMistake-Bench , a dataset of 211 single-shot QA pairs that were validated by medical experts. 1.1 Related work: Single-turn medical QA benchmarks. Most LLM benchmarks on medical tasks focus on single-turn QA. A large body of work has applied this paradigm to clinical knowledge and exam-style questions, for example MedQA [ 8 ] , MedMCQA [ 15 ] , PubMedQA [ 9 ] and MultiMedQA [ 18 ] . These benchmarks have been critical in showing that LLMs encode substantial clinical knowledge and can approach or exceed physician-level performance on written exam questions [ 19 ] . Medical evaluation frameworks beyond single-turn QA. More recent work broadens the evaluation paradigm from the pure knowledge single QA testing to multi-task and safety-oriented evaluation. MedHELM evaluates performance across question answering, summarization, information extraction, and safety-oriented tasks under a unified reporting framework [ 2 , 13 ] . HealthBench focuses on realistic and safety-critical healthcare scenarios, combining knowledge, reasoning, and safety checks across diverse tasks and settings [ 14 , 7 ] . MedSafetyBench [ 6 ] zooms in further on medical safety failure modes, systematically probing how models handle contradictions, unsafe advice, and other risk patterns. Automatic mistake synthesis: Several recent works [ 17 , 20 , 21 , 12 ] built pipelines that automatically synthesize mistakes that language models typically make. [ 17 ] introduced MWP-MISTAKE , a dataset of math word problems with both correct and incorrect reasoning steps generated by rule-based algorithms and small language models. [ 20 ] released BIG-Bench Mistake , a dataset of LLM-generated logical mistakes represented as chain-of-thought traces from PaLM-2 with annotated error locations. [ 21 ] introduced LLM-Attack , which generates adversarial examples for LLMs using a two-stage approach involving (1) a word ranking step and (2) a synonym replacement step. [ 12 ] introduced TableEG , a framework that fine-tunes LLMs to