3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark
Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko · 2025
Read 3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark by Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13861.txt.
Introduction
Ivan Sviridov and co-authors present 3MDBench in 2025, a test bed for AI doctors that chat with patients. Models that read images and text are being tried in telemedicine, but running a full consultation takes more than a good diagnosis. It takes the right questions in the right tone.
The framework runs fake consultations with a patient agent whose temperament varies, and an assessor agent grades the visit. It holds close to three thousand cases across thirty-four diagnoses, drawn from real telemedicine records with images. Chatting with reasoning beat one-shot diagnosis, and feeding in a specialist model's guess lifted scores a lot. The code is open for other teams to use.
Catalog record
| CREATOR | Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko Abstract Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench ( M edical M ultimodal M ulti-agent D ialogue Bench mark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM’s context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench . * * footnotetext: These authors contributed equally to this work. 1 Introduction Telemedicine expands healthcare access and efficiency by enabling real-time consultations and early diagnosis Stoltzfus et al. (2023) . In these consultations, effective communication is essential for diagnostic accuracy and treatment adherence Mirzaei and Kashian (2020) ; Bhaskar et al. (2020) . Large Language Models (LLMs) and Vision-Language Models (LVLMs) further enhance telehealth via real-time analysis Nwankwo et al. (2024) , chronic care management Adeghe et al. (2024) , and decision support Perez et al. (2025) , including symptom assessment, test interpretation, and patient interaction Blinov et al. (2024) ; Lu et al. (2024b) ; Mayer et al. (2024) ; Kumichev et al. (2024) . Figure 1: The patient’s response during the dialogue with the doctor depends on the temperament. Despite recent advances in LLMs’ application in telemedicine, effective doctor-patient dialogue remains essential. Diagnostic quality depends on how patients articulate symptoms, shaped by emotional, cognitive, and systemic factors Amelung et al. (2020) ; Singh and Sittig (2015) . Miscommunication, fear, and low health literacy delay diagnosis Heyhoe et al. (2018) ; Nguyen et al. (2024) , while temperament influences responsiveness, expressiveness, and trust Graedon and Graedon (2014) ; Meyer et al. (2013) . However, existing LLM benchmarks for assessing telemedicine quality offer limited realism by restricting models to multiple-choice tasks Jin et al. (2020) ; Kim et al. (2024b) , imposing factual and non-reactive dialogue templates with artificial interruptions that prevent full consultations Li et al. (2024c) ; Johri et al. (2024) , and omitting image modalities representing patient symptoms Zhu and Wu (2025) . To address these problems, we introduce 3MDBench ( M edical M ultimodal M ulti-agent Dialogue Benchmark) to evaluate LVLM-based consultations in dynamic, realistic scenarios. We use classical temperament theory, which categorizes individuals into four types (sanguine, choleric, melancholic, and phlegmatic) Steiner (1985) , to model the personal characteristics that might affect engagement and medical alliance Paap et al. (2022) ; Hanney et al. (2023) . Hence, our 3MDBench features a Patient Agent , simulating one of four temperament types shown in Fig. 1 , a Doctor Agent , LVLM to conduct the telemedicine consultation and serving as the benchmarked component, and an Assessor Agent , evaluating diagnostic accuracy and communication quality. We select models that best match defined metrics and human annotations to construct these agents. Built on 34 diagnoses obtained from real-world telemedicine consultations using medical image datasets enriched with textual information, 3MDBench supports both textual and image modalities. We benchmark commercial and open-source LVLMs in different dialogue and non-dialogue settings. Our results show that information-seeking dialogue strategies using medical reasoning and image modality increase the F1 score to 6.5%, highlighting the importance of adaptive, context-aware interaction. Finally, our novel approach, incorporating top-3 predictions from a ConvNet trained on diagnosis into the LVLM’s context, boosts the F1 score up to 20%. In this work: 1. We propose 3MDBench , an open-source benchmark for evaluating medical dialogue systems, with an Assessor Agent measuring diagnostic accuracy and communication quality, and a Patient Agent simulating temperament-based personality-driven doctor-patient conversations. 2. Using our standardized framework for assessing AI-driven medical consultation quality, we extensively compare open-source and state-of-the-art LVLMs. 3. We analyze multiple dialogue strategies for a Doctor Agent: with and without image modality, implementing rationale generation, and using external cues. We demonstrate the importance of image modality and conducting information-seeking conversations with internal reasoning. Moreover, we show that the diagnostic F1 score of the Doctor Agent improves up to 20% by incorporating top-3 predictions from a specially trained diagnostic convolutional neural network into the LVLM. Table 1: Comparison of 3MDBench with existing medical benchmarks and datasets. The columns are: T (Type: Dataset (DS) or Benchmark (BM)), TDT (Text Data Type: Question-Answer pairs (QA) or Dialogues (D)), N (Name of Dataset/Benchmark), M (Modality: Text-only (T) or Multimodal (M)), S (Size of test part of a Benchmark of full size of a Dataset), D (Dialogues present), A (Multi-Agent approach used), P (Personality modeling used), CQ (Consultation and communication qualities tested), F (Full-fledged consultation simulated until both agents naturally conclude the dialogue), and L (Language of data). T TDT N M S D A P CQ F L DS D MedDialog-EN Zeng et al. (2020) T 300K + - - - - EN DS D MedDialog-CN Zeng et al. (2020) T 1100K + - - - - CN DS D MedDG Liu et al. (2022) T 18K + - - - - CN DS D CMtMedQA Yang et al. (2023) T 70K + - - - - CN DS D Icliniq-10K Li et al. (2023b) T 10K + - - - - EN DS D / QA BianQueCorpus Chen et al. (2023) T 2437K + - - - - CH DS D / QA HealthCareMagic-100k Li et al. (2023c) T 100K + - - - - EN DS D / QA Psych8k Yuan et al. (2025) T 8K + - - - - EN DS D IMCS-21 Chen et al. (2022) T 811 + + - - - CN DS D NoteChat Wang et al. (2024a) T 30K + + - - - EN DS D MTMedDialog Feng et al. (2025) T 10.1K + + - - - EN BM QA Cholec80-VQA Twinanda et al. (2016) M 9K - - - - - EN BM QA VQA-RAD Lau et al. (2018) M 3.5K - - - - - EN BM QA PathVQA He et al. (2020) M 6K - - - - - EN BM QA SLAKE Liu et al. (2021) M 2K - - - - - EN BM QA RadBench AI (2024) M 137K - - - - - EN BM QA MMMU (H & M) Yue et al. (2024) M 11.5K - - - - - EN BM QA OmniMedVQA Hu et al. (2024) M 128K - - - - - EN BM QA GMAI-MMBench Chen et al. (2024) M 26K - - - - - EN BM QA Medical-Diff-VQA Hu et al. (2025) M 70K - - - - - EN BM D MediQ Li et al. (2024c) T 1.2K + + - - - EN BM D AgentClinic Schmidgall et al. (2024) M 457 + + - -