3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko · 2025

Read 3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark by Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13861.txt.

Introduction

Ivan Sviridov and co-authors present 3MDBench in 2025, a test bed for AI doctors that chat with patients. Models that read images and text are being tried in telemedicine, but running a full consultation takes more than a good diagnosis. It takes the right questions in the right tone.

The framework runs fake consultations with a patient agent whose temperament varies, and an assessor agent grades the visit. It holds close to three thousand cases across thirty-four diagnoses, drawn from real telemedicine records with images. Chatting with reasoning beat one-shot diagnosis, and feeding in a specialist model's guess lifted scores a lot. The code is open for other teams to use.

Catalog record

CREATORIvan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13861

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko Abstract Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench ( M edical M ultimodal M ulti-agent D ialogue Bench mark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM’s context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench . * * footnotetext: These authors contributed equally to this work. 1 Introduction Telemedicine expands healthcare access and efficiency by enabling real-time consultations and early diagnosis Stoltzfus et al. (2023) . In these consultations, effective communication is essential for diagnostic accuracy and treatment adherence Mirzaei and Kashian (2020) ; Bhaskar et al. (2020) . Large Language Models (LLMs) and Vision-Language Models (LVLMs) further enhance telehealth via real-time analysis Nwankwo et al. (2024) , chronic care management Adeghe et al. (2024) , and decision support Perez et al. (2025) , including symptom assessment, test interpretation, and patient interaction Blinov et al. (2024) ; Lu et al. (2024b) ; Mayer et al. (2024) ; Kumichev et al. (2024) . Figure 1: The patient’s response during the dialogue with the doctor depends on the temperament. Despite recent advances in LLMs’ application in telemedicine, effective doctor-patient dialogue remains essential. Diagnostic quality depends on how patients articulate symptoms, shaped by emotional, cognitive, and systemic factors Amelung et al. (2020) ; Singh and Sittig (2015) . Miscommunication, fear, and low health literacy delay diagnosis Heyhoe et al. (2018) ; Nguyen et al. (2024) , while temperament influences responsiveness, expressiveness, and trust Graedon and Graedon (2014) ; Meyer et al. (2013) . However, existing LLM benchmarks for assessing telemedicine quality offer limited realism by restricting models to multiple-choice tasks Jin et al. (2020) ; Kim et al. (2024b) , imposing factual and non-reactive dialogue templates with artificial interruptions that prevent full consultations Li et al. (2024c) ; Johri et al. (2024) , and omitting image modalities representing patient symptoms Zhu and Wu (2025) . To address these problems, we introduce 3MDBench ( M edical M ultimodal M ulti-agent Dialogue Benchmark) to evaluate LVLM-based consultations in dynamic, realistic scenarios. We use classical temperament theory, which categorizes individuals into four types (sanguine, choleric, melancholic, and phlegmatic) Steiner (1985) , to model the personal characteristics that might affect engagement and medical alliance Paap et al. (2022) ; Hanney et al. (2023) . Hence, our 3MDBench features a Patient Agent , simulating one of four temperament types shown in Fig. 1 , a Doctor Agent , LVLM to conduct the telemedicine consultation and serving as the benchmarked component, and an Assessor Agent , evaluating diagnostic accuracy and communication quality. We select models that best match defined metrics and human annotations to construct these agents. Built on 34 diagnoses obtained from real-world telemedicine consultations using medical image datasets enriched with textual information, 3MDBench supports both textual and image modalities. We benchmark commercial and open-source LVLMs in different dialogue and non-dialogue settings. Our results show that information-seeking dialogue strategies using medical reasoning and image modality increase the F1 score to 6.5%, highlighting the importance of adaptive, context-aware interaction. Finally, our novel approach, incorporating top-3 predictions from a ConvNet trained on diagnosis into the LVLM’s context, boosts the F1 score up to 20%. In this work: 1. We propose 3MDBench , an open-source benchmark for evaluating medical dialogue systems, with an Assessor Agent measuring diagnostic accuracy and communication quality, and a Patient Agent simulating temperament-based personality-driven doctor-patient conversations. 2. Using our standardized framework for assessing AI-driven medical consultation quality, we extensively compare open-source and state-of-the-art LVLMs. 3. We analyze multiple dialogue strategies for a Doctor Agent: with and without image modality, implementing rationale generation, and using external cues. We demonstrate the importance of image modality and conducting information-seeking conversations with internal reasoning. Moreover, we show that the diagnostic F1 score of the Doctor Agent improves up to 20% by incorporating top-3 predictions from a specially trained diagnostic convolutional neural network into the LVLM. Table 1: Comparison of 3MDBench with existing medical benchmarks and datasets. The columns are: T (Type: Dataset (DS) or Benchmark (BM)), TDT (Text Data Type: Question-Answer pairs (QA) or Dialogues (D)), N (Name of Dataset/Benchmark), M (Modality: Text-only (T) or Multimodal (M)), S (Size of test part of a Benchmark of full size of a Dataset), D (Dialogues present), A (Multi-Agent approach used), P (Personality modeling used), CQ (Consultation and communication qualities tested), F (Full-fledged consultation simulated until both agents naturally conclude the dialogue), and L (Language of data). T TDT N M S D A P CQ F L DS D MedDialog-EN Zeng et al. (2020) T 300K + - - - - EN DS D MedDialog-CN Zeng et al. (2020) T 1100K + - - - - CN DS D MedDG Liu et al. (2022) T 18K + - - - - CN DS D CMtMedQA Yang et al. (2023) T 70K + - - - - CN DS D Icliniq-10K Li et al. (2023b) T 10K + - - - - EN DS D / QA BianQueCorpus Chen et al. (2023) T 2437K + - - - - CH DS D / QA HealthCareMagic-100k Li et al. (2023c) T 100K + - - - - EN DS D / QA Psych8k Yuan et al. (2025) T 8K + - - - - EN DS D IMCS-21 Chen et al. (2022) T 811 + + - - - CN DS D NoteChat Wang et al. (2024a) T 30K + + - - - EN DS D MTMedDialog Feng et al. (2025) T 10.1K + + - - - EN BM QA Cholec80-VQA Twinanda et al. (2016) M 9K - - - - - EN BM QA VQA-RAD Lau et al. (2018) M 3.5K - - - - - EN BM QA PathVQA He et al. (2020) M 6K - - - - - EN BM QA SLAKE Liu et al. (2021) M 2K - - - - - EN BM QA RadBench AI (2024) M 137K - - - - - EN BM QA MMMU (H & M) Yue et al. (2024) M 11.5K - - - - - EN BM QA OmniMedVQA Hu et al. (2024) M 128K - - - - - EN BM QA GMAI-MMBench Chen et al. (2024) M 26K - - - - - EN BM QA Medical-Diff-VQA Hu et al. (2025) M 70K - - - - - EN BM D MediQ Li et al. (2024c) T 1.2K + + - - - EN BM D AgentClinic Schmidgall et al. (2024) M 457 + + - -

Read the complete text