Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guideliness

Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee, Eunho Yang · 2026

Read Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guideliness by Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee, Eunho Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03627.txt.

Introduction

Jean Seo and seven co-authors introduce EPAG, a benchmark for judging how well language models handle the pre-consultation stage of medical care. Before meeting a doctor, a patient describes their history and symptoms, and a good system should follow diagnostic guidelines from that account. EPAG tests models both by comparing their interview output with guidelines and by checking final diagnoses.

One striking result: small open models fine-tuned on a well-curated, task-specific dataset beat frontier models at this job. More patient history did not always improve diagnosis, either. The language of the consultation also changed the dialogue's character. The team released the dataset and pipeline openly. For clinics, the lesson is that careful local tuning can outrank raw model size.

Catalog record

CREATORJean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee, Eunho Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guideliness Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee, Eunho Yang Abstract We introduce EPAG , a benchmark dataset and framework designed for E valuating the P re-consultation A bility of LLMs using diagnostic G uidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG , we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings. 1 Introduction Large Language Models (LLMs) are increasingly integrated into clinical applications, transforming healthcare industry by automating various tasks ( 38 ; 42 ; 30 ; 35 ) . One example is pre-consultation, where LLMs assist history-taking ( 34 ; 39 ) and decision-making ( 25 ; 18 ) . However, it is crucial to acknowledge the significant risks involved. Erroneous outputs can result in severe adverse consequences such as mistreatment or incorrect drug prescription, highlighting the necessity of rigorous evaluations ( 15 ; 32 ) . Figure 1: EPAG pipeline. (1) Dialogue Generation : The patient-agent acts as a patient given a specific profile, while the doctor-agent conducts a pre-consultation using only the basic information and chief complaint. After n turns, the doctor-agent is assessed through two tasks: (2) HPI-Diagnostic Guideline Comparison , where the organizer model extracts HPI units and the comparer model determines which of the diagnostic guidelines is most relevant, and (3) Disease Diagnosis , where the dialogue is given to a separate diagnostician-agent for diagnosis. We propose EPAG ( E valuating the P re-consultation A bility of LLMs using diagnostic G uidelines), a benchmark dataset and evaluation pipeline specifically designed for pre-consultation. Given basic patient information, such as age, sex, and chief complaints, pre-consultation models ask questions to elicit symptoms related to potential diagnoses. EPAG benchmark dataset comprises 520 patient profiles, spanning 26 diseases, 10 ICD-11 chapters, 10 primary specialties, and 22 secondary specialties, along with pre-defined diagnostic guidelines. In EPAG, the pre-consultation dialogue is evaluated through two tasks: (1) HPI-Diagnostic Guideline Comparison, and (2) Disease Diagnosis. In our experiments, eleven LLMs are evaluated across various numbers of dialogue turns. The main contributions of our work are: • Developing a systematic framework and constructing a high-quality dataset for evaluating the clinical pre-consultation ability of LLMs. • Open-sourcing the dataset and pipeline. • Implementing targeted experiments and sharing the results with in-depth analysis. 2 Related Work 2.1 Medical LLMs in Clinical Applications Existing clinical chatbot applications include HuatuoGPT ( 40 ) , ChatDoctor ( 19 ) , MedChatZH ( 29 ) , MedAide ( 1 ) , and MILD Bot ( 14 ) . Other medical LLM applications not limited to chatbots are 16 ; 41 ; 36 ; 8 ; 33 . LLMs have demonstrated diagnostic accuracy comparable to that of physicians in certain contexts ( 23 ) , with existing works primarily focusing on final diagnostic outcomes ( 20 ; 28 ; 31 ) . However, research on patient information collection during LLM pre-consultation remains limited. To address this, we propose a fine-grained framework that evaluates LLM pre-consultation capabilities. 2.2 Evaluation of Medical LLMs Multiple-choice QA is widely used for medical evaluation, as demonstrated by Med-HALT ( 22 ) , MedMCQA ( 21 ) , PubMedQA ( 11 ) , and KoreMedMCQA ( 17 ) . However, it is insufficient for assessing real-world clinical conversational abilities ( 2 ; 4 ) . More sophisticated evaluation frameworks in the clinical domain have been proposed, including MEDIC ( 13 ) , LLM-Mini-CEX ( 27 ) , CRAFT-MD ( 12 ) . Other evaluation benchmarks regarding disease diagnosis include works by 9 , 43 , 3 , 5 , 26 , 24 , 7 . While 37 and 6 propose evaluation pipelines for pre-consultation, their dataset coverage is limited and peripheral. Figure 2: EPAG benchmark dataset construction process. Expert clinicians collect all possible diagnostic guidelines of diseases from credible clinical sources. They then filter diseases based on whether they can be reasonably diagnosed through consultation alone and sufficiently common to ensure unbiased evaluation. Next, clinicians verify that the disease list is comprehensive enough to serve as a generalizable evaluation set. Using the finalized list, synthetic patient profiles are generated and finalized through qualitative analysis by clinicians. 3 EPAG Benchmark We assess pre-consultation models designed to collect as much relevant information as possible from the patient, including symptoms, family history, and other factors, referred to as the History of Present Illness (HPI). This section covers the tasks, dataset construction process, and evaluation pipeline of EPAG. 3.1 Evaluation Tasks As Figure 1 demonstrates, we propose a two-tiered evaluation framework based on the collected HPI. 3.1.1 HPI-Diagnostic Guideline Comparison For direct evaluation, we focus on how effectively the models capture information necessary for accurate disease identification. The evaluation process involves pre-consultation simulation with a patient-agent exhibiting symptoms of a specific disease and a doctor-agent, which is the subject of evaluation. During this interaction, the doctor-agent asks questions and provides multiple options for the patient-agent to choose from. The HPI collected is then compared against a set of diagnostic guidelines for the specific disease. The diagnostic guidelines represent a collection of essential information for diagnosing a particular disease, curated by human clinicians from trusted sources with further details in Section 3.2.1 . 3.1.2 Disease Diagnosis For indirect evaluation, we assess how well the collected HPI supports accurate diagnoses when provided to a separate diagnostic model. While this is not a direct evaluation of the HPI extracted by LLMs, it is a crucial assessment as one of the eventual goals of LLM pre-consultation is to assist in correct diagnosis and treatment. 3.2 Dataset Figure 2 shows the dataset construction process. 3.2.1 Diagnostic Guideline To evaluate whether each dialogue turn elicits meaningful patient information for diagnosis, we construct a gold-label diagnostic guideline dataset. The following steps are implemented by professional clinicians based on credible clinical associations and organizations in Appendix A : (1) collect diagnostic guidelines with explicit references; (2-1) filter diseases that are diagnosable through consultation alone, without reliance on physical exams, X-ray or MRI; (2-2) exclude diseases that are too rare. As exemplified in Appendix B , each diagnostic guideline specifies key symptoms, ancillary symptoms, family history, and other relevant risk factors. Each feature is assigned a weight of either high or medium . 3.2.2 Disease As our primary goal is to evaluate language models rather than multi-modal models, we focus on diseases that can be differentiated without reliance on other examination results. Through extensive discussions with clinicians, we identify 26 such diseases spanning 10 primary specialties and 22 secondary specialties. To ensure that the selection of 26 diseases provides sufficient clinical generalizability, clinicians classify them according to the International Classification of Diseases, 11th Revision (ICD-11) 1 1 1 https://icd.who.int/en . This categorization confirms that the

Read the complete text