PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models
Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang · 2024
Read PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models by Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06287.txt.
Introduction
Qian Zhang and co-authors built PediaBench in 2024, a Chinese data set to test language models on child health. Medical test sets exist, but few focus on pediatrics, and many hold only fixed-choice questions. Such sets cannot gauge how well a model writes answers.
PediaBench holds thousands of objective and essay-style questions across twelve groups of child disease. Scoring blends difficulty levels and checks instruction following, grasp of knowledge, and clinical skill. The set gives builders a common yardstick for a high-stakes field. Hospitals and parents alike need models whose child health answers have been tested, not assumed.
Catalog record
| CREATOR | Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang Abstract The emergence of Large Language Models (LLMs) in the medical domain has stressed a compelling need for standard datasets to evaluate their question-answering (QA) performance. Although there have been several benchmark datasets for medical QA, they either cover common knowledge across different departments or are specific to another department rather than pediatrics. Moreover, some of them are limited to objective questions and do not measure the generation capacity of LLMs. Therefore, they cannot comprehensively assess the QA ability of LLMs in pediatrics. To fill this gap, we construct PediaBench, the first Chinese pediatric dataset for LLM evaluation. Specifically, it contains 4,117 objective questions and 1,632 subjective questions spanning 12 pediatric disease groups. It adopts an integrated scoring criterion based on different difficulty levels to thoroughly assess the proficiency of an LLM in instruction following, knowledge understanding, clinical case analysis, etc. Finally, we validate the effectiveness of PediaBench with extensive experiments on 20 open-source and commercial LLMs. Through an in-depth analysis of experimental results, we offer insights into the ability of LLMs to answer pediatric questions in the Chinese context, highlighting their limitations for further improvements. Our code and data are published anonymously at https://github.com/ACMISLab/PediaBench . 1 Introduction Figure 1: Illustration of the overall framework of PediaBench. Question Answering (QA) is an important task in Natural Language Processing (NLP) that has received considerable attention over several decades ( Simmons, 1965 ; Hirschman and Gaizauskas, 2001 ; Lan and Jiang, 2020 ; Li et al., 2024 ) . Recently, Large Language Models (LLMs) ( Yang et al., 2023 ; Bai et al., 2023a ; OpenAI et al., 2024 ) , with their remarkable language understanding, reasoning, and generation capabilities, have shown exceptional performance in QA tasks compared to classic deep learning methods Devlin et al. (2019) , especially for subjective questions. Naturally, there is also an increasing interest in applying LLMs to medical QA ( Chen et al., 2023a ; Singhal et al., 2023 ; Zhang et al., 2023a ; Liévin et al., 2024 ) . In this paper, we focus on the QA tasks in pediatrics , a medical department that involves the care of infants, children, adolescents, and young adults. Since pediatrics often involves the manifestations and treatments of diseases that differ from those of adults, LLMs with common medical knowledge might not perform equally well on pediatric QA. Therefore, evaluating the proficiency of LLMs in pediatric QA is an urgent need for their application in this domain. Several medical QA benchmarks have been proposed in the literature ( Jin et al., 2019 ; Jin et al., 2021 ; Pal et al., 2022 ; Fansi Tchango et al., 2022 ; Zhang et al., 2022 ; Li et al., 2023b ; Liu et al., 2023 ; Wang et al., 2023a ; Wang et al., 2023c ; Zhu et al., 2023 ; Cai et al., 2024a ; Yue et al., 2024 ) . However, they still have some limitations that hinder their effectiveness in the pediatric context. First, they are mostly general medical benchmarks across multiple departments and are not tailored for pediatrics. As such, their coverage of knowledge in pediatrics is often very limited. Second, most of them contain only objective questions, e.g., true/false and multiple-choice questions. Although they can serve as an indicator of the capacity of LLMs to comprehend medical knowledge, they cannot assess the capacity of LLMs to generate medical texts. In addition, some QA datasets originating from doctor-patient interactions in real-world scenarios Li et al. (2023b) include some subjective questions to evaluate the conversation ability of LLMs. However, they focus only on a limited number of common diseases and often lack depth of knowledge in medicine. Therefore, existing benchmarks are insufficient to provide comprehensive evaluations of LLMs in terms of pediatric capability. To address the above challenges, in this paper, we introduce PediaBench, the first comprehensive Chinese benchmark dataset in pediatrics. As shown in Figure 1 , PediaBench contains 4,117 objective questions and 1,632 subjective questions collected from various sources, such as the Chinese National Medical Licensing Examination (CNMLE), final exams of universities in medicine, pediatric disease diagnosis and treatment standards, and clinical guidelines. PediaBench encompasses five distinct types of questions, namely true or false, multiple choice, pairing, essay/short answer, and case analysis, across 12 typical pediatric disease groups. Furthermore, to provide an accurate evaluation of the performance of each LLM for QA in pediatrics, we use a scoring criterion that combines difficulty levels and automatic scoring. On the one hand, we assign different scores to objective questions based on the difficulty factor; on the other hand, we adopt the Language-Model-as-an-Examiner (LME) framework Bai et al. (2023b) to score the objective questions based on GPT-4o. Finally, we conduct extensive experiments with 20 LLMs on the PediaBench dataset and provide an in-depth analysis of the performance of LLMs for QA in pediatrics. Our main contributions are summarized as follows: • We introduce PediaBench, a high-quality QA dataset specific to pediatrics in the Chinese context. We also devise an integrated scoring scheme to measure the QA performance of each LLM on PediaBench. • We evaluate PediaBench with 20 LLMs, including open-source and commercial general-purpose models of different scales and specialized models in the medical domain. The results indicate that PediaBench is a challenging dataset that can gauge the capacities of LLMs in terms of pediatric QA. • We provide a detailed analysis of the results, highlighting the limitations of current LLMs and suggesting their potential opportunities in this emerging domain. 2 Related Work General LLM Benchmarks The rapid advances in LLMs ( Touvron et al., 2023 ; Yang et al., 2023 ; Du et al., 2022 ; Bai et al., 2023a ; OpenAI et al., 2024 ) have underscored the need for benchmarks to evaluate their performance in a variety of NLP tasks. To this end, a large number of benchmarks specifically designed for LLMs, e.g., ( Wang et al., 2019 ; Xu et al., 2020 ; Xu et al., 2023 ; Zhang et al., 2023b ; Zhong et al., 2023 ; Huang et al., 2023 ; Li et al., 2023a ; Suzgun et al., 2023 ; Gu et al., 2024 ) , were proposed. We refer interested readers to ( Chang et al., 2024 ) for an extensive survey. These general benchmarks often require LLMs to answer questions from standard examinations ( Gu et al., 2024 ; Huang et al., 2023 ) in a broad spectrum of domains to evaluate their capacity in text understanding, logical reasoning, calculation, generation, etc. However, although medical questions are included in some of these benchmarks, they are about common knowledge that can be answered without specialization, which cannot accurately reflect the proficiency of LLMs in medical tasks. Medical LLMs and Benchmarks More recently, many efforts have been made to build specialized LLMs in the medical domain, e.g., DoctorGLM ( Xiong et al., 2023 ) , ChatDoctor ( Li et al., 2023c ) , BianQue ( Chen et al., 2023a ) , PMC-LLaMA ( Wu et al., 2024 ) , BioMistral ( Labrak