MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
Shufeng Kong, Xingru Yang, Yuanyuan Wei, Zijie Wang, Hao Tang, Jiuqi Qin, Shuting Lan, Yingheng Wang, Junwen Bai, Zhuangbin Chen, Zibin Zheng, Caihua Liu, Hao Liang · 2025
Read MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine by Shufeng Kong, Xingru Yang, Yuanyuan Wei, Zijie Wang, Hao Tang, Jiuqi Qin, Shuting Lan, Yingheng Wang, Junwen Bai, Zhuangbin Chen, Zibin Zheng, Caihua Liu, Hao Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01252.txt.
Introduction
Traditional Chinese medicine speaks its own diagnostic language, and general AI benchmarks barely test whether models can use it safely. Shufeng Kong and co-authors built a benchmark with certified practitioners. Twelve datasets cover five arenas: factual knowledge, language understanding, diagnostic reasoning, prescription writing, and safety.
Sources include real case records, national licensing exams, and classical texts. Early results draw a clear line. Today's models handle foundational knowledge respectably. They stumble on clinical reasoning, on composing prescriptions, and on safety compliance, exactly where errors hurt patients. The authors release everything publicly and argue domain benchmarks like this one are prerequisites for trustworthy medical AI.
Catalog record
| CREATOR | Shufeng Kong, Xingru Yang, Yuanyuan Wei, Zijie Wang, Hao Tang, Jiuqi Qin, Shuting Lan, Yingheng Wang, Junwen Bai, Zhuangbin Chen, Zibin Zheng, Caihua Liu, Hao Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01252
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine Shufeng Kong, Xingru Yang, Yuanyuan Wei, Zijie Wang, Hao Tang, Jiuqi Qin, Shuting Lan, Yingheng Wang, Junwen Bai, Zhuangbin Chen, Zibin Zheng, Caihua Liu, Hao Liang Abstract Traditional Chinese Medicine (TCM) is a holistic medical system with millennia of accumulated clinical experience, playing a vital role in global healthcare—particularly across East Asia. However, the implicit reasoning, diverse textual forms, and lack of standardization in TCM pose major challenges for computational modeling and evaluation. Large Language Models (LLMs) have demonstrated remarkable potential in processing natural language across diverse domains, including general medicine. Yet, their systematic evaluation in the TCM domain remains underdeveloped. Existing benchmarks either focus narrowly on factual question answering or lack domain-specific tasks and clinical realism. To fill this gap, we introduce MTCMB—a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, prescription generation, and safety evaluation. The benchmark integrates real-world case records, national licensing exams, and classical texts, providing an authentic and comprehensive testbed for TCM-capable models. Preliminary results indicate that current LLMs perform well on foundational knowledge but fall short in clinical reasoning, prescription planning, and safety compliance. These findings highlight the urgent need for domain-aligned benchmarks like MTCMB to guide the development of more competent and trustworthy medical AI systems. All datasets, code, and evaluation tools are publicly available at: https://github.com/Wayyuanyuan/MTCMB. 1 Introduction Traditional Chinese Medicine (TCM) is a holistic healthcare system with millennia of clinical history, serving as a cornerstone of medical practice in China and East Asia while gaining global prominence hu2015traditional ; who_tcm . However, its implicit reasoning mechanisms, diverse textual forms, and lack of standardization present significant challenges for computational modeling. TCM employs symbolic terminology (e.g., pulse/tongue diagnostics) and pattern-based reasoning distinct from Western biomedicine. Critical concepts remain inconsistently digitized, herb names exhibit polysemy, and diagnostic categories overlap without universal ontologies. Furthermore, TCM knowledge is predominantly encoded in classical texts and case narratives rather than structured data. These factors—symbolic inference, linguistic complexity, and terminological ambiguity—hinder conventional AI systems from faithfully representing TCM principles yue2024tcmbench . Large Language Models (LLMs) demonstrate transformative potential in medicine through corpus-scale pattern recognition, achieving state-of-the-art performance on medical QA tasks thirunavukarasu2023large . However, models pretrained on general text (e.g., GPT-4) or fine-tuned for Western medicine (e.g., Med-PaLM2 singhal2025toward ) lack domain-specific understanding of TCM’s diagnostic frameworks or herbal pharmacopeia. Crucially, deploying LLMs in TCM poses unique safety risks: uninformed models may generate harmful herb combinations or overlook critical contraindications. Existing safety benchmarks (e.g., SafetyBench zhang2023safetybench , SG-Bench mou2024sg ) focus on general or Western medical contexts, failing to address TCM’s nuanced safety protocols. These challenges necessitate a multi-task benchmark to evaluate LLMs on TCM-specific knowledge, reasoning, and safety. Current resources either target general medicine (MedQA jin2021disease , PubMedQA DBLP:conf/emnlp/JinDLCL19 , MedMCQA pmlr-v174-pal22a ) or focus narrowly on Chinese licensing exams (TCMBench yue2024tcmbench , TCMD yu2024tcmd ). To our knowledge, no benchmark integrates TCM knowledge retrieval, clinical reasoning, prescription generation, and safety evaluation in realistic scenarios. To bridge this gap, we introduce MTCMB, a multi-task benchmark co-developed with certified TCM practitioners. We curate eleven sub-datasets from authoritative sources—national exams, classical texts, patient records, and safety guidelines—spanning five task categories: (1) factual QA, (2) textual comprehension, (3) diagnostic reasoning, (4) prescription generation, and (5) safety evaluation. Furthermore, we evaluate diverse LLMs across three categories: (1) General LLMs: GPT-4.1, Claude 3.7, Gemini 3, Qwen-Max, Doubao-1.5-Pro, GLM-Pro, Deepseek-V3; (2) Medical LLMs: BaiChuan-14B-M1, HuatuoGPT-01-72B, DISC-MedLLM, Taiyi-LLM, WiNGPT2-14B-Chat; and (3) Reasoning-focused LLMs: Qwen3-235B-A22B, O4-Mini, Deepseek-R1. Our analysis reveals persistent gaps: while models like GPT-4.1 and Qwen-Max excel at factual recall, all categories struggle with TCM-specific reasoning (e.g., syndrome differentiation) and safety-critical decisions. Medical LLMs such as HuatuoGPT-01-72B marginally outperform general models in prescription tasks but exhibit comparable failure rates in safety evaluations. Reasoning-optimized models like Deepseek-R1 show improved diagnostic accuracy yet remain prone to hallucinating non-standard herb combinations. These results underscore the need for domain-specific training paradigms and safety guardrails in TCM applications. By providing a comprehensive, clinically-grounded evaluation framework, MTCMB advances the development of reliable, TCM-competent LLMs. Our contributions include: • The first multi-task benchmark (MTCMB) for TCM LLMs, featuring expert-curated data spanning diverse modalities (QA pairs, case studies, prescriptions). • Formally defined evaluation tasks with domain-aware metrics: accuracy, BLEU/ROUGE-L for text generation, BERTScore for semantic alignment, and expert-derived safety criteria. • Systematic analysis of 14 LLMs—including proprietary, open-source, and medically-tuned variants—under zero-shot, few-shot, and chain-of-thought settings, revealing critical capability gaps across model classes. • Public release of the benchmark dataset, evaluation scripts, and detailed task guidelines to support reproducible research. 2 Related Works General NLP Benchmarks. Multi-task evaluation frameworks have driven progress in natural language understanding. Benchmarks like GLUE and SuperGLUE wang2020supergluestickierbenchmarkgeneralpurpose established standardized tasks for evaluating general linguistic competence, while MMLU hendrycks2020measuring extended this paradigm to 57 academic disciplines. However, these frameworks lack domain-specific adaptations for specialized fields like Traditional Chinese Medicine (TCM), which requires modeling implicit reasoning patterns and symbolic terminology absent from general corpora. Medical LLM Evaluation. Domain-specific benchmarks have emerged to assess medical LLMs. Early efforts like MedQA jin2021disease and MedMCQA pmlr-v174-pal22a focused on Western medical knowledge through multiple-choice questions, later expanded by MultiMedQA singhal2023large to include diverse clinical formats. For Chinese medical AI, CMExam liu2023benchmarking aggregates 60,000+ licensing exam questions, and CMB wang2023cmb integrates TCM queries within a broader clinical dataset. While these benchmarks evaluate factual recall and diagnostic reasoning, they inadequately address TCM’s unique diagnostic frameworks (e.g., pulse/tongue analysis) and herbal pharmacopeia. TCM-Specific Resources. Recent work has begun targeting TCM’s computational challenges. TCM-Bench yue2024tcmbench evaluates LLMs on licensing exam questions but omits safety and generative tasks. Qibo-Benchmark zhang2024qibo , accompanying a LLaMA-based TCM model, introduces entity recognition and syndrome differentiation tasks but lacks multi-turn clinical dialogues. TCMEval-SDT wang2025tcmeval provides 300 annotated cases for syndrome differentiation, yet its narrow scope excludes prescription generation and safety evaluation. These efforts highlight growing interest in TCM AI but prioritize isolated capabilities over holistic clinical competence. Safety and Reliability. LLM safety research has produced benchmarks like MedHallu pandit2025medhallu for medical hallucinations and BeHonest chern2024behonest for output veracity. While these address general medical risks, they neglect TCM-specific hazards—improper herb combinations, dosage errors, and contraindications (e.g., pregnancy restrictions). Existing tools also fail to account for TCM’s linguistic nuances, where polysemous terms like “dampness” or “wind” carry domain-specific meanings distinct from biomedical usage. Distinctiveness of MTCMB. Prior benchmarks either generalize across medicine (losing TCM specificity) or focus narrowly on individual tasks (e.g., syndrome differentiation). MTCMB advances the field by integrating five clinically grounded task categories: (1) factual QA, (2) textual comprehension (entity extraction, classical text parsing), (3) diagnostic reasoning (multi-turn dialogues, case analysis), (4) prescription generation (herb selection, dosage adjustment), and (5) safety evaluation (contraindication detection, risk mitigation). By synthesizing these dimensions with expert-validated TCM safety protocols, MTCMB enables comprehensive assessment of LLMs’ readiness for real-world TCM applications—a critical gap in existing resources. 3 Formulations and Background 3.1 Language Models We treat an LLM as a conditional probabilistic model p(y|x;\theta) that assign a probability to an output text y=(y_{1},\dots,y_{T}) given an input text