MentalBench : A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models
Hoyun Song Migyeong Kang Jisu Shin Jihyun Kim Chanbi Park, Hangyeol Yoo Jihyun An Alice Oh Jinyoung Han KyungTae Lim · 2026
Read MentalBench : A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models by Hoyun Song Migyeong Kang Jisu Shin Jihyun Kim Chanbi Park, Hangyeol Yoo Jihyun An Alice Oh Jinyoung Han KyungTae Lim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12871.txt.
Description
Language models are more and more put forward as aids for mind-health testing, but most mind-health benchmarks use social media posts or caring chats, not formal diagnosis. Hoyun Song and nine co-authors present MentalBench, a benchmark that tests whether models can use DSM-5 rules and tell-apart diagnosis rules under varying doubt. Its spine is MentalKG, a knowledge graph built and checked by mind-health doctors spanning 23 illnesses.
From that spine the team made 24,750 mock cases that vary in fullness and hardness. Tests show leading models do well on clean knowledge questions but struggle to tune trust when illnesses share signs. That gap raises doubts about how much to trust them in real diagnosis help and argues for testing that mirrors the messy doubt doctors face in real life.
Catalog record
| CREATOR | Hoyun Song Migyeong Kang Jisu Shin Jihyun Kim Chanbi Park, Hangyeol Yoo Jihyun An Alice Oh Jinyoung Han KyungTae Lim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12871