When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation

Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi · 2025

Read When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation by Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19032.txt.

Introduction

Abeer Badawi and eight co-authors ask when large language models can be trusted in mental health support, and build large benchmarks to answer it. Judging such models is hard because therapeutic dialogue is emotionally and cognitively complex, and older benchmarks were small or built from synthetic and social media data.

One benchmark gathers ten thousand real-scenario conversations, each with nine model responses, for one hundred thousand pairs. A second compares four strong model judges against human experts across seventy thousand ratings. Together they test both response quality and whether automated judging itself can be trusted.

Catalog record

CREATORAbeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19032

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi Abstract Evaluating Large Language Models (LLMs) for mental health support is challenging due to the emotionally and cognitively complex nature of therapeutic dialogue. Existing benchmarks are limited in scale, reliability, often relying on synthetic or social media data, and lack frameworks to assess when automated judges can be trusted. To address the need for large-scale dialogue datasets and judge-reliability assessment, we introduce two benchmarks that provide a framework for generation and evaluation. MentalBench-100k consolidates 10,000 one-turn conversations from three real scenarios datasets, each paired with nine LLM-generated responses, yielding 100,000 response pairs. MentalAlign-70k reframes evaluation by comparing four high-performing LLM judges with human experts across 70,000 ratings on seven attributes, grouped into Cognitive Support Score (CSS) and Affective Resonance Score (ARS). We then employ the Affective–Cognitive Agreement Framework , a statistical methodology using intraclass correlation coefficients (ICC) with confidence intervals to quantify agreement, consistency, and bias between LLM judges and human experts. Our analysis reveals systematic inflation by LLM judges, strong reliability for cognitive attributes such as guidance and informativeness, reduced precision for empathy, and some unreliability in safety and relevance. Our contributions establish new methodological and empirical foundations for reliable, large-scale evaluation of LLMs in mental health. We release the benchmarks and codes at: https://github.com/abeerbadawi/MentalBench/ 1 Introduction Integrating Large Language Models (LLMs) into mental health support systems presents both a transformative opportunity and a significant challenge. Given the critical shortage of mental health professionals, estimated at just 13 per 100,000 individuals by WHO Organization 2021 , LLMs present a promising opportunity to enhance mental health care by improving access, scalability, and timely support ( Badawi et al. 2025 ) . With the rise of Generative AI tools such as ChatGPT, individuals are increasingly using online platforms to ask mental health questions and seek therapy support ( Gualano et al. 2025 ) . This growing reliance underscores the urgent need for consistent systems to evaluate the safety, accuracy, and clinical appropriateness of responses ( Bedi et al. 2023 ) . However, despite rapid advancements in generative AI, mental health remains one of the least prioritized domains for AI adoption ( Insights and Healthcare 2024 ) . This under-utilization reflects persistent concerns around ethical risks and the absence of datasets that capture authentic therapeutic dynamics ( Ji et al. 2023 ; Bedi et al. 2025 ) . Moreover, many existing LLM evaluation studies rely on synthetic conversations or social media content, which fail to capture the nuanced emotional and contextual complexities in mental health support for reliable evaluation ( Yuan et al. 2024 ; Guo et al. 2024a ) . Figure 1: Overview of our proposed system: MentalBench-100k provides mental health conversations with multi-LLM responses. MentalAlign-70k benchmarks cognitive and affective attributes using human experts and LLMs as judges. Affective–Cognitive Agreement framework applies ICC and bias detection to quantify reliability. The scarcity of authentic therapeutic dialogues, combined with the absence of frameworks to assess evaluator reliability, raises a fundamental question: How can we reliably evaluate LLMs’ responses in real-world mental health scenarios, where both affective and cognitive support are essential? To answer this question, we compile a multi-source dataset of generated clinical counseling conversations paired with responses and generate multiple LLM replies per context; we refer to this benchmark as MentalBench-100k. MentalBench-100k focuses on single-session mental health support scenarios, reflecting real-world scenarios such as crisis helplines, mobile apps, or one-turn interactions with tools like ChatGPT (e.g., “I feel anxious—what should I do?”) ( Ji et al. 2023 ) . Building on this foundation, we design an evaluation benchmark comparing human experts with LLM judges named MentalAlign-70k. We introduce a dual-axis evaluation grounded in psychological instruments: Cognitive Support Score (CSS), measuring guidance, informativeness, relevance, and safety, and Affective Resonance Score (ARS), capturing empathy, helpfulness, and understanding ( Hua et al. 2024 ) . Four LLMs serve as judges alongside human experts, enabling systematic evaluation across all seven therapeutic dimensions. Finally, we present the Affective–Cognitive Agreement Framework, which quantifies agreement between LLM judges and human experts across three critical dimensions of consistency, agreement, bias, and distills these into actionable reliability categories. This framework reveals when reliability can be trusted versus when human oversight is mandatory through empirical comparisons with human experts in mental health dialogue. Together with our benchmarks, we establish a comprehensive foundation for evaluating LLMs in mental health and for advancing the development of safer, clinically informed, and trustworthy AI systems. This work makes the following contributions: (i) MentalBench–100k Benchmark: A consolidation of all publicly available counseling and clinically grounded therapeutic conversations, creating a benchmark of 10,000 context–response dialogues and 100,000 additional replies generated by nine diverse LLMs. We generated responses using diverse LLMs to enable a critical evaluation, given the increasing exploration of their use in real-world scenario therapeutic settings. (ii) MentalAlign–70k Benchmark: A clinically grounded dual-axis evaluation benchmark comprising Cognitive Support Score (CSS) and Affective Resonance Score (ARS), validated by human expert judgment against 4 LLM judges across 70,000 ratings. This establishes the first comprehensive human-AI evaluation comparison in mental health dialogue with seven attributes. (iii) Affective–Cognitive Agreement Framework: A dual reliability framework with a three-pillar (consistency, agreement, bias), and a reliability classification scheme. This framework investigates when reliability can be trusted versus when human oversight is mandatory, providing an evidence-based reliability guidance for mental health AI systems. (iv) Open-Source Contribution: We release both benchmarks and the accompanying code as open-source resources. They are available on GitHub ( https://github.com/abeerbadawi/MentalBench-Align ) and Hugging Face ( https://huggingface.co/datasets/abadawi/MentalBench-Align ). 2 Related Work Mental Health Data. A key challenge in advancing LLMs for mental health applications is the scarcity of publicly available datasets based on real therapeutic interactions. Most existing resources rely on synthetic dialogues, crowdsourced role-play, or social media content, which lack the depth and fidelity of clinical conversations ( Hua et al. 2024 ; Jin et al. 2025 ; Guo et al. 2024b ) . Notable datasets such as EmpatheticDialogues ( Rashkin et al. 2019 ) , ESConv ( Liu et al. 2021 ) , PsyQA ( Sun et al. 2021 ) , D4 ( Yao et al. 2022 ) , and ChatCounselor ( Liu et al. 2023 ) are primarily constructed from artificial, closed-source data or semi-structured scenarios. Recent data, such as MentalChat16K ( Xu et al. 2025a ) , although partially grounded in real data, includes synthetic content. Comprehensive reviews confirm that the majority of mental health datasets are drawn from platforms like Reddit and X, often lacking expert annotation or therapeutic grounding ( Jin et al. 2025 ; Guo et al. 2024b ) . The reliance on pseudo-clinical text introduces concerns about the validity, safety, and applicability of LLMs in real-world support systems ( Gabriel et al. 2024 ) . As highlighted in recent literature ( Hua et al. 2024 ; Stade et al. 2024 ) , expanding access to high-quality, ethically sourced therapeutic conversations remains essential for responsible AI development in this domain. For instance, Bedi et al. 2025 found that 5% of studies incorporate data from actual care settings, with the majority relying

Read the complete text