A Gold Standard Dataset and Evaluation Framework for Depression Detection and Explanation in Social Media using LLMs

Prajval Bolegave, Pushpak Bhattacharya · 2025

Read A Gold Standard Dataset and Evaluation Framework for Depression Detection and Explanation in Social Media using LLMs by Prajval Bolegave, Pushpak Bhattacharya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19899.txt.

Introduction

Prajval Bolegave and Pushpak Bhattacharya build a trusted yardstick for spotting depression in social media posts in this 2025 paper. Early warning from online writing could speed help to people in trouble. Most datasets label a whole post as depressed or not, which is too coarse to judge whether a model's explanation makes clinical sense.

Their dataset holds 1,017 posts marked by experts at the phrase level, each span tied to one of twelve symptom categories. Around it they built a scoring framework that grades both the spans a model finds and the explanations it writes. They tested three leading commercial models with plain and example-guided prompts. Scores differed widely between models and prompt styles. The authors argue expert guidance is what makes such systems safer, and offer the dataset as a step toward transparent tools for mental health.

Catalog record

CREATORPrajval Bolegave, Pushpak Bhattacharya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Gold Standard Dataset and Evaluation Framework for Depression Detection and Explanation in Social Media using LLMs Prajval Bolegave, Pushpak Bhattacharya Abstract Early detection of depression from online social media posts holds promise for providing timely mental health interventions. In this work, we present a high-quality, expert-annotated dataset of 1,017 social media posts labeled with depressive spans and mapped to 12 depression symptom categories. Unlike prior datasets that primarily offer coarse post-level labels Cohan et al. 2018 , our dataset enables fine-grained evaluation of both model predictions and generated explanations. We develop an evaluation framework that leverages this clinically grounded dataset to assess the faithfulness and quality of natural language explanations generated by large language models (LLMs). Through carefully designed prompting strategies, including zero-shot and few-shot approaches with domain-adapted examples, we evaluate state-of-the-art proprietary LLMs including GPT-4.1, Gemini 2.5 Pro, and Claude 3.7 Sonnet. Our comprehensive empirical analysis reveals significant differences in how these models perform on clinical explanation tasks, with zero-shot and few-shot prompting. Our findings underscore the value of human expertise in guiding LLM behavior and offer a step toward safer, more transparent AI systems for psychological well-being. 1 Introduction Mental health disorders, particularly depression, have become increasingly prevalent and are often expressed through social media platforms Cohan et al. 2018 ; Guntuku et al. 2017 . The ubiquity of user-generated content has prompted researchers to explore language-based signals for early identification of psychological distress Calvo et al. 2017 ; Xu et al. 2023 . However, most existing machine learning models for depression detection operate on weakly-labeled data or binary classification tasks, limiting their clinical utility and interpretability Chancellor and De Choudhury 2020 . Further, the explainability of these models remains a critical bottleneck, especially when applied to sensitive domains like mental health. While large language models (LLMs) such as GPT-4 and Claude offer impressive fluency, their generated explanations often lack alignment with clinically grounded reasoning Yang et al. 2023 ; Rajani et al. 2019 . This gap between linguistic coherence and domain-specific faithfulness raises important concerns for deploying AI in healthcare contexts Doshi-Velez and Kim 2017 . To address these challenges, we introduce a method for constructing a high-quality, expert-annotated dataset for depression detection and explanation. Each post is manually labeled by licensed mental health professionals with two layers of annotation: (i) spans of text indicative of depressive symptoms, and (ii) corresponding mappings to one or more of twelve standard symptoms derived from the DSM-5 criteria (e.g., anhedonia, hopelessness, suicidal ideation) Association 2013 . This dual-layer annotation enables more granular supervision for both training and evaluation. Our contributions are: 1. A gold-standard mental health dataset consisting of 1,017 social media posts annotated by clinical experts for both depressive spans and symptom categories, enabling rigorous evaluation of explainability in mental health prediction (Section 4 ). 2. An evaluation framework leveraging the annotated dataset to assess the Faithfulness , which is captured through Span-Recall, measuring the presence of expert-annotated depressive spans in model-generated explanation (weighted 30%), Clinical relevance , which is assessed using Symptom-F1, which evaluates alignment between predicted symptoms and ground truth labels (weighted 40%), and Coherence Score , reflecting the fluency and interpretability of the explanation (weighted 30%). (Section 3.2 ). 3. Prompting strategies for explanation generation, including Zero-Shot and Few-Shot prompting with domain-adapted examples using advanced LLMs such as GPT-4.1, Gemini 2.5 pro, and Claude 3.7 Sonnet, to explore their impact on explanation quality in NLE tasks (Section 3 , Appendix A.1 ). 4. Empirical comparison of proprietary LLMs (GPT-4.1, Gemini 2.5 Pro, Claude 3.7) on expert-annotated mental health data, highlighting their relative strengths and weaknesses in clinical explanation tasks across different prompting settings. (Section 6 ) 2 Related Work 2.1 Mental Health Datasets and Annotation Practices Annotated datasets have played a critical role in advancing NLP for mental health analysis. The SMHD corpus provided large-scale Reddit data labeled for multiple mental health conditions, enabling early detection through linguistic signals Cohan et al. 2018 . EmoMent focused on emotion-annotated posts from South Asian populations, highlighting cultural nuances in mental health expression Atapattu et al. 2022 . More recently, the IMHI dataset introduced by the MentaLLaMA framework offered expert-written explanations for multiple mental health tasks, supporting interpretable model development Yang et al. 2023 . However, these resources predominantly feature document-level annotations and lack token-level span highlights or symptom-specific mappings grounded in clinical criteria. In contrast, our dataset provides expert annotations at both the span and symptom level based on DSM-5 categories, enabling more granular and clinically faithful evaluation. 2.2 Faithfulness and Explainability in LLMs Explainability has become a key concern in applying LLMs to sensitive domains such as mental health. The Mental-LLM benchmark evaluated models like GPT-3.5 and GPT-4 on diagnostic tasks and found limitations in zero-shot and few-shot settings, particularly in generating faithful rationales Xu et al. 2023 . Similarly, MentalGLM improved explanation generation through supervised fine-tuning on mental health-specific data Zhai et al. 2024 . Yet, few studies have proposed structured frameworks to quantify the alignment between model explanations and clinical reasoning. Our work addresses this gap through a span- and symptom-level evaluation framework that enables systematic measurement of explanation faithfulness and diagnostic relevance. 2.3 Prompting Strategies for Mental Health Tasks Prompt-based learning, especially in the form of in-context learning (ICL), has proven effective for adapting LLMs to specialized domains. Previous work has shown the utility of prompting for classification and emotion detection in mental health contexts Shi et al. 2024 . Inspired by these findings, we design zero-shot and few-shot prompts tailored to symptom identification and rationale generation, leveraging state-of-the-art proprietary LLMs such as GPT-4.1, Gemini 2.5 Pro, and Claude 3.7 Sonnet. Unlike prior works that use weak supervision or synthetic data, our prompting approach is grounded in a gold-standard dataset annotated by clinical experts. 2.4 Comparative Evaluation of LLMs Comparative analyses have shed light on the trade-offs between proprietary and open-source LLMs in mental health applications. Studies like Mental-LLM highlight how instruction tuning can boost model alignment with mental health tasks Xu et al. 2023 , while other efforts have explored cost-efficient adaptation strategies such as LoRA Shi et al. 2024 . In our work, we benchmark leading proprietary models using consistent prompting strategies on our expert-annotated dataset. This allows us to evaluate differences in explanation quality, faithfulness, and symptom coverage, offering insights into how well current LLMs align with clinical reasoning under realistic zero and few-shot conditions. 3 Methodology Our methodology is designed to evaluate the capability of large language models (LLMs) to generate faithful, clinically meaningful natural language explanations (NLEs) for depressive content in social media posts. We leverage our expert-annotated dataset to probe models under prompting-based setups and assess their outputs using a structured evaluation framework. 3.1 Model Setup and Prompting Strategies We experiment with state-of-the-art proprietary LLMs, including GPT-4.1, Gemini 2.5 Pro, and Claude 3.7 Sonnet, accessed via APIs. To generate explanations, we employ two prompting strategies: Zero-Shot Prompting. In this setup, models are given only a task description and the input post. The prompt instructs the model to identify depressive cues and explain its reasoning without any examples. (Appendix A.1.1 ) Few-Shot Prompting. Here, we provide models with a small set of in-context examples curated from our expert-annotated

Read the complete text