Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text
1 st Avinash Patil, 2 nd Amardeep Kour Gedhu · 2025
Read Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text by 1 st Avinash Patil, 2 nd Amardeep Kour Gedhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10095.txt.
Introduction
Web text can bare a mind's state, and word tools that show their steps may sort such text more true than bare tools. Avinash Patil and Amardeep Kour Gedhu test step tools in a 2025 study on crowd text sets. Chain, self check, and tree thought forms are run next to plain asks and schooled small tools.
Step tools beat plain asks, most in the hard cases. Few shot chain thought led the pack. Gains on two sets ran past both the schooled tools and a mind tuned mill. Two hard sets, grave grade and risk grade, ran the wrong way. The team log the split as a map of where step thought helps, and where it fails.
Catalog record
| CREATOR | 1 st Avinash Patil, 2 nd Amardeep Kour Gedhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text 1 st Avinash Patil, 2 nd Amardeep Kour Gedhu Abstract Large Language Models (LLMs) have demonstrated potential in predicting mental health outcomes from online text, yet traditional classification methods often lack interpretability and robustness. This study evaluates structured reasoning techniques—Chain-of-Thought (CoT), Self-Consistency (SC-CoT), and Tree-of-Thought (ToT)—to improve classification accuracy across multiple mental health datasets sourced from Reddit. We analyze reasoning-driven prompting strategies, including Zero-shot CoT and Few-shot CoT, using key performance metrics such as Balanced Accuracy, F1 score, and Sensitivity/Specificity. Our findings indicate that reasoning-enhanced techniques improve classification performance over direct prediction, particularly in complex cases. Compared to baselines such as Zero Shot non-CoT Prompting, and fine-tuned pre-trained transformers such as BERT and Mental-RoBerta, and fine-tuned Open Source LLMs such as Mental Alpaca and Mental-Flan-T5, reasoning-driven LLMs yield notable gains on datasets like Dreaddit (+0.52% over M-LLM, +0.82% over BERT) and SDCNL (+4.67% over M-LLM, +2.17% over BERT). However, performance declines in Depression Severity, and CSSRS predictions suggest dataset-specific limitations, likely due to our using a more extensive test set. Among prompting strategies, Few-shot CoT consistently outperforms others, reinforcing the effectiveness of reasoning-driven LLMs. Nonetheless, dataset variability highlights challenges in model reliability and interpretability. This study provides a comprehensive benchmark of reasoning-based LLM techniques for mental health text classification. It offers insights into their potential for scalable clinical applications while identifying key challenges for future improvements. Code, prompts and llm reasoning for classification are available at https://github.com/av9ash/cognitive_mental_llm I Introduction Mental health disorders, such as depression, anxiety, and suicidal ideation, represent a growing global concern, with millions of individuals affected annually [ 1 , 2 ] . Online platforms, mainly social media and mental health forums, have become vital spaces for individuals to express their emotions and seek support [ 3 ] . This has led to a surge in interest in AI-driven methods for analyzing and classifying mental health-related text. However, accurately interpreting such text remains a significant challenge due to the complexity and variability of natural language in mental health discourse [ 4 ] . Traditional NLP-based classifiers, such as BERT [ 5 ] and RoBERTa [ 6 ] , have demonstrated strong performance in general text classification but often struggle with mental health data due to subtle linguistic cues, contextual ambiguity, and the need for structured reasoning. Prior studies [ 7 , 8 , 9 ] have explored deep learning and transformer-based approaches, yet these models exhibit limitations in interpretability and robustness when applied to real-world mental health assessments. To address these challenges, we investigate the role of structured reasoning in mental health classification using OpenAI’s o3-mini , a small reasoning-focused large language model (LLM). Our study evaluates four structured reasoning prompting strategies: Chain-of-Thought (CoT) [ 10 ] , Self-Consistency CoT (SC-CoT) [ 11 ] , Few-Shot CoT [ 12 ] , and Tree-of-Thought (ToT) [ 13 ] . These approaches encourage the model to generate step-by-step reasoning before classification, improving robustness and interpretability. Our main contributions are as follows: • We apply structured reasoning techniques to mental health text classification and evaluate their effectiveness across five benchmark datasets: Dreaddit [ 8 ] , CSSRS [ 9 ] , SDCNL [ 14 ] , DepSeverity [ 15 ] , and RedSam [ 16 ] . • We compare reasoning-based prompting strategies against zero-shot classification and prior state-of-the-art transformer models ( BERT, RoBERTa, Alpaca, FLAN-T5 ) [ 7 ] . • We conduct a detailed analysis of classification performance, showing that Few-Shot CoT improves multi-class classification tasks while CoT and SC-CoT enhance binary classification robustness. Our results demonstrate that structured reasoning strategies improve classification accuracy, particularly in datasets with nuanced language and multi-class labels. Notably, Few-Shot CoT performs superior in CSSRS and DepSeverity, while CoT and SC-CoT enhance classification for Dreaddit and SDCNL. Paper Organization: The remainder of this paper is structured as follows: Section II reviews related work. Section III describes our methodology, including datasets and prompting techniques. Section IV presents experimental results, and Section V concludes with key findings and future directions. II Related Work The application of Large Language Models (LLMs) in mental health prediction has garnered significant attention in recent years. Traditional machine learning approaches, such as Support Vector Machines (SVMs) and logistic regression, have been employed for mental health text classification [ 17 ] . However, these models often require extensive feature engineering and struggle with capturing contextual nuances in text data. With the advent of transformer-based architectures, LLMs have demonstrated improved capabilities in understanding and generating human-like text. Xu et al. [ 7 ] introduced Mental-LLM , leveraging LLMs for mental health prediction via online text data. Their study highlighted that LLMs outperform traditional methods in predictive accuracy. However, these models primarily rely on direct classification and often lack interpretability, which is crucial in mental health assessments. Several prompting techniques have been proposed to enhance reasoning capabilities and interpretability. Wei et al. [ 10 ] introduced Chain-of-Thought (CoT) prompting, enabling LLMs to generate intermediate reasoning steps, improving performance on complex tasks. Building upon this, Wang et al. [ 11 ] proposed Self-Consistency (SoTC), which involves generating multiple reasoning paths and selecting the most consistent answer, enhancing reliability. Yao et al. [ 13 ] extended these concepts with Tree-of-Thought (ToT) prompting, introducing a structured, hierarchical reasoning process for deliberate problem-solving. Recent research has explored the ability of large language models (LLMs) to evaluate responses to suicidal ideation [ 18 ] . In an observational, cross-sectional study, three widely used LLMs—ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—were assessed on their capacity to rate clinician responses from the revised Suicidal Ideation Response Inventory (SIRI-2). The study compared LLM-generated ratings to expert suicidologists’ ratings using linear regression analyses and z-score outlier detection. Recent advancements have also explored the integration of reasoning and acting within LLMs. Yao et al. [ 19 ] proposed the ReAct pattern, synergizing reasoning and acting in language models to improve task performance. This approach allows LLMs to reason through problems and take actions based on their reasoning, leading to more effective problem-solving strategies. Despite these advancements, applying reasoning techniques in LLMs to mental health text classification remains underexplored. Patil [ 20 ] provides a comprehensive overview of promising methods and approaches in advancing reasoning in LLMs, emphasizing their potential to enhance model interpretability and decision-making. However, there is a lack of studies systematically evaluating the impact of these reasoning techniques on mental health prediction tasks. In this study, we aim to bridge the gap in mental health text classification by systematically evaluating structured reasoning techniques—Chain-of-Thought (CoT), Self-Consistency (SC-CoT), Few-Shot Learning with CoT (FS-CoT), and Tree-of-Thought (ToT). We benchmark these reasoning-driven approaches against established models, including BERT, RoBERTa, and the best-performing supervised fine-tuned large language models from [ 7 ] . Our comprehensive analysis spans multiple datasets, providing deeper insights into the effectiveness of reasoning techniques in enhancing LLM-based mental health assessments. III Methodology This study evaluates the effectiveness of structured reasoning techniques for mental health text classification using OpenAI’s o3-mini [ 21 ] model, a compact, reasoning-oriented language model known for its strong performance in scientific, mathematical, and programming tasks. We examine four prompting strategies—Chain-of-Thought (CoT), Tree-of-Thought (ToT),