Evaluating LLM Reasoning for Suicide Screening with the Columbia-Suicide Severity Rating Scale
1 st Avinash Patil, 2 nd Siru Tao, 3 rd Amardeep Kour Gedhu · 2025
Read Evaluating LLM Reasoning for Suicide Screening with the Columbia-Suicide Severity Rating Scale by 1 st Avinash Patil, 2 nd Siru Tao, 3 rd Amardeep Kour Gedhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13480.txt.
Introduction
As people in crisis increasingly confide in chatbots, Avinash Patil and co-authors test whether language models can grade suicide risk responsibly. They used the Columbia-Suicide Severity Rating Scale, a seven-level clinical ladder, to score posts in zero-shot fashion. Six models, including Claude, GPT, Mistral, and LLaMA variants, were compared against human annotations. Claude and GPT tracked human judgments most closely, while Mistral made the smallest ranking errors.
Most mistakes landed on neighboring severity levels, suggesting the models sense gradations of risk. The authors stress human oversight, transparency, and cautious deployment before any clinical use, and they share their code openly.
Catalog record
| CREATOR | 1 st Avinash Patil, 2 nd Siru Tao, 3 rd Amardeep Kour Gedhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13480
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating LLM Reasoning for Suicide Screening with the Columbia-Suicide Severity Rating Scale 1 st Avinash Patil, 2 nd Siru Tao, 3 rd Amardeep Kour Gedhu Abstract Suicide prevention remains a critical public health challenge. While online platforms such as Reddit’s r/SuicideWatch have historically provided spaces for individuals to express suicidal thoughts and seek community support, the advent of large language models (LLMs) introduces a new paradigm—where individuals may begin disclosing ideation to AI systems instead of humans. This study evaluates the capability of LLMs to perform automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS). We assess the zero-shot performance of six models—including Claude, GPT, Mistral, and LLaMA—in classifying posts across a 7-point severity scale (Levels 0–6). Results indicate that Claude and GPT closely align with human annotations, while Mistral achieves the lowest ordinal prediction error. Most models exhibit ordinal sensitivity, with misclassifications typically occurring between adjacent severity levels. We further analyze confusion patterns, misclassification sources, and ethical considerations, underscoring the importance of human oversight, transparency, and cautious deployment. Full code and supplementary materials are available at https://github.com/av9ash/llm_cssrs_code . I Introduction Suicide is a leading cause of death worldwide, with over 700,000 deaths occurring each year according to the World Health Organization (WHO) [ 1 ] . Early identification and intervention for individuals at risk of suicide are crucial for effective prevention. Social media platforms have emerged as significant outlets where individuals express suicidal thoughts and seek peer support. Among these, Reddit’s r/SuicideWatch serves as an anonymous community where users share their struggles, providing a valuable resource for analyzing suicidal ideation using computational methods. Traditional suicide risk assessment methods rely on clinical evaluations and structured psychological interviews [ 2 ] . However, given the scale and anonymity of social media data, manual screening and intervention remain infeasible. This has driven research interest in automated methods for suicide risk assessment using Natural Language Processing (NLP) and machine learning techniques [ 3 , 4 ] . In particular, Large Language Models (LLMs) have demonstrated significant capabilities in understanding and classifying human emotions, making them suitable for evaluating suicide risk from textual data. The Columbia-Suicide Severity Rating Scale (C-SSRS) is a widely used and validated framework for assessing suicidal ideation and behavior [ 5 ] . While prior work has explored suicide risk classification in mental health research [ 6 ] , limited studies have applied structured clinical assessment tools like C-SSRS in conjunction with LLMs. This paper aims to bridge this gap by leveraging LLMs for evaluating r/SuicideWatch posts based on the C-SSRS scale. The key contributions of this work are: • The application of C-SSRS for structured suicide risk assessment in online social media posts. • The use of zero-shot learning with Large Language Models for automated classification of suicide risk levels. • Comparison of Model performance with traditional NLP techniques and human annotations. • Limitations of using LLMs for suicide risk screening. The remainder of this paper is structured as follows: Section 2 discusses related work in suicide risk assessment using NLP. Section 3 presents the methodology, including data preprocessing and model implementation. Section 4 details experimental results and evaluation metrics. Section 5 provides conclusions and directions for future work. II Related Work Understanding and detecting suicide risk in online text has been an active research area in recent years. Prior studies have explored traditional machine learning techniques, transformer-based models, and clinically validated scales for mental health assessment. This section discusses (1) suicide risk assessment using NLP, (2) applications of large language models in mental health, and (3) the use of the Columbia-Suicide Severity Rating Scale (C-SSRS) in computational psychiatry. II-A Suicide Risk Assessment in NLP Natural Language Processing (NLP) has been widely used to detect signs of suicidal ideation in online forums, social media, and clinical narratives. Early studies employed lexicon-based sentiment analysis to identify suicidal expressions in text [ 4 ] . Later, machine learning techniques such as Support Vector Machines (SVMs) and Random Forest classifiers improved suicide risk prediction by incorporating linguistic and psychological features [ 7 , 8 ] . With the advent of deep learning, transformer-based architectures such as BERT [ 9 ] and RoBERTa [ 10 ] have demonstrated superior performance in detecting mental health disorders from text. Recent studies apply fine-tuned BERT models on suicide-related datasets to classify posts into different risk levels [ 11 , 12 ] . However, these models often lack explainability, raising ethical concerns regarding their real-world application. II-B Large Language Models for Mental Health Large Language Models (LLMs) such as GPT-3 and GPT-4 [ 13 , 14 ] have shown promising results in mental health applications, including sentiment analysis, chatbot-based counseling, and suicide risk prediction. Studies have explored LLMs for generating synthetic mental health dialogues and assisting in psychological assessments [ 15 , 16 ] . Despite their potential, challenges such as hallucination, bias, and lack of interpretability remain major concerns in deploying LLMs for sensitive tasks like suicide screening. II-C The Columbia-Suicide Severity Rating Scale (C-SSRS) in Computational Psychiatry and Risk Assessment The Columbia-Suicide Severity Rating Scale (C-SSRS) is a clinically validated tool widely used for suicide risk assessment [ 2 ] . As illustrated in Figure 1 , it classifies risk based on the severity of suicidal ideation, intent, and history of previous attempts. Multiple studies have explored mapping social media text to C-SSRS labels due to its wide applicability [ 6 ] [ 17 ] [ 18 ] . Recent efforts integrate C-SSRS with NLP techniques to develop automated risk assessment systems, but the accuracy and reliability of such approaches remain underexplored. The C-SSRS consists of a structured set of questions designed to assess suicidal ideation and behaviors. The core questions aim to determine the presence, severity, and progression of suicidal thoughts and actions. Figure 1 provides a representation of the C-SSRS. Fig. 1 : C-SSRS: Suicide Risk Screening Tool By leveraging these structured questions, NLP models can be trained to detect suicide risk from textual data, enhancing computational psychiatry applications. However, challenges persist in ensuring that these automated methods accurately interpret and classify responses. III Dataset We collected user-generated posts from the r/SuicideWatch subreddit using the Reddit API [ 19 ] . These posts reflect individuals’ expressions of suicidal ideation, emotional distress, and personal challenges. Initially, over 3,000 posts were gathered from December 2024. From this set, approximately 1,200 posts were selected for analysis based on specific criteria, which are discussed in a later section. Prior research has established the utility and reliability of Reddit data for mental health studies [ 20 , 3 , 18 ] . In this section, we present insights from our exploratory data analysis and outline the procedures used for data annotation and labeling. III-A Exploratory Data Analysis: • Word Count Distribution by Severity: The word count of posts varies significantly across severity levels. The box plot (Figure 2 ) provides insights into the median, interquartile range (IQR), and potential outliers. Posts with higher severity scores (CSSR-S \geq 4 ) tend to exhibit greater variability in word count, with some containing substantially more words than lower-severity posts. However, lower-severity posts (CSSR-S \leq 2 ) demonstrate a more consistent word count distribution with fewer extreme values. Fig. 2