Hierarchical Dual-Head Model for Suicide Risk Assessment via MentalRoBERTa

Chang YANG Department of Informatics King’s College London London, United Kingdom [email protected] Ziyi WANG School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing, China [email protected] Wangfeng TAN School of Computer Science and Technology Anhui · 2025

Read Hierarchical Dual-Head Model for Suicide Risk Assessment via MentalRoBERTa by Chang YANG Department of Informatics King’s College London London, United Kingdom [email protected] Ziyi WANG School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing, China [email protected] Wangfeng TAN School of Computer Science and Technology Anhui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20085.txt.

Introduction

Chang Yang and five co-authors propose a hierarchical dual-head network for judging suicide risk from social media posts in four levels, from indicator through ideation and behavior to attempt. Risk levels are both ordered and categorical, posts arrive over time, and the classes are heavily unbalanced, which makes the task hard. Their model builds on MentalRoBERTa, a language model tuned for mental health text.

One prediction head preserves the ordering of the risk levels, while a second head allows flexible category distinctions on the same shared representation. A transformer encoder with time interval embeddings captures posting patterns over time. A combined loss balances ordering, overconfidence, and class imbalance during training with cross-validation.

Catalog record

CREATORChang YANG Department of Informatics King’s College London London, United Kingdom [email protected] Ziyi WANG School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing, China [email protected] Wangfeng TAN School of Computer Science and Technology Anhui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20085

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Dual-Head Model for Suicide Risk Assessment via MentalRoBERTa Chang YANG Department of Informatics King’s College London London, United Kingdom [email protected] Ziyi WANG School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing, China [email protected] Wangfeng TAN School of Computer Science and Technology Anhui University An Hui, China [email protected] Zhiting TAN School of Software Nankai University Tianjin, China [email protected] Changrui JI Department of Digital Humanities King’s College London London, United Kingdom [email protected] Zhiming ZHOU School of Arificial Intelligence Anhui University Anhui, China [email protected] Abstract Social media platforms have become important sources for identifying suicide risk, but automated detection systems face multiple challenges including severe class imbalance, temporal complexity in posting patterns, and the dual nature of risk levels as both ordinal and categorical. This paper proposes a hierarchical dual-head neural network based on MentalRoBERTa for suicide risk classification into four levels: indicator, ideation, behavior, and attempt. The model employs two complementary prediction heads operating on a shared sequence representation: a CORAL (Consistent Rank Logits) head that preserves ordinal relationships between risk levels, and a standard classification head that enables flexible categorical distinctions. A 3-layer Transformer encoder with 8-head multi-head attention models temporal dependencies across post sequences, while explicit time interval embeddings capture posting behavior dynamics. The model is trained with a combined loss function (0.5 CORAL + 0.3 Cross-Entropy + 0.2 Focal Loss) that simultaneously addresses ordinal structure preservation, overconfidence reduction, and class imbalance. To improve computational efficiency, we freeze the first 6 layers (50%) of MentalRoBERTa and employ mixed-precision training. The model is evaluated using 5-fold stratified cross-validation with macro F1 score as the primary metric. I Introduction Suicide is a major global public health issue, with over 700,000 deaths reported annually by the World Health Organization [ 1 ] . Early identification of at-risk individuals is critical for timely intervention and prevention. In recent years, social media platforms have become venues where individuals express mental distress and suicidal thoughts, often before seeking professional help [ 2 ] . Reddit, in particular, hosts numerous mental health communities (e.g., r/depression, r/SuicideWatch) where users discuss anxiety and suicidal ideation. This linguistic data provides opportunities for developing automated risk detection systems based on natural language processing. However, suicide risk detection from social media text presents several technical challenges. First, the data exhibits severe class imbalance. High-risk samples (behavior and attempt categories) are significantly less frequent than low-risk samples (indicator and ideation), causing standard classifiers to be biased toward majority classes [ 3 ] . Second, risk levels have dual characteristics: they follow an ordinal progression (indicator < ideation < behavior < attempt) in terms of severity, but also have categorical distinctions that require different clinical intervention strategies. Existing models typically employ either ordinal regression methods or standard classification, but not both, thus missing complementary information [ 4 ] . Third, suicide risk evolves over time, reflected in temporal patterns across multiple posts. Capturing these temporal dependencies requires sequence modeling beyond simple bag-of-words or single-post analysis [ 5 ] . A sudden increase in posting frequency or shift in sentiment may indicate crisis escalation. Fourth, while large pre-trained language models like BERT and RoBERTa offer strong semantic understanding [ 6 ] , they are computationally expensive to fine-tune on domain-specific mental health data, requiring efficient training strategies. Recent work has explored deep learning approaches for suicide risk detection, including LSTM-based sequence models [ 7 ] and BERT-based classifiers. However, most existing methods focus on single-head architectures with simple loss functions like cross-entropy. Few approaches explicitly model both the ordinal structure and categorical distinctions of risk levels, or combine multiple complementary objectives to address class imbalance and temporal evolution simultaneously. This paper proposes a hierarchical dual-head model that addresses these challenges through several key innovations. First, this paper employs a dual-head architecture combining CORAL ordinal regression [ 8 ] and standard classification. Both heads operate on the same learned sequence representation, allowing their outputs to be ensembled at inference time for more robust predictions. The CORAL head is trained with a single CORAL loss to preserve ordinal relationships, while the classification head is optimized with a combined loss function incorporating both cross-entropy with label smoothing and focal loss to address overconfidence and class imbalance simultaneously. Second, this paper uses a 3-layer Transformer encoder to model inter-post dependencies and temporal evolution. Time intervals between consecutive posts are explicitly encoded via a learned embedding network with LayerNorm, capturing posting frequency patterns that signal mental state changes. Third, this paper optimizes a weighted combination of three losses: CORAL loss for preserving ordinal structure, cross-entropy with label smoothing to prevent overconfident predictions, and focal loss to emphasize minority classes and hard samples. The fixed weights (0.5, 0.3, 0.2) balance these complementary objectives. Fourth, this paper employs MentalRoBERTa, a RoBERTa model pre-trained on Reddit mental health texts, and freezes 50% of its parameters to reduce computational cost while maintaining domain knowledge. This paper also applies mixed-precision training to reduce memory usage. Finally, the model supports optional statistical feature extraction, including post length patterns and time interval statistics, which can be fused with the text representation when enabled. The main contributions of this work are: • A dual-head architecture combining CORAL ordinal regression and classification for suicide risk prediction, leveraging complementary prediction paradigms through ensemble. • Temporal modeling through a 3-layer Transformer encoder with explicit time interval embeddings to capture posting behavior dynamics and mental state evolution. • A combined loss function (CORAL + Cross-Entropy + Focal) addressing ordinal structure preservation, overconfidence reduction, and class imbalance simultaneously with fixed weights. • Efficient training strategy with partial RoBERTa freezing (6 layers, 50% parameters), differentiated learning rates, and mixed-precision computation. • Model evaluation framework using 5-fold stratified cross-validation with macro F1 score as the primary performance metric. II Methodology II-A Problem Formulation We formulate suicide risk assessment as predicting a user’s risk level y\in\{0,1,2,3\} from their chronologically ordered sequence of N posts \mathcal{P}_{u}=\{p_{1},p_{2},\ldots,p_{N}\} with corresponding timestamps \mathcal{T}_{u}=\{t_{1},t_{2},\ldots,t_{N}\} . Each post p_{i} is a text string, and each timestamp t_{i} is a Unix epoch time. The four risk levels are defined as: (0) indicator - expression of suicide-related indicators or warning signs; (1) ideation - expression of suicidal thoughts without specific plans; (2) behavior - planning or preparatory actions for suicide; (3) attempt - explicit intent to attempt or history of past attempts. These labels exhibit ordinal relationships ( 0<1<2<3 ) reflecting increasing severity, while also requiring categorical distinction for clinical intervention design. In practice, we construct sequences by grouping consecutive posts from the same user. Each sequence consists of 5 consecutive posts with their corresponding timestamps, enabling temporal analysis of posting patterns. The risk level label is determined by the 6th post immediately following the sequence, which serves as the ground truth for predicting the user’s current risk state. II-B Model Architecture Fig. 1: Overview of the hierarchical dual-head architecture. Figure 1 shows the model architecture. The model operates at two hierarchical levels: post-level encoding processes individual posts to obtain semantic representations, and sequence-level aggregation models the temporal sequence of posts to produce a fixed sequence representation. The final prediction is made by two complementary heads operating on this sequence representation. The architecture consists of the following components: 1) Post-Level Encoding

Read the complete text