Why Do Self-Harm Prediction Models Struggle to Generalise? Lexical and Semantic Variations in Emergency Department Triage Notes

Liuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova · 2026

Read Why Do Self-Harm Prediction Models Struggle to Generalise? Lexical and Semantic Variations in Emergency Department Triage Notes by Liuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01678.txt.

Introduction

Chen, Conway, Robinson, and Rozova ask why self-harm models fail across hospitals. Self-harm visits to emergency departments link to higher suicide risk. NLP models detect self-harm well in triage notes at one hospital. Performance often drops when they move to another institution. The team compare triage notes from two hospitals. They analyze lexical traits and predictive features across the sites. Salient topics in the notes get compared too. Lexical expression varies a lot between the hospitals.

Feature importance for self-harm also shifts across sites. Core themes like self-poisoning and self-injury stay consistent, though. Documentation differences track the reduced cross-site performance. The findings show how institutional variation shapes clinical text models. They also hint at methods to improve generalization across sites.

Catalog record

CREATORLiuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Do Self-Harm Prediction Models Struggle to Generalise? Lexical and Semantic Variations in Emergency Department Triage Notes Thanks: This paper has been accepted to the Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026). Liuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova Abstract Self-harm presentations to emergency departments (EDs) are strongly associated with higher suicide risk. NLP models have shown robust performance in detecting self-harm from triage notes within single hospitals, yet performance often declines across institutions. To examine potential causes, we compare ED triage notes from two hospitals by analyzing lexical characteristics, highly associated predictive features, and salient topics. Our results reveal variation in lexical expression and feature importance related to self-harm across hospitals, despite consistent core themes such as self-poisoning and self-injury. These documentation differences are associated with reduced cross-site performance. Our findings provide insight into how institutional variation affects the identification of self-harm in clinical text and highlight potential methods to improve model generalisability. 1 Introduction Generalisability of predictive models is important in the clinical domain, where data collection and annotation are often expensive and logistically challenging Goetz et al. 2024 . Yet achieving robust generalisation remains a key challenge in real-world clinical applications for various reasons. For example, limited representativeness in the training data can impact model transfer to new datasets Goetz et al. 2024 , while the predictive factors that support or hinder generalisability are poorly understood Futoma et al. 2021 . Prior work on clinical NLP portability has also shown that model performance can degrade substantially when systems are applied to new clinical domains, motivating domain adaptation strategies that account for differences in documentation style, institution, specialty, and data-sharing constraints Laparra et al. 2020 . Self-harm – defined as “an intentional act of self-poisoning (e.g., drug overdose) or self-injury (e.g., self-cutting) regardless of suicidal intent" – has become increasingly common among young people and is a major global health concern Witt et al. 2023 . This has driven researchers to develop methods for self-harm detection in clinical documentation over the years Obeid et al. 2020 ; Ayre et al. 2021 ; Iorfino et al. 2020 ; Rozova et al. 2022 . Recently published studies using natural language processing have shown strong performance. For instance, Obeid et al. 2020 reported an AUROC of 0.88 and an F1-score of 0.77 on electronic health records (EHRs). Rozova et al. 2022 reported an AUPRC of 0.85 when applying a Gradient Boosting model to emergency department (ED) triage notes. However, these results were obtained on test sets with the same distributions as the training data. Whether these models can generalise to other contexts remains under-explored. Building on Rozova et al. 2022 ’s ( Rozova et al. 2022 ) study, we aim to examine the model’s external validity in a different hospital context. The model was trained on ED data from the Royal Melbourne Hospital (RMH), a major tertiary referral centre in Melbourne, Victoria, Australia. When applied to Latrobe Regional Health (LRH), a regional hospital serving rural communities in Victoria, model performance declined, with AUPRC decreasing from 0.85 \pm 0.01 to 0.78 \pm 0.01. To investigate the potential contributors to this cross-site generalisation gap, we conduct a corpora comparison analysis on ED triage notes between RMH and LRH. Triage notes describe the reason for patient presentation to the ED and are typically short and hastily written, resulting in non-standard grammar, misspellings and extensive use of abbreviations, which can vary considerably between hospital systems. Through the analysis of lexical characteristics, features highly associated with the outcome variable, and topic modelling, we aim to identify potential contributors to the model’s diminished performance when ported to a different context. 2 Methodology 2.1 Datasets This study uses ED triage notes from RMH and LRH between 2012 and 2017 Witt et al. 2023 , which were manually annotated by psychology experts in suicide prevention as positive ( Self-harm ) or negative ( Control ). The RMH dataset consisted of N=399,111 notes with 1.4% notes annotated as positive. The LRH dataset contained N=171,170 notes, of which 1.7% were considered positive. RMH is a large metropolitan public tertiary hospital, while LRH is the principal referral hospital for the Gippsland region, serving a predominantly rural and regional catchment. Table 1 summarises key contextual differences between the two hospitals. Table 1: Comparisons between RMH and LRH. RMH LRH Funding Public Public Location Metropolitan Regional ED records, 2012–2017 399,111 171,170 Catchment area 137 sq. km 42,000 sq. km Population served 550,000 300,000 Emergency mental health team Yes No Age 48 \pm 21 47 \pm 23 Sex Female 48% 52% Male 52% 48% Intersex <1\% <1\% Unknown <1\% <1\% 2.2 Corpora Comparison To compare the corpora, we examined lexical characteristics (e.g., number of characters and sentences), identified important features in each dataset, and applied topic modelling to explore the most frequent topics in the self-harm group. Important Features Selection Following Rozova et al. 2022 , we employed TF–IDF representations to ensure comparability and enable external validation. We computed the TF-IDF matrix and selected important features using both the Chi-Square and XGBoost algorithms. Chi-Square measures the correlation between a feature and the target variable (i.e., self-harm), and features with p < .001 were retained. XGBoost feature importance reflects each feature’s predictive contribution, and we selected features above the 95th percentile (top 5%). We applied Chi-Square test and XGBoost independently on each dataset, both across all years and separately for each year. The final set of important features was the intersection of features selected by both methods. We further compare these final selected important features between the two hospitals. Topic Modelling To explore common themes in the self-harm class, BERTopic was applied Grootendorst 2022 separately to each dataset, using only the positive class. We selected ‘all-mpnet-base-v2’ as our sentence embedding model, which currently performs the best in capturing semantic information HuggingFace 2025 . The topic-wise embedding semantic similarity between hospitals was calculated by cosine similarity. To identify unique topics, we set a threshold of 0.75, selected based on a sensitivity analysis across the 0.60–0.90 range. 3 Results 3.1 Lexical Characteristics Figure 1 provides descriptive statistics regarding the number of characters and sentences. (a) Character count (b) Sentence count Figure 1: Lexical characteristics 3.2 Comparison of Important Features 3.2.1 Overall Comparison Across the entire dataset, we identified 472 unigram, 701 bigram, and 637 trigram important features at RMH and 365 unigram, 460 bigram, and 426 trigram features at LRH. The top 10 selected features for each hospital are presented in Appendix, Table 5 . Of those, 213 unigram, 156 bigram, and 136 trigram features were shared between the two datasets, indicating their transferability. Table 2 presents the 10 most important shared features. As shown, these features mostly contain generic self-harm related keywords (e.g., self harm , self inflicted ), self-harm methods (predominantly self-poisoning, e.g., od , intentional od ), and self-injury (e.g., superficial cuts to ). Suicide-related keywords are also common in shared features, such as suicidal , suicide , and suicide intent . Shared features Unigram od, pain, self, tablets, overdose, sob, resolved, suicidal, suicide, attempt Bigram self inflicted, od of, self harm, intentional od, suicide attempt, to end, to kill, polypharmacy od, wanted to, suicide intent Trigram superficial lac(s) to, superficial cuts

Read the complete text