Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants

Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon Yoonsoo Nam, Myles Phung Jonathan May John R. Blosnich Swabha Swayamdipta · 2025

Read Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants by Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon Yoonsoo Nam, Myles Phung Jonathan May John R. Blosnich Swabha Swayamdipta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18541.txt.

Introduction

Jaspreet Ranjit and co-authors test language models as helpers in suicide prevention data work in a 2025 computing paper. A national system records the facts around each suicide in long text narratives. Human coders extract fifty set facts from each record by hand, a slow and heavy task. The team asks whether language models can speed the coding and even improve it.

Model answers matched the human coding about eighty-five percent of the time. Where they disagreed, expert review found the model had caught real human errors in over a third of cases. The authors sketch a workflow where models assist rather than replace the experts. Careful help like this could free specialists to study prevention instead of paperwork.

Catalog record

CREATORJaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon Yoonsoo Nam, Myles Phung Jonathan May John R. Blosnich Swabha Swayamdipta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18541

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon Yoonsoo Nam, Myles Phung Jonathan May John R. Blosnich Swabha Swayamdipta Abstract The National Violent Death Reporting System (NVDRS) documents information about suicides in the United States, including free text narratives (e.g., circumstances surrounding a suicide). In a demanding public health data pipeline, annotators manually extract structured information from death investigation records following extensive guidelines developed painstakingly by experts. In this work, we facilitate data-driven insights from the NVDRS data to support the development of novel suicide interventions by investigating the value of language models (LMs) as efficient assistants to these (a) data annotators and (b) experts. We find that LM predictions match existing data annotations about 85% of the time across 50 NVDRS variables. In the cases where the LM disagrees with existing annotations, expert review reveals that LM assistants can surface annotation discrepancies 38% of the time. Finally, we introduce a human-in-the-loop algorithm to assist experts in efficiently building and refining guidelines for annotating new variables by allowing them to focus only on providing feedback for incorrect LM predictions. We apply our algorithm to a real-world case study for a new variable that characterizes victim interactions with lawyers and demonstrate that it achieves comparable annotation quality with a laborious manual approach. Our findings provide evidence that LMs can serve as effective assistants to public health researchers who handle sensitive data in high-stakes scenarios. 1 Introduction Figure 1: LM assistants can reduce the emotional burden of annotating NVDRS narratives when a codebook is available (top) and help experts efficiently develop new codebooks for novel variables (bottom). When a codebook is available for a variable, we incorporate it in the LM instruction for Chain-of-Thought predictions. For a new variable of interest without an existing codebook, we propose an efficient, LM-assisted expert codebook development algorithm. Here experts focus on providing feedback for incorrect LM predictions, reducing manual codebook development time from weeks to hours (bottom right). For the annotation of a novel variable: victim interactions with legal professionals, we find that our algorithm results in similar annotation quality to a fully manual approach. Warning: This paper discusses topics of suicide and suicidal ideation, which may be distressing to some readers. Each year, approximately 50,000 people in the United States fall victim to suicide ( Cammack 2024 ) . The Centers for Disease Control and Prevention (CDC) documents this information in the National Violent Death Reporting System (NVDRS) 1 1 1 https://www.cdc.gov/nvdrs/about/index.html , which contains structured (e.g., more than 600 demographic and circumstance variables) and unstructured data (e.g., narrative summaries surrounding the circumstances of death) for more than 270,000 suicides ( Figure 1 ; top left). Structured data in NVDRS is manually labeled by annotators (or NVDRS data abstractors 2 2 2 We use the term data abstractors and annotators interchangeably. ) using codebooks—precise definitions and annotation guidelines—developed painstakingly by experts. Given the sensitive nature of the topic and the scale of NVDRS, a data annotator’s job can be extremely demanding and emotionally taxing ( Fincham et al. 2008 ; Murthy 2024 ; Nazarov et al. 2019 ) . Moreover, variation in manual reporting and data abstraction leaves room for annotation inconsistencies ( Dang et al. 2023 ) . Given its scale, NVDRS can be an extremely valuable source for data-driven discovery of novel opportunities for suicide intervention. However, identifying such opportunities in NVDRS is not trivial due to complex and overlapping risk factors in the victim’s circumstances, such as mental health and life disruptions ( cdc 2024 ; Wang et al. 2024 ) . To uncover actionable insights for new suicide interventions, experts must move beyond existing structured variables and extract new, data-driven evidence from qualitative narratives recorded as unstructured variables. One such example is the presence of an interaction between the victim and legal professionals in the days before their death; such interactions may be common given that many victims experience triggering life disruptions (e.g., divorce). Experts hypothesize that the evidence of such interactions may lead to targeted interventions for legal professionals to help clients at risk. However, verifying this hypothesis requires experts to manually analyze NVDRS narratives, develop a codebook for this new variable, and abstractors to retroactively annotate each of the 270K cases. Not only would this process be time-consuming and laborious, but it would also place a severe emotional burden on the data abstractors. In this work, we ask: Can language models help public health and social work researchers improve efficiency and reduce the emotional labor of manual data annotation and analysis for suicide prevention research? Our investigation is based on the promise language models have shown in social science ( Rytting et al. 2023 ; Pangakis et al. 2023 ; Halterman & Keith 2024 ; Ziems et al. 2024 ) . Specifically, we are interested in two research questions: RQ1: Can a language model alleviate the burden of annotating variables with existing reference codebooks in NVDRS and surface discrepancies between the structured and unstructured data? RQ2: Can a language model assist experts in the annotation of new variables that go beyond current NVDRS codebooks, potentially leading to new intervention opportunities? For RQ1 , we deploy LMs as annotation assistants to data abstractors to reduce the emotional burden of analyzing sensitive and explicit data, as well as surface potential inconsistencies between the narratives and the annotated structured variables. Our LM assistant achieves an average agreement of 85% with data abstractors across 50 variables (§ 2.2.1 ), making it a reliable peer validator for high-agreement variables. The assistant is also useful for low-agreement variables: from our expert review of six variables with the lowest agreement, we find that our LM assistant surfaces annotation inconsistencies for 38% of the instances where the LM disagrees with the data abstractor (§ 2.2.2 ). We address RQ2 by introducing a human-in-the-loop codebook development algorithm that helps experts iteratively develop codebook guidelines for new variables by providing feedback for incorrect LM predictions (§ 3.1 ). We first demonstrate the effectiveness of our algorithm by developing codebooks for variables with existing guidelines using only the variable name as our starting point. Our experiments show that our algorithm produces guidelines that enable LMs to achieve an average 80% agreement with data annotators, outperforming LMs conditioned on the existing NVDRS guidelines (75% agreement) for 12 variables (§ 3.2 ). Figure 1 illustrates our approach. We further demonstrate that our algorithm is effective at characterizing new variables with a real-world case study: identifying (explicit or implicit) victim interactions with legal professionals (§ 3.3 ). We compare our algorithm to a fully manual codebook development process, where we collect annotations for characterizing victim interactions with legal professionals from a suicide prevention research expert on 634 NVDRS narratives ( Halterman & Keith 2024 ; Rytting et al. 2023 ) . We find that our algorithm reduces codebook development time from weeks to hours without compromising on annotation quality, demonstrating its potential to efficiently develop guidelines for a wider range of variables (e.g., interactions with other nonclinical professionals). In summary, our results indicate the ability of language models to serve as efficient assistants that enable experts to mobilize the NVDRS dataset more

Read the complete text