Speech as a Multimodal Digital Phenotype for Multi-Task LLM-based Mental Health Prediction

Mai Ali 1 , Christopher Lucasius *1 , Tanmay P. Patel *2 , Madison Aitken 3,4 , Jacob Vorstman 5,6 , Peter Szatmari 3,5,6 , Marco Battaglia 6 , Deepa Kundur 1 · 2025

Read Speech as a Multimodal Digital Phenotype for Multi-Task LLM-based Mental Health Prediction by Mai Ali 1 , Christopher Lucasius *1 , Tanmay P. Patel *2 , Madison Aitken 3,4 , Jacob Vorstman 5,6 , Peter Szatmari 3,5,6 , Marco Battaglia 6 , Deepa Kundur 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23822.txt.

Introduction

Mai Ali and co-authors use recordings of speech to predict mental health problems in teenagers, in a 2025 paper. Speech is easy to collect and carries clues about mood, yet most studies treat it as sound alone. Their system reads speech three ways at once: the words spoken, landmarks in the sound, and measures of the voice. A large language model then judges all three streams together.

The system predicts three outcomes at the same time: depression, suicidal thoughts, and sleep problems. It also follows how a patient's speech changes across visits. On a clinical data set, the full setup reached a balanced accuracy of about 71 percent. That score beat versions using one stream, one task, or single visits only. The authors argue speech deserves treatment as a rich, multi-part health signal.

Catalog record

CREATORMai Ali 1 , Christopher Lucasius *1 , Tanmay P. Patel *2 , Madison Aitken 3,4 , Jacob Vorstman 5,6 , Peter Szatmari 3,5,6 , Marco Battaglia 6 , Deepa Kundur 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speech as a Multimodal Digital Phenotype for Multi-Task LLM-based Mental Health Prediction Mai Ali 1 , Christopher Lucasius *1 , Tanmay P. Patel *2 , Madison Aitken 3,4 , Jacob Vorstman 5,6 , Peter Szatmari 3,5,6 , Marco Battaglia 6 , Deepa Kundur 1 Abstract Speech is a noninvasive digital phenotype that can offer valuable insights into mental health conditions, but it is often treated as a single modality. In contrast, we propose the treatment of patient speech data as a trimodal multimedia data source for depression detection. This study explores the potential of large language model-based architectures for speech-based depression prediction in a multimodal regime that integrates speech-derived text, acoustic landmarks, and vocal biomarkers. Adolescent depression presents a significant challenge and is often comorbid with multiple disorders, such as suicidal ideation and sleep disturbances. This presents an additional opportunity to integrate multi-task learning (MTL) into our study by simultaneously predicting depression, suicidal ideation, and sleep disturbances using the multimodal formulation. We also propose a longitudinal analysis strategy that models temporal changes across multiple clinical interactions, allowing for a comprehensive understanding of the conditions’ progression. Our proposed approach, featuring trimodal, longitudinal MTL is evaluated on the Depression Early Warning dataset. It achieves a balanced accuracy of 70.8%, which is higher than each of the unimodal, single-task, and non-longitudinal methods. Emails: {maia.ali, christopher.lucasius, tanmay.patel}@mail.utoronto.ca [email protected], [email protected], {peter.szatmari, marco.battaglia}@utoronto.ca, [email protected] * * footnotetext: These authors contributed equally to this work. I Introduction Depression, suicidal ideation, and sleep disturbances are prevalent and interconnected mental health problems among adolescents. Major depressive disorder (MDD) affects 8–12% of adolescents globally [ 1 ] , while suicide is the second leading cause of death among individuals aged 15–24 [ 1 ] . Sleep disturbances impact 20–50% of adolescents, increasing the risk of depression and suicidality [ 2 ] . Depression typically emerges between the ages of 12 and 18, with 20% of adolescents experiencing a major depressive episode by 18 [ 3 ] . Sleep disturbances, driven by biological and lifestyle factors, are strongly linked to depression and suicidal ideation. Notably, sleep deprivation independently raises the risk of suicidality, with each hour of sleep loss increasing suicidal thoughts by 11% [ 3 ] . Given the bidirectional relationships among these conditions, integrating sleep and suicidality assessments into depression screenings could enhance early detection and intervention. I-A The Need for Multimodal AI-Driven Mental Healthcare Speech provides a non-invasive and suprisingly multimodal source for assessing mental health, allowing for the extraction of linguistic (word choice, syntax) and paralinguistic (tone, pitch, rhythm) features [ 4 ] . Studies have shown that individuals with depression and suicidal ideation exhibit distinct speech characteristics, including slower speech rate, longer pauses, and increased monotonicity [ 5 ] . Additionally, linguistic patterns in speech, such as self-referential language and negative emotion words, have been strongly linked to depression and suicidality [ 6 ] . Recent advancements in large language models (LLMs) have revolutionized speech and text-based mental health assessments. LLMs pre-trained on large corpora of conversational and clinical data capture subtle linguistic and acoustic markers associated with depression, suicidality, and sleep disturbances, making them well-suited for automated screening in adolescents [ 7 ] . In addition, multi-task learning (MTL) approaches allow LLMs to model shared representations across multiple mental health conditions, improving generalization and reducing the need for extensive labeled data [ 8 ] . I-B Contributions While the use of LLMs for depression detection has grown into a rich field, there are several notable gaps. We offer the following innovations in this space: 1. A comprehensive multimedia framework for depression detection that integrates three speech-derived modalities: speech transcriptions, acoustic landmarks [ 9 ] , and vocal biomarkers. This unified approach provides a more holistic understanding of speech-based depression indicators. 2. A longitudinal analysis framework that tracks changes across multiple patient-physician interactions by treating the interactions collectively as a continuous LLM ‘conversation’. This type of analysis is not novel; however, to the best of our knowledge, we are the first to apply it to multimodal speech and text-based mental health analysis, an area where temporal understanding is typically viewed by clinicians to be critical [ 10 ] . 3. A multi-task learning architecture that extends our trimodal approach beyond depression detection to related clinical research assessments, leveraging multimodal speech data and maximizing the utility of data collected across multiple related domains. We assess these contributions relative to current state-of-the-art methods on the Depression Early Warning (DEW) dataset, a longitudinal multimodal dataset described in Section III-A . II Background and Literature Review The potential of machine learning models for predicting mental health conditions has been well established in the literature. Prior studies have explored various modalities as predictive biomarkers, including actigraphy-based activity and sleep patterns, ecological momentary assessments, facial expressions, and speech characteristics [ 11 ] . Recent advancements in LLMs have enhanced their ability to process long-form transcripts and infer underlying cognitive and emotional states. These models present a novel opportunity to extract clinically relevant indicators of mental health conditions from speech data, facilitating more accurate and scalable diagnostic support systems [ 12 ] . II-A Prediction Based on Speech Semantics and Text In recent months, the use of LLMs to detect depression from patients’ text or transcribed speech data has become an expanding research area. A central work is that of Xu et al. [ 7 ] which performs systematic benchmarking of widespread general-purpose LLMs against mental health classification tasks. They find limited, but promising, potential in zero-shot and few-shot regimes. Their work introduces two new LLMs—Mental-Alpaca and Mental-FLAN-T5—that were instruction fine-tuned for multi-task mental health classification. They outperform significantly larger pre-trained models [ 7 ] . Another active research area is the use of longitudinal methods for LLM-based depression detection that leverage connections between text samples from different points in a patient’s history. They capture a patient’s treatment trajectory and use the conversational nature of LLMs to analyze this dimension. However, existing works employ longitudinal methods in single modality regimes, chiefly on text data. One example is [ 13 ] which uses tweets collected over long periods to engage an LLM in a ‘conversation’ that generates a prediction capable of capturing the time-dependence of patient data. We extend these ideas to our multimodal framework by treating each clinical research assessment as an episode that’s part of a larger LLM interaction (see Section IV-D ). II-B Prediction Based on Vocal Biomarkers Several studies have explored the use of various speech features, with an emphasis on vocal biomarkers (e.g., fundamental frequency, mel-frequency cepstrum coefficients (MFCC)), which have been found to correlate with mental disorders. Tasnim et al. propose a dataset for machine learning-based depression detection that extracts hand-curated vocal features from patient speech samples (based on clinical domain knowledge) to enhance predictive power [ 14 ] . Our work unifies these biomarkers with other speech-based modalities, discussed below, that are amenable to LLM-based analysis. The biomarkers we use are outlined in Section IV-A3 . II-C Prediction Based on Acoustic Landmarks Although transcribed speech data serve as a natural input to language models, speech data contain other multimedia features that are amenable to LLM-based analysis for depression detection. An approach introduced in [ 9 ] extracts

Read the complete text