Multimodal machine learning for video based single question mental health assessment Authors: Bradley Grimm, Pernille Yilmam, Brett Talbot, Loren Larsen Published: 2025 ABSTRACT This study demonstrates that a single video question can predict self-reported depression (PHQ-9), anxiety (GAD-7), and trauma (PCL-5) through text and voice analysis. As mental health screening needs increase, efficient multi-condition assessment methods could reduce patient burden in clinical settings. Our multimodal model, integrating MPNet for textual analysis and HuBERT for voice prosody, was trained on data from 2420 participants. Our approach achieves 64.6% reduced assessment time (78.4 s vs 221.7 s) while screening all three conditions from one response, with only 1.4% of participants unwilling to use video-based screening. Results demonstrate strong performance and demographic consistency across age, gender, and race/ethnicity supporting the feasibility of efficient multi-condition screening from brief video responses. Introduction Since 1990, the prevalence of mental illness has risen by 48.1%1,2, further exacerbated by the COVID-19 pandemic, which led to a widespread global increase in rates of depression and anxiety3. In parallel, the United States and other countries face a significant shortage of mental health providers4,5, significantly affecting timely access to assessment and treatment6. This paper presents a simplified, multimodal assessment model that predicts mental health assessment scores using a single open-ended question, demonstrating substantial time efficiency compared to traditional sequential questionnaire administration. Depression, anxiety and PTSD are among the most common mental health challenges in primary care settings, frequently co-occur, and their comorbidity significantly impacts treatment planning and outcomes7,8. These conditions are typically screened for with standardized self-report surveys, such as the Generalized Anxiety Disorder 7 (GAD-7), Patient Health Questionnaire 9 (PHQ-9), and PTSD Checklist (PCL-5), each of which is well-validated for identifying clinical signs of anxiety, depression, and PTSD, respectively. However, despite their scalability, these surveys have limited scope, often assessing only one mental health condition at a time. Even brief instruments like the PHQ-9 or GAD-7 can feel repetitive or disengaging in high-volume or longitudinal settings, potentially leading to fatigue and reduced engagement that compromise reliability. In contrast, clinical interviews offer more comprehensive and engaging assessments but require substantial time from an already limited pool of mental health professionals, creating an efficiency bottleneck compounded by provider shortages. To address these limitations, our study introduces a model that consolidates multiple mental health assessments into a single spoken response, enabling simultaneous screening for depression, anxiety, and trauma symptoms with substantial time savings compared to sequential questionnaire administration. Through iterative testing, we identify a single question that effectively predicts scores on validated assessments for depression, anxiety, and trauma: “In the last 2 weeks have you felt down, nervous, depressed, anxious, hopeless or on edge? If so, please explain in detail how it has bothered you or impacted your life?” This question, which we refer to as the “Feelings Merged” question throughout this paper, combines elements from previously validated assessment tools while simultaneously encouraging open-ended responses suitable for multimodal video analysis. Our research demonstrates that responses to this question can predict PHQ-9, GAD-7, and PCL-5 scores with performance comparable to traditional questionnaire administration. Recent advancements in remote assessment technologies, including biosensors and predictive algorithms, have begun to alleviate the mental healthcare/mental health screening bottleneck9–11. Often deployed via smartphones or web browsers and frequently using video-based methods, these tools capture real-time thoughts, feelings, and behaviors through methods such as voice analysis (prosodic features like tone and rhythm), semantic analysis (linguistic content), and facial expression analysis12–15. Compared to validated metrics such as GAD-7 and PHQ-9, studies are finding that remote assessment technologies can reliably identify individuals with mental health challenges11,16,17. While this is promising for building scalable and cost-effective mental health screenings, significant challenges remain in implementing these tools. This paper demonstrates three methodological advances for remote assessment technologies. First, traditional remote assessments rely on surveys like the GAD-7 and PHQ-9, requiring a lengthy series of questions to screen for multiple conditions or risking an incomplete evaluation that overlooks comorbidities. In 2022, we showed that GAD-7 and PHQ-9 scores could be estimated from answers to five questions using semantic, voice, and facial analysis11. Here, we reduce assessment to a single question while improving model accuracy, substantially reducing respondent burden while preserving assessment quality. Compared to traditional administration of three separate questionnaires (PHQ-9, GAD-7, PCL-5), our single-question approach achieves a 64.6% time reduction (78.4 s versus 221.7 s, n = 438, p < 0.001), enabling comprehensive multi-condition screening with high user acceptance (90.7% willing to use video-based screening). Second, trauma history increases risk for mental illness and complicates treatment planning18. While our model does not assess trauma exposure directly, it predicts scores on the PCL-5, a validated indicator of current PTSD symptoms and their severity, alongside anxiety and depression scores, all from a single question. This enables a more complete understanding of each individual’s mental health needs. Third, while previous research has typically focused on single modalities - with some studies examining text-based patterns12,19, others analyzing speech characteristics15, and still others investigating semantic content14. Recent multimodal studies have demonstrated enhanced predictive power by combining multiple data streams. Our approach combines semantic content (text) with prosodic features (voice characteristics) within an ultra-brief single-question format. Using a multimodal approach across five independent cohorts from 2022 to 2024, our study has developed a simplified, scalable assessment method that reliably predicts validated mental health scores (PHQ-9, GAD-7, and PCL-5) with a single, empirically validated question. This method demonstrates strong performance across diverse demographic groups and provides practical clinical utility by enabling providers to identify patients requiring further evaluation based on severity thresholds. By incorporating response quality analysis, the model can determine when responses may be insufficient for accurate assessment, improving overall precision. Our model provides an efficient alternative to sequential questionnaire administration, reducing assessment time by nearly two-thirds while maintaining comparable predictive performance. By consolidating multiple screening assessments into a single interaction, this approach could reduce time burden in settings requiring routine multi-condition mental health screening. Our goal is to demonstrate an efficient, multimodal screening approach that predicts established questionnaire scores (PHQ-9, GAD-7, PCL-5) from a single video response. By combining semantics (what people say) and prosody (how they say it), the approach enables multi-condition assessment in one step with substantial time savings. This framework could potentially extend to other assessment domains, as video-based analysis can capture multiple signal types simultaneously. Video input creates opportunities to layer on models for movement disorders such as tardive dyskinesia, while audio features support disorders such as Parkinson’s disease, pointing toward a unified platform for multimodal clinical assessment. Results Our multimodal approach, which integrates semantic text analysis (MPNet) and audio prosody features (HuBERT), proved effective in predicting validated mental health scores (PHQ-9, GAD-7, PCL-5) from a single question. Results showed consistency across diverse demographic groups, supporting the potential of this streamlined assessment tool for scalable, remote mental health screening. We measured the performance across several candidate question texts and, while multiple options showed promising results, the “Feelings Merged” question emerged as the top performer, achieving strong predictive accuracy (AUC > 0.90) across all targets. We explore the performance of these questions in more detail in the following section. Question performance We previously reported that GAD-7 and PHQ-9 scores could be predicted from a 5-question video-based assessment11. Of the five questions, four were adapted from the PHQ-2 and the GAD-2 to allow for open-ended questions, and the fifth, presented in a similar format, asked about feelings more generally. In the present study, we reduced the assessment to a single question while improving predictive performance for PHQ-9 and GAD-7 and adding PCL-5 prediction. To identify the optimal single question, we undertook a systematic evaluation process. We developed 12 unique questions (Table 1) to evaluate the effectiveness of different question formulations. For each question, we analyzed both the semantic content and prosodic features of video responses to predict PHQ-9, GAD-7, and PCL-5 scores. While all questions were assessed for depression and anxiety predictions, a subset was specifically designed and evaluated for trauma assessment. The comparative performance of these questions provided insights into which formulations were most effective at eliciting diagnostically valuable responses.Table 1Question texts and cohort distribution for each assessmentQuestionN = 9128PCL-5CohortQuestion textFeelings general1327N1Please explain, in detail, your feelings during the past 2 weeks, and how it has impacted your life.PHQ 11445N1In the last 2 weeks have you had little interest or pleasure in doing things? If so, please explain in detail how has it bothered you or impacted your life?PHQ 21421N1In the last 2 weeks have you felt down, depressed or hopeless? If so, please explain in detail how has it bothered you or impacted your life?GAD 11462N1In the last 2 weeks have you felt nervous, anxious or on edge? If so, please explain in detail how it has bothered you or impacted your life?GAD 21437N1In the last 2 weeks have you not been able to stop or control worrying? If so, please explain in detail how it has bothered you or impacted your life?Feelings 1181Y2Regarding your health, tell us about how you’ve been feeling in the last 2 weeks. Describe what’s going well and what’s not?Feelings 2164Y2Regarding your health, tell us about how you’ve been feeling in the last few weeks. Please describe any notable events that have impacted you.Feelings 3171Y2Regarding your health, please describe what has been going on in the past few weeks and how you have been feeling.Big Events153Y2Please describe any big events from the past few weeks, positive or negative that have impacted your overall mood or well-being.Yesterday164Y2Please think about yesterday, from the morning until the end of the day. Think about where you were, what you were doing, who you were with, and how you felt. Would you like to have more days just like yesterday? Please explain why or why not.Feelings Merged642Y3In the last 2 weeks have you felt down, nervous, depressed, anxious, hopeless or on edge? If so, please explain in detail how it has bothered you or impacted your life?Stressful468Y3Regarding any memorable stressful experience(s), have you felt fear, horror, anger, guilt, or shame about the experience in the last 2 weeks? If so, please explain in detail how it has bothered you or impacted your life?Feelings Merged (Spanish)93Y4Translated into Spanish:¿En las últimas dos semanas, te has sentido triste, nervioso, deprimido, ansioso, sin esperanza o al límite? Si es así, por favor explica con detalle cómo eso ha alterado o impactado tu vida.The table lists each question prompt, the number of responses (N), whether it was used in the PCL-5 cohort, the cohort number, and the exact phrasing shown to participants. Spanish-translated variants are also included to reflect cross-language applicability.PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL-5 PTSD checklist. Table 1Question texts and cohort distribution for each assessmentQuestionN = 9128PCL-5CohortQuestion textFeelings general1327N1Please explain, in detail, your feelings during the past 2 weeks, and how it has impacted your life.PHQ 11445N1In the last 2 weeks have you had little interest or pleasure in doing things? If so, please explain in detail how has it bothered you or impacted your life?PHQ 21421N1In the last 2 weeks have you felt down, depressed or hopeless? If so, please explain in detail how has it bothered you or impacted your life?GAD 11462N1In the last 2 weeks have you felt nervous, anxious or on edge? If so, please explain in detail how it has bothered you or impacted your life?GAD 21437N1In the last 2 weeks have you not been able to stop or control worrying? If so, please explain in detail how it has bothered you or impacted your life?Feelings 1181Y2Regarding your health, tell us about how you’ve been feeling in the last 2 weeks. Describe what’s going well and what’s not?Feelings 2164Y2Regarding your health, tell us about how you’ve been feeling in the last few weeks. Please describe any notable events that have impacted you.Feelings 3171Y2Regarding your health, please describe what has been going on in the past few weeks and how you have been feeling.Big Events153Y2Please describe any big events from the past few weeks, positive or negative that have impacted your overall mood or well-being.Yesterday164Y2Please think about yesterday, from the morning until the end of the day. Think about where you were, what you were doing, who you were with, and how you felt. Would you like to have more days just like yesterday? Please explain why or why not.Feelings Merged642Y3In the last 2 weeks have you felt down, nervous, depressed, anxious, hopeless or on edge? If so, please explain in detail how it has bothered you or impacted your life?Stressful468Y3Regarding any memorable stressful experience(s), have you felt fear, horror, anger, guilt, or shame about the experience in the last 2 weeks? If so, please explain in detail how it has bothered you or impacted your life?Feelings Merged (Spanish)93Y4Translated into Spanish:¿En las últimas dos semanas, te has sentido triste, nervioso, deprimido, ansioso, sin esperanza o al límite? Si es así, por favor explica con detalle cómo eso ha alterado o impactado tu vida.The table lists each question prompt, the number of responses (N), whether it was used in the PCL-5 cohort, the cohort number, and the exact phrasing shown to participants. Spanish-translated variants are also included to reflect cross-language applicability.PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL-5 PTSD checklist. Question texts and cohort distribution for each assessment The table lists each question prompt, the number of responses (N), whether it was used in the PCL-5 cohort, the cohort number, and the exact phrasing shown to participants. Spanish-translated variants are also included to reflect cross-language applicability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL-5 PTSD checklist. Table 2 summarizes the model performance across each question. Within the first cohort, the PHQ-2 question emerged as the most effective for predicting depression, while the GAD-1 question was optimal for predicting anxiety. Both high-performing questions share notable characteristics: they explicitly prompt participants to explore and articulate their feelings and emotions in detail, and they include terms commonly associated with depression, anxiety, and trauma. This strategic wording likely helps guide participants’ responses, improving predictive accuracy. These insights informed the development of the “Feelings Merged” question, which not only performed well on anxiety and depression metrics but also yielded the highest scores on trauma assessments.Table 2Model performance per question typeQuestionPHQ-9 (Mean, 95% CI)GAD-7 (Mean, 95% CI)PCL (Mean, 95% CI)Feelings merged0.900 (0.886–0.915)0.907 (0.892–0.921)0.897 (0.877–0.914)PHQ 20.891 (0.881–0.902)0.838 (0.821–0.854)-GAD 10.834 (0.819–0.848)0.878 (0.865–0.890)-GAD 20.824 (0.809–0.839)0.877 (0.864–0.889)-Feelings merged (Spanish)0.859 (0.802–0.910)*0.849 (0.812–0.883)0.855 (0.812–0.895)PHQ 10.883 (0.872–0.894)0.813 (0.797–0.831)-Feelings General0.868 (0.854–0.880)0.819 (0.804–0.836)-Stressful0.829 (0.802–0.854)0.820 (0.784–0.851)0.877 (0.854–0.897)Feelings 20.839 (0.800–0.874)0.815 (0.778–0.850)0.810 (0.774–0.840)Feelings 10.873 (0.840–0.901)0.791 (0.744–0.833)*0.780 (0.742–0.813)Feelings 30.842 (0.799–0.881)0.782 (0.724–0.835)*0.792 (0.758–0.824)Big events0.724 (0.672–0.778)*0.701 (0.631–0.768)*0.700 (0.622–0.771)*Yesterday0.709 (0.648–0.763)*0.591 (0.523–0.653)*0.647 (0.608–0.683)*Area under the curve (AUC) scores with 95% confidence intervals for each question across PHQ-9, GAD-7, and PCL-5 assessments. The “Feelings Merged” question consistently achieved the strongest performance. *Indicates a high coefficient of variation ( ≥ 0.025) across bootstrap samples.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist. Table 2Model performance per question typeQuestionPHQ-9 (Mean, 95% CI)GAD-7 (Mean, 95% CI)PCL (Mean, 95% CI)Feelings merged0.900 (0.886–0.915)0.907 (0.892–0.921)0.897 (0.877–0.914)PHQ 20.891 (0.881–0.902)0.838 (0.821–0.854)-GAD 10.834 (0.819–0.848)0.878 (0.865–0.890)-GAD 20.824 (0.809–0.839)0.877 (0.864–0.889)-Feelings merged (Spanish)0.859 (0.802–0.910)*0.849 (0.812–0.883)0.855 (0.812–0.895)PHQ 10.883 (0.872–0.894)0.813 (0.797–0.831)-Feelings General0.868 (0.854–0.880)0.819 (0.804–0.836)-Stressful0.829 (0.802–0.854)0.820 (0.784–0.851)0.877 (0.854–0.897)Feelings 20.839 (0.800–0.874)0.815 (0.778–0.850)0.810 (0.774–0.840)Feelings 10.873 (0.840–0.901)0.791 (0.744–0.833)*0.780 (0.742–0.813)Feelings 30.842 (0.799–0.881)0.782 (0.724–0.835)*0.792 (0.758–0.824)Big events0.724 (0.672–0.778)*0.701 (0.631–0.768)*0.700 (0.622–0.771)*Yesterday0.709 (0.648–0.763)*0.591 (0.523–0.653)*0.647 (0.608–0.683)*Area under the curve (AUC) scores with 95% confidence intervals for each question across PHQ-9, GAD-7, and PCL-5 assessments. The “Feelings Merged” question consistently achieved the strongest performance. *Indicates a high coefficient of variation ( ≥ 0.025) across bootstrap samples.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist. Model performance per question type Area under the curve (AUC) scores with 95% confidence intervals for each question across PHQ-9, GAD-7, and PCL-5 assessments. The “Feelings Merged” question consistently achieved the strongest performance. *Indicates a high coefficient of variation ( ≥ 0.025) across bootstrap samples. PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist. The results underscore the importance of asking carefully crafted questions to obtain meaningful responses. Performance varied widely among the questions, with AUC scores ranging from 0.592 for the low-performing “Yesterday” question to 0.907 for the high-performing “Feelings Merged” question. Notably, questions that emphasized recent events rather than emotions tended to score lower. Additionally, results suggest that specific wording can significantly impact model performance. For instance, three questions from the second cohort (Feelings 1–3) focus on emotions but lack direct mention of specific emotional states. As a result, they performed substantially worse than “Feelings Merged”, highlighting the importance of precise, example-driven prompts in eliciting informative responses for model training. Selected question performance The “Feelings Merged” question emerged as the optimal single question for mental health assessment, consistently achieving an AUC range of 0.90–0.91 across all assessment types (PHQ-9, GAD-7, PCL-5). Unlike other questions that excelled in predicting only one or two conditions, this question demonstrated reliable performance across depression, anxiety, and trauma measures. This broader predictive capability is particularly valuable for comprehensive mental health screening. While this broader predictive capability is valuable given the common comorbidities in mental health, it is important to note that the model generates distinct predictions for each condition. While the predictions are correlated (reflecting common comorbidity), they reflect distinct symptom dimensions as captured by the PHQ-9, GAD-7, and PCL-5. High scores across measures may reflect genuine comorbidity or symptom overlap, requiring clinical judgment for interpretation. While the “Feelings Merged” question predicts PCL-5 scores effectively, it does not differentiate PTSD from anxiety disorders. Rather, it identifies individuals who may benefit from comprehensive assessment where proper differential diagnosis can be conducted. Figure 1 presents the ROC curves for each assessment type, illustrating the strong predictive power achieved with this single question approach.Fig. 1Model performance on “Feelings Merged” question.ROC curves for models predicting PHQ (a), GAD (b), and PCL (c) scores from responses to the “Feelings Merged” question. Shaded regions show 95% confidence intervals across bootstrap samples. Area under the curve (AUC) scores were 0.900 for PHQ, 0.907 for GAD, and 0.897 for PCL. Black dashed lines represent the performance of a random chance model. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Model performance on “Feelings Merged” question. ROC curves for models predicting PHQ (a), GAD (b), and PCL (c) scores from responses to the “Feelings Merged” question. Shaded regions show 95% confidence intervals across bootstrap samples. Area under the curve (AUC) scores were 0.900 for PHQ, 0.907 for GAD, and 0.897 for PCL. Black dashed lines represent the performance of a random chance model. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Time efficiency analysis To address the practical feasibility of single-question assessment, we conducted a comprehensive time comparison analysis using response duration data from 438 participants who completed both the single-question approach and traditional surveys (PHQ-9, GAD-7, PCL-5). The single-question approach required a mean of 78.4 s (median 57.3 s), including all overhead for reading instructions, recording the response, and submitting. In comparison, completing all three traditional surveys required a mean of 221.7 s (median 192.7 s). This represents a 64.6% time reduction, saving a mean of 143.3 s per assessment (paired t-test: t = 15.785, p < 0.001). When comparing individual survey durations, the single question was slower than individual surveys for depression-only (PHQ-9: 55.1 s) or anxiety-only (GAD-7: 35.5 s) screening. However, the single question was 52.6 seconds faster than trauma-only screening (PCL-5: 131.1 s) and demonstrated a 13.5% time reduction compared to combined depression and anxiety screening (PHQ + GAD: 90.6 s versus single: 78.4 s). These findings demonstrate that the primary efficiency advantage emerges when comprehensive screening for multiple conditions is needed. For 20 patients per day screened for all three conditions, this approach would save approximately 48 min of total assessment time compared to traditional survey administration. These time savings become more pronounced when screening requirements expand beyond depression and anxiety. For example, adding trauma screening with traditional methods would require an additional 131 seconds, while our single-question approach captures all three domains simultaneously. This efficiency advantage suggests the approach could extend to even broader assessment batteries, potentially screening for multiple conditions (e.g., movement disorders, speech pathologies, additional mental health domains) from a single brief interaction. User acceptance Among 982 participants surveyed in the earlier 5-question study, 90.7% expressed positive willingness to use video-based screening (somewhat willing or better), with 50.1% indicating they were “very willing”. Only 1.4% reported being “not at all willing”. The current single-question approach, with 64.6% reduced time and 80% fewer questions, likely maintains or improves upon this acceptance level. Clinical threshold analysis To support practical application, we evaluated model performance across different threshold levels. These thresholds align with established clinical categories (mild: 5–9, moderate: 10–14, severe: ≥15 for PHQ-9 and GAD-7; mild: 15-29, moderate: 30–49, severe: ≥50 for PCL-5) and serve multiple practical purposes. While the model outputs continuous scores, providers often need discrete categories to guide care decisions. General care providers may favor lower thresholds for early intervention, while specialized care providers managing high-acuity cases may focus on higher thresholds. Results for three binary threshold levels across assessment types are detailed in Table 3, showing strong performance across all thresholds, with scores improving slightly at higher levels. This adaptability demonstrates the model’s utility for alerting providers across varied care contexts. The ROC curves (averaged across all three thresholds) for “Feelings Merged” are shown in Fig. 1, and the ROC curves when broken down by each threshold are shown in Fig. 2.Fig. 2AUC performance across mild, moderate, and severe thresholds.ROC curves show model performance across mild, moderate, and severe thresholds for PHQ (a), GAD (b), and PCL (c). Performance shown for mild (dark purple lines), moderate (red lines), and severe (orange lines) thresholds. Gray dashed lines indicate the performance of a random classifier. Sensitivity and specificity values are annotated for each tier. PHQ Patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve.Table 3Model performance at selected thresholdsThresholdPearson RAUCSensitivitySpecificityPHQ-9 (Mean, 95% CI)0.780 (0.751–0.805)Mild (≥5)0.893 (0.870–0.914)0.885 (0.856–0.916)0.708 (0.661–0.759)Moderate (≥10)0.891 (0.869–0.910)0.820 (0.775–0.865)0.767 (0.733–0.801)Severe (≥15)0.918 (0.896–0.938)0.755 (0.682–0.827)0.889 (0.865–0.912)Averaged0.900 (0.879–0.921)0.820 (0.771–0.869)0.788 (0.753–0.824)GAD-7 (Mean, 95% CI)0.782 (0.752–0.809)Mild (≥5)0.889 (0.866–0.909)0.865 (0.832–0.900)0.726 (0.683–0.772)Moderate (≥10)0.914 (0.892–0.934)0.820 (0.767–0.873)0.817 (0.784–0.850)Severe (≥15)0.918 (0.894–0.938)0.694 (0.602–0.786)0.916 (0.895–0.938)Averaged0.907 (0.884–0.927)0.793 (0.734–0.853)0.820 (0.787–0.853)PCL-5 (Mean, 95% CI)0.756 (0.721 - 0.786)Mild (≥15)0.877 (0.848–0.904)0.833 (0.790–0.873)0.780 (0.731–0.827)Moderate (≥30)0.887 (0.859–0.913)0.794 (0.729–0.860)0.856 (0.824 - 0.888)Severe (≥50)0.926 (0.900–0.952)0.617 (0.500–0.735)0.937 (0.918–0.956)Averaged0.897 (0.869–0.923)0.748 (0.673–0.823)0.858 (0.824–0.890)Performance metrics at each threshold reflect the model’s accuracy across mild, moderate, and severe severity levels, aligning with standard clinical cutoffs for PHQ-9, GAD-7, and PCL assessments. These thresholds enable the model to provide actionable insights based on established scoring categories. Average refers to the average performance of all three thresholds.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve). AUC performance across mild, moderate, and severe thresholds. ROC curves show model performance across mild, moderate, and severe thresholds for PHQ (a), GAD (b), and PCL (c). Performance shown for mild (dark purple lines), moderate (red lines), and severe (orange lines) thresholds. Gray dashed lines indicate the performance of a random classifier. Sensitivity and specificity values are annotated for each tier. PHQ Patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Table 3Model performance at selected thresholdsThresholdPearson RAUCSensitivitySpecificityPHQ-9 (Mean, 95% CI)0.780 (0.751–0.805)Mild (≥5)0.893 (0.870–0.914)0.885 (0.856–0.916)0.708 (0.661–0.759)Moderate (≥10)0.891 (0.869–0.910)0.820 (0.775–0.865)0.767 (0.733–0.801)Severe (≥15)0.918 (0.896–0.938)0.755 (0.682–0.827)0.889 (0.865–0.912)Averaged0.900 (0.879–0.921)0.820 (0.771–0.869)0.788 (0.753–0.824)GAD-7 (Mean, 95% CI)0.782 (0.752–0.809)Mild (≥5)0.889 (0.866–0.909)0.865 (0.832–0.900)0.726 (0.683–0.772)Moderate (≥10)0.914 (0.892–0.934)0.820 (0.767–0.873)0.817 (0.784–0.850)Severe (≥15)0.918 (0.894–0.938)0.694 (0.602–0.786)0.916 (0.895–0.938)Averaged0.907 (0.884–0.927)0.793 (0.734–0.853)0.820 (0.787–0.853)PCL-5 (Mean, 95% CI)0.756 (0.721 - 0.786)Mild (≥15)0.877 (0.848–0.904)0.833 (0.790–0.873)0.780 (0.731–0.827)Moderate (≥30)0.887 (0.859–0.913)0.794 (0.729–0.860)0.856 (0.824 - 0.888)Severe (≥50)0.926 (0.900–0.952)0.617 (0.500–0.735)0.937 (0.918–0.956)Averaged0.897 (0.869–0.923)0.748 (0.673–0.823)0.858 (0.824–0.890)Performance metrics at each threshold reflect the model’s accuracy across mild, moderate, and severe severity levels, aligning with standard clinical cutoffs for PHQ-9, GAD-7, and PCL assessments. These thresholds enable the model to provide actionable insights based on established scoring categories. Average refers to the average performance of all three thresholds.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve). Model performance at selected thresholds Performance metrics at each threshold reflect the model’s accuracy across mild, moderate, and severe severity levels, aligning with standard clinical cutoffs for PHQ-9, GAD-7, and PCL assessments. These thresholds enable the model to provide actionable insights based on established scoring categories. Average refers to the average performance of all three thresholds. PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve). Lastly, we assessed the performance of the “Feelings Merged” question in Spanish (see Table 2). Responses were translated into English for scoring, resulting in a similar AUC of 0.85–0.86. The strong performance of the Spanish language variant (AUC of 0.85–0.86) demonstrates the model’s ability to generalize across languages and contexts, suggesting broad applicability in diverse clinical settings. Prediction distribution The model predicts continuous scores for three mental health assessments: the Patient Health Questionnaire (PHQ-9), the generalized anxiety disorder scale (GAD-7), and the PTSD Checklist (PCL-5). In clinical applications, such as triggering alerts for clinician intervention, these continuous predictions require well-defined thresholds. Using mean squared error (MSE) loss can often lead to conservative predictions that cluster toward the mean, potentially resulting in mismatched distributions and uninformative thresholds20,21. Figure 3 demonstrates that our model’s predicted score distributions closely mirror the original assessment score distributions for PHQ-9, GAD-7, and PCL-5. This alignment is crucial because it ensures that the model’s predictions maintain the nuanced characteristics of the original assessments. The close correspondence is further validated by the model’s strong sensitivity and specificity scores in Table 3.Fig. 3Distribution of predicted vs. target scores by assessment type.Histogram distributions of predicted scores (blue) and observed target scores (orange) for PHQ (a), GAD (b), and PCL (c) assessments. Visual alignment between predicted and observed score distributions supports the use of standard clinical thresholds for downstream applications. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. Distribution of predicted vs. target scores by assessment type. Histogram distributions of predicted scores (blue) and observed target scores (orange) for PHQ (a), GAD (b), and PCL (c) assessments. Visual alignment between predicted and observed score distributions supports the use of standard clinical thresholds for downstream applications. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. Demographics performance The expression of mental illness symptoms is influenced by cultural background, age, gender, and ethnicity22, and previous research highlights that clinicians may carry biases affecting whether individuals are assessed, diagnosed, or treated for mental health disorders23. This issue extends to machine learning and artificial intelligence models, which may inadvertently reflect such biases24. Therefore, it is critical to assess the reliability and validity of predictive models across different demographic groups. Our voice-based model performed similarly across gender (AUC 0.88–0.92), ethnicity (AUC 0.82–0.90), and age groups (AUC 0.88–0.92) as shown in Table 4; Figs. 4–7. These results suggest the model’s predictive power is robust across different adult demographic categories. Additional demographics performance is found in the supplemental materials.Fig. 4Receiver operating characteristic (ROC) AUC by sex.Receiver operating characteristic (ROC) curves stratified by sex for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are shown separately for male participants (orange lines) and female participants (blue lines). Shaded regions represent 95% confidence intervals across bootstrap samples. Performance was comparable across sexes, indicating consistent model behavior and generalizability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve.Fig. 5Receiver operating characteristic (ROC) AUC by race/ethnicity.Receiver operating characteristic (ROC) curves stratified by self-reported race and ethnicity for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are shown for each subgroup, with shaded regions indicating 95% confidence intervals across bootstrap samples. While AUC values were generally high across groups, smaller sample sizes in some subgroups (e.g., Asian, Black, Latino) may contribute to increased variability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve.Fig. 6Receiver operating characteristic (ROC) AUC by age group.Receiver operating characteristic (ROC) curves stratified by age group for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are provided for each age bracket. Shaded regions represent 95% confidence intervals across bootstrap samples. Model performance remained strong across all age groups, with some variation potentially due to smaller subgroup sizes. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve.Fig. 7ROC AUC scores by response quality classification for PHQ-9, GAD-7, and PCL-5 assessments.Receiver operating characteristic (ROC) curves showing the model’s ability to classify response quality as invalid responses (purple line), low-quality responses (orange line), or high-quality responses (green line) across 3289 video responses. Shaded regions represent 95% confidence intervals across bootstrap samples. These results reflect the model’s performance in predicting response quality labels, enabling automated filtering or flagging of low- and non-usable inputs prior to mental health scoring. ROC receiver operating characteristic, AUC area under the curve.Table 4Demographic-specific model performance for PHQ-9, GAD-7, and PCL-5 assessmentsVariablePHQ-9 (Mean, 95% CI)GAD-7 (Mean, 95% CI)PCL (Mean, 95% CI)Age group18–290.908 (0.875–0.938)0.865 (0.818–0.906)a0.844 (0.808–0.877)30–390.889 (0.864–0.912)0.910 (0.886–0.931)0.915 (0.894–0.934)40–490.891 (0.860–0.919)0.909 (0.879–0.935)0.900 (0.865–0.932)50–590.935 (0.908–0.961)0.950 (0.929–0.968)0.903 (0.890–0.917)60–690.907 (0.872–0.940)0.898 (0.834–0.948)a0.924 (0.901–0.944)Ethnicity/RaceAsian0.902 (0.861–0.941)0.919 (0.880–0.948)0.871 (0.852–0.885)Black0.846 (0.799–0.891)a0.856 (0.808–0.892)-Latino0.833 (0.762–0.892)a0.888 (0.845–0.926)-Multiracial0.915 (0.882–0.946)0.835 (0.754–0.903)a0.909 (0.894–0.922)White0.909 (0.894–0.923)0.917 (0.902–0.931)0.911 (0.895–0.926)SexFemale0.884 (0.859–0.906)0.903 (0.881–0.924)0.890 (0.865–0.913)Male0.919 (0.902–0.934)0.911 (0.890–0.930)0.901 (0.874–0.926)Performance scores are reported as AUC values across major demographic categories, highlighting the model’s reliability and validity across age, gender, and racial/ethnic groups. Note that for starred groups, sample sizes for the PCL-5 assessment were insufficient to yield meaningful results, and only PHQ-9 and GAD-7 scores are presented.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve.aHigh coefficient of variation between bootstrapped scores (≥0.025). Receiver operating characteristic (ROC) AUC by sex. Receiver operating characteristic (ROC) curves stratified by sex for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are shown separately for male participants (orange lines) and female participants (blue lines). Shaded regions represent 95% confidence intervals across bootstrap samples. Performance was comparable across sexes, indicating consistent model behavior and generalizability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Receiver operating characteristic (ROC) AUC by race/ethnicity. Receiver operating characteristic (ROC) curves stratified by self-reported race and ethnicity for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are shown for each subgroup, with shaded regions indicating 95% confidence intervals across bootstrap samples. While AUC values were generally high across groups, smaller sample sizes in some subgroups (e.g., Asian, Black, Latino) may contribute to increased variability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Receiver operating characteristic (ROC) AUC by age group. Receiver operating characteristic (ROC) curves stratified by age group for PHQ (a), GAD (b), and PCL (c) models. Area under the curve (AUC) values are provided for each age bracket. Shaded regions represent 95% confidence intervals across bootstrap samples. Model performance remained strong across all age groups, with some variation potentially due to smaller subgroup sizes. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. ROC AUC scores by response quality classification for PHQ-9, GAD-7, and PCL-5 assessments. Receiver operating characteristic (ROC) curves showing the model’s ability to classify response quality as invalid responses (purple line), low-quality responses (orange line), or high-quality responses (green line) across 3289 video responses. Shaded regions represent 95% confidence intervals across bootstrap samples. These results reflect the model’s performance in predicting response quality labels, enabling automated filtering or flagging of low- and non-usable inputs prior to mental health scoring. ROC receiver operating characteristic, AUC area under the curve. Table 4Demographic-specific model performance for PHQ-9, GAD-7, and PCL-5 assessmentsVariablePHQ-9 (Mean, 95% CI)GAD-7 (Mean, 95% CI)PCL (Mean, 95% CI)Age group18–290.908 (0.875–0.938)0.865 (0.818–0.906)a0.844 (0.808–0.877)30–390.889 (0.864–0.912)0.910 (0.886–0.931)0.915 (0.894–0.934)40–490.891 (0.860–0.919)0.909 (0.879–0.935)0.900 (0.865–0.932)50–590.935 (0.908–0.961)0.950 (0.929–0.968)0.903 (0.890–0.917)60–690.907 (0.872–0.940)0.898 (0.834–0.948)a0.924 (0.901–0.944)Ethnicity/RaceAsian0.902 (0.861–0.941)0.919 (0.880–0.948)0.871 (0.852–0.885)Black0.846 (0.799–0.891)a0.856 (0.808–0.892)-Latino0.833 (0.762–0.892)a0.888 (0.845–0.926)-Multiracial0.915 (0.882–0.946)0.835 (0.754–0.903)a0.909 (0.894–0.922)White0.909 (0.894–0.923)0.917 (0.902–0.931)0.911 (0.895–0.926)SexFemale0.884 (0.859–0.906)0.903 (0.881–0.924)0.890 (0.865–0.913)Male0.919 (0.902–0.934)0.911 (0.890–0.930)0.901 (0.874–0.926)Performance scores are reported as AUC values across major demographic categories, highlighting the model’s reliability and validity across age, gender, and racial/ethnic groups. Note that for starred groups, sample sizes for the PCL-5 assessment were insufficient to yield meaningful results, and only PHQ-9 and GAD-7 scores are presented.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve.aHigh coefficient of variation between bootstrapped scores (≥0.025). Demographic-specific model performance for PHQ-9, GAD-7, and PCL-5 assessments Performance scores are reported as AUC values across major demographic categories, highlighting the model’s reliability and validity across age, gender, and racial/ethnic groups. Note that for starred groups, sample sizes for the PCL-5 assessment were insufficient to yield meaningful results, and only PHQ-9 and GAD-7 scores are presented. PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD checklist, AUC area under the receiver operating characteristic curve. aHigh coefficient of variation between bootstrapped scores (≥0.025). Quality score performance In addition to predicting mental health assessment scores, the model is trained to evaluate the quality of participant responses, referred to as the “quality score”. This quality score serves two main purposes. First, it enables the model to withhold scoring when video or audio quality is poor or when responses lack sufficient detail or participant effort, which prevents misleading confidence in results, an essential safeguard in medical applications25–27. This addresses a key limitation of traditional surveys, where minimal-effort responses (e.g., selecting “0” for all items without reading) cannot be distinguished from genuine low symptom burden. Second, the quality score highlights when follow-up questions may be beneficial for gaining a deeper understanding of a participant’s mental health status, and provides a pathway for incorporating this into a conversational chat question in future research. In the current study, the model demonstrated strong performance in assessing response quality. Specifically, the model achieved a 0.99 AUC for detecting invalid samples, 0.95 AUC for detecting low-quality samples, and 0.97 AUC for detecting high-quality samples. In this method, each quality category is compared against all other categories combined (one-vs-rest), allowing for a comprehensive evaluation of the model’s ability to distinguish between different quality levels. These results demonstrate the model’s ability to distinguish response quality levels, enabling identification of cases requiring follow-up questioning before generating scores. This capability could support future conversational assessment systems that adaptively prompt for additional detail when initial responses are insufficient. The ROC curves demonstrating the performance for predicting each quality score are shown in Fig. 7. To demonstrate the relationship between response quality and predictive accuracy, we compared the performance of samples with low quality scores to the performance of samples with high quality scores. While the performance of the full data set reported above, including both low and high quality scores, performed consistently well with an AUC between 0.90 and 0.92, there is a clear difference in the performance of low versus high quality samples (Fig. 8). Samples labeled high quality had a much higher AUC of 0.91–0.94, while the low quality samples ranged from 0.77 to 0.87. ROC curves comparing the performance of these different quality groups are found in Fig. 8. The performance gap between quality levels primarily reflects the model’s reliance on semantic content to distinguish symptom severity, as brief responses lack the linguistic detail necessary for accurate assessment. While the multimodal architecture incorporates prosodic features from audio, the reduced duration of low-quality responses (mean 22.4 s vs longer high-quality responses) also limits acoustic feature extraction. This analysis demonstrates that quality assessment can identify when responses are likely to yield less accurate predictions.Fig. 8ROC AUC scores for low quality vs high quality responses.Receiver operating characteristic (ROC) curves comparing model performance on high-quality responses (blue lines) and low-quality responses (orange lines), as well as the full dataset (green lines), for PHQ (a), GAD (b), and PCL (c). Area under the curve (AUC) values are shown for each group, with shaded regions representing 95% confidence intervals across bootstrap samples. Performance was consistently higher on high-quality responses, underscoring the importance of input quality in model accuracy and supporting the utility of quality-based scoring safeguards. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. ROC AUC scores for low quality vs high quality responses. Receiver operating characteristic (ROC) curves comparing model performance on high-quality responses (blue lines) and low-quality responses (orange lines), as well as the full dataset (green lines), for PHQ (a), GAD (b), and PCL (c). Area under the curve (AUC) values are shown for each group, with shaded regions representing 95% confidence intervals across bootstrap samples. Performance was consistently higher on high-quality responses, underscoring the importance of input quality in model accuracy and supporting the utility of quality-based scoring safeguards. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist, ROC receiver operating characteristic, AUC area under the curve. Discussion This study demonstrates that a multimodal single-question assessment model can predict mental health screening scores (PHQ-9, GAD-7, PCL-5) with strong performance (AUC 0.90–0.91) and substantial time efficiency compared to traditional sequential questionnaire administration. By consolidating three separate assessments into a single 78-s interaction, this approach achieves a 64.6% time reduction compared to administering three separate questionnaires (78.4 s versus 221.7 s, n = 438, p < 0.001). This efficiency advantage is particularly pronounced for comprehensive multi-condition screening, where our approach enables simultaneous assessment of depression, anxiety, and trauma symptoms that would otherwise require sequential administration of condition-specific questionnaires. These time savings address a practical barrier to comprehensive mental health screening in settings where nearly half of residents lack timely access6. This approach builds upon previous work in remote assessment technologies9–11, while reducing the patient burden. As with any screening tool, positive results indicating significant symptom burden warrant follow-up clinical evaluation for differential diagnosis, particularly given the frequent co-occurrence and symptom overlap among depression, anxiety, and PTSD. Our multimodal approach analyzes open-ended narrative responses, capturing both semantic content and prosodic features that may provide clinical context beyond numerical scores. Assessing trauma symptoms (PCL-5) alongside depression and anxiety enables more comprehensive screening, as trauma history often complicates mental health presentation and treatment planning20,21. Our multimodal approach, combining semantic analysis12 and voice prosody15, ensures that clinicians capture a nuanced view of mental health beyond binary or limited-response answers, enhancing the quality and depth of patient assessment with minimal respondent burden. In our previous study using a five-question format, we identified cases where video-based assessment detected elevated depression or anxiety signals that participants had not recognized in their self-reported questionnaire responses. One participant noted while recording: “It really does impact my life. Probably more than I realized, now sitting here thinking about it”. This suggests that the reflective nature of open-ended video responses may help some individuals recognize symptoms they would not endorse on structured questionnaires. However, whether this phenomenon translates to improved clinical outcomes requires validation against gold-standard diagnostic interviews rather than self-report questionnaires alone. The brief format suggests potential applications in longitudinal symptom monitoring between clinical appointments, particularly for patients requiring frequent assessment due to symptom fluctuations (e.g., bipolar disorder) or complex comorbidities. In the U.S., an estimated one-third of patients experience both anxiety and depression28, often leading to higher symptom severity, increased suicide risk, and lower quality of life. These patients typically require customized care and would benefit from efficient tools that can monitor comorbid symptoms with minimal burden. However, test-retest reliability, sensitivity to clinical change, and patient adherence to repeated video assessments would need to be established before implementing longitudinal monitoring protocols. Our participant sample predominantly consisted of non-Hispanic, White females recruited through Prolific, which may limit the model’s generalizability across different demographics. Prolific participants tend to be more tech-savvy and comfortable with video-based interactions, potentially excluding populations with limited technological access or privacy concerns about video recording. Additionally, while we used the GAD-7, PHQ-9, and PCL-5 as benchmark measures due to their validation and widespread use, these metrics may not fully capture the nuanced variations in gender- or race-specific symptomatology, which could impact the model’s sensitivity to diverse mental health presentations. Several methodological considerations merit discussion. First, all three target instruments (PHQ-9, GAD-7, and PCL-5) assess current symptom severity, not diagnostic status or underlying causes. Our model is designed to complement, not replace, traditional screening practices by enabling scalable and repeatable symptom monitoring in digital or asynchronous settings. Second, participants answered multiple questions per session, potentially creating spill-over effects despite randomization. Third, our “Feelings Merged” question includes symptom-related terms that may prime responses, though this also helps participants understand what to discuss. We also acknowledge that our iterative question development process and inclusion of participants across multiple cohorts may introduce biases. However, allowing repeat participants supports future research on within-person change and test-retest reliability. Additionally, while we did not assess discriminant validity between conditions, high scores across assessments may reflect comorbidity or symptom overlap. Beyond mental health screening, the video-based assessment framework demonstrated here could potentially extend to other health domains. Because video captures multiple signal types simultaneously (speech prosody, content, movement), preliminary work suggests this approach could screen for conditions like movement disorders or speech pathologies alongside mental health assessment. This would further strengthen the time efficiency advantage, as a single interaction could replace multiple separate screening procedures. However, such multi-domain screening would require extensive validation to ensure reliable performance across diverse clinical targets. Methods Participant sample Participants were recruited through Prolific, an online platform for research participants. To qualify, participants had to be at least 18 years old, able to provide video responses in English (or Spanish for Cohort 4), and provide informed consent. From 2915 initial participants, we retained 2420 for analysis after data processing. We excluded 495 participants for: (1) technical failures preventing data collection (e.g., black video, silent audio), (2) failure to complete the study protocol (e.g., not answering the question, incomplete assessments), or (3) responses where non-participant audio dominated (e.g., television audio transcribed instead of participant response). To train the quality detection model described below, a subset of 3289 responses were manually labeled for quality assessment. This labeled subset included responses across all quality levels to enable the model to identify when follow-up questioning may be needed rather than producing scores from insufficient responses. Recruitment occurred across four cohorts from 2022 to 2024, with each cohort responding to iteratively improved questions in Table 1. Each participant was asked one or more questions, with responses recorded for later use as input to the machine learning models. Questions were randomized within cohorts. Cross-validation kept all participant responses in the same fold. All four cohorts responded on video. After completing their responses, participants were administered two or three standardized self-report assessments, which served as prediction targets for the model. These assessments, widely recognized as gold standards, measure depression, anxiety, and trauma. Additionally, participants in the earlier 5-question study (Cohort 1, n = 982) were asked about their willingness to use video-based screening in clinical settings if recommended by their healthcare provider. While this data was collected during the 5-question study, it provides insight into general acceptance of video-based assessment approaches. The assessments used were the Patient Health Questionnaire-9 (PHQ-9), a nine-item scale that assesses depression severity29,30; the generalized anxiety disorder-7 (GAD-7), a seven-item scale used to measure anxiety levels31,32; and the PTSD checklist for DSM-5 (PCL-5), a 20-item scale that evaluates symptoms of post-traumatic stress disorder (PTSD)33,34. It is important to note that the PCL-5 assesses current PTSD symptom severity rather than identifying whether a traumatic event occurred. The model was trained to predict the raw, numerical assessment scores. These raw scores are categorized into four severity levels: none, mild, moderate, and severe using established clinical thresholds from the original validation studies29–34. These categories are used to calculate binary validation metrics for each assessment type. The specific boundaries for each severity category ranges are outlined in Table 5.Table 5Severity ranges for PHQ-9, GAD-7, and PCL-5 assessmentsAssessmentNoneMildMediumSeverePHQ-90-45-910-1415+GAD-70-45-910-1415+PCL-50-1415-2930-4950+Cutoff values for clinical severity categories as defined by Kroenke et al.28 for PHQ-9, Löwe et al.30 for GAD-7, and Blevins et al.31 for PCL-5.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD Checklist). Table 5Severity ranges for PHQ-9, GAD-7, and PCL-5 assessmentsAssessmentNoneMildMediumSeverePHQ-90-45-910-1415+GAD-70-45-910-1415+PCL-50-1415-2930-4950+Cutoff values for clinical severity categories as defined by Kroenke et al.28 for PHQ-9, Löwe et al.30 for GAD-7, and Blevins et al.31 for PCL-5.PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD Checklist). Severity ranges for PHQ-9, GAD-7, and PCL-5 assessments Cutoff values for clinical severity categories as defined by Kroenke et al.28 for PHQ-9, Löwe et al.30 for GAD-7, and Blevins et al.31 for PCL-5. PHQ-9 patient health questionnaire, GAD-7 generalized anxiety disorder scale, PCL-5 PTSD Checklist). This paper primarily analyzes individual responses, though in some cases, such as Fig. 9, it references sessions and participants. A response refers to a video answer to a single question. A session represents a continuous set of questions that a single participant completes in one sitting. A participant is an individual who may appear across multiple cohorts, which means a participant can have multiple sessions, and each session can contain multiple responses.Fig. 9Sample population and exclusion breakdown by assessment type.Flow diagram summarizing response inclusion and exclusion across PHQ-9, GAD-7, and PCL-5 assessments. Responses were excluded due to incomplete sessions, failure to pass quality review, or absence of recorded speech. Final sample sizes differ by assessment type due to study design and cohort availability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. Sample population and exclusion breakdown by assessment type. Flow diagram summarizing response inclusion and exclusion across PHQ-9, GAD-7, and PCL-5 assessments. Responses were excluded due to incomplete sessions, failure to pass quality review, or absence of recorded speech. Final sample sizes differ by assessment type due to study design and cohort availability. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. To address potential order effects and spill-over between questions, we implemented several methodological controls. First, questions within each cohort were presented in randomized order to minimize systematic bias. Second, our cross-validation strategy ensured that all responses from a single participant were kept in the same fold, preventing the model from learning participant-specific patterns across questions. While we cannot entirely eliminate the possibility of spill-over effects, our analysis of question performance in Table 2 shows consistent results across different question types and cohorts, suggesting that any such effects did not substantially impact our findings. We acknowledge this as a limitation and recommend future studies consider single-question sessions to fully isolate response characteristics. Quality scores categories Quality assessment was performed on a labeled training subset of 3289 video responses to train a quality detection model. This subset was purposefully sampled to include higher proportions of potentially problematic responses, such as shorter responses and sessions with lower preliminary model scores, enabling effective training of the quality classifier. Responses were categorized into three quality tiers by trained reviewers who assessed content detail and coherence to determine whether responses represented genuine assessment attempts and contained sufficient information to confidently determine depression, anxiety, and trauma symptom severity. The percentages reported below (invalid: 12.5%, low-quality: 18.2%, high-quality: 69.2%) reflect the distribution within this training subset only, not the overall dataset. Invalid responses (N = 412, 12.5% of labeled subset) included cases where participants did not provide substantive answers, such as speaking to someone off-screen, providing completely off-topic content, or having audio quality too poor for transcription. These responses were excluded from mental health prediction analyses. The proportion of invalid responses likely reflects characteristics of compensated online research settings, where some participants provide minimal effort. Low-quality responses (N = 600, 18.2% of labeled subset) consisted of legitimate but brief responses lacking sufficient detail for confident severity assessment. These responses demonstrated honest effort (mean 38.0 words, 22.4 s duration) but provided limited elaboration. Critically, these responses were retained in all analyses as they represent authentic brief disclosures. The quality detection model identifies such cases to flag when follow-up questioning may improve assessment accuracy. This addresses a key limitation of traditional surveys, where minimal-effort responses (e.g., selecting “0” for all items without reading) cannot be distinguished from genuine low symptom burden. High-quality responses (N = 2277, 69.2% of labeled subset) provided detailed descriptions enabling confident assessment. After excluding invalid responses, 642 valid responses remained for analysis of the selected question, comprising low-quality, high-quality, and unlabeled responses. All three categories were retained in analyses, as both low-quality responses (brief but legitimate) and unlabeled responses (assumed valid based on the quality model’s strong performance in detecting invalid cases) represent genuine assessment attempts. Sample demographics Participants were enlisted on a first-come, first-serve basis, with no targeted recruitment for specific demographics, aside from the requirement that participants be 18 years or older. Enrollment continued until a reasonable distribution was achieved across major demographic categories in order to measure “fair” outcomes across different demographic classes35,36. Table 6 presents the overall demographic characteristics of the dataset after applying exclusion criteria, while Table 7 presents the demographics for the “Feelings Merged” question.Table 6Demographic breakdown of participantsVariableParticipants (N = 2420)Age, years (M ± SE)35.2 ± 12.9Sex (%) Female1459 (60.3) Male889 (36.7) Not provided72 (3.0)Race/Ethnicity (%) White1645 (68.0) Multiracial205 (8.5) Latino204 (8.4) Black162 (6.7) Asian138 (5.7) Not provided44 (1.8) Middle Eastern7 (0.3) American Indian7 (0.3) Hawaiian2 (0.1) Other6 (0.2)Participant demographics are reported as total counts and percentages across participants, sessions, and responses. Age is presented as the mean ± standard error (M ± SE).Table 7Demographic breakdown for ‘Feelings Merged’VariableSessions (N = 642)Age, years (M ± SE)40.8 ± 13.9Sex (%)N = 642 Male336 (52.3) Female301 (46.9) Not provided5 (0.8)Race/Ethnicity (%)N = 642 White459 (71.5) Multiracial51 (7.9) Black45 (7.0) Latino40 (6.2) Asian36 (5.6) Other4 (0.6) American Indian4 (0.6) Not provided2 (0.3) Middle Eastern1 (0.2)Demographic breakdown includes sessions analyzed for each question and target score (PHQ-9, GAD-7, and PCL-5), with age reported as mean ± standard error (M ± SE) and sex and race/ethnicity presented as counts and percentages. Table 6Demographic breakdown of participantsVariableParticipants (N = 2420)Age, years (M ± SE)35.2 ± 12.9Sex (%) Female1459 (60.3) Male889 (36.7) Not provided72 (3.0)Race/Ethnicity (%) White1645 (68.0) Multiracial205 (8.5) Latino204 (8.4) Black162 (6.7) Asian138 (5.7) Not provided44 (1.8) Middle Eastern7 (0.3) American Indian7 (0.3) Hawaiian2 (0.1) Other6 (0.2)Participant demographics are reported as total counts and percentages across participants, sessions, and responses. Age is presented as the mean ± standard error (M ± SE). Demographic breakdown of participants Participant demographics are reported as total counts and percentages across participants, sessions, and responses. Age is presented as the mean ± standard error (M ± SE). Table 7Demographic breakdown for ‘Feelings Merged’VariableSessions (N = 642)Age, years (M ± SE)40.8 ± 13.9Sex (%)N = 642 Male336 (52.3) Female301 (46.9) Not provided5 (0.8)Race/Ethnicity (%)N = 642 White459 (71.5) Multiracial51 (7.9) Black45 (7.0) Latino40 (6.2) Asian36 (5.6) Other4 (0.6) American Indian4 (0.6) Not provided2 (0.3) Middle Eastern1 (0.2)Demographic breakdown includes sessions analyzed for each question and target score (PHQ-9, GAD-7, and PCL-5), with age reported as mean ± standard error (M ± SE) and sex and race/ethnicity presented as counts and percentages. Demographic breakdown for ‘Feelings Merged’ Demographic breakdown includes sessions analyzed for each question and target score (PHQ-9, GAD-7, and PCL-5), with age reported as mean ± standard error (M ± SE) and sex and race/ethnicity presented as counts and percentages. After selecting the final question text, we evaluated the model’s performance across various demographic groups, including age, sex, and race/ethnicity. For all demographic analyses, we focused specifically on the performance of the selected question, “Feelings Merged.” Question selection The process of finding the optimal question was iterative. We began by analyzing the individual performance of the five questions from Cohort #1 (“Feelings General”, “PHQ-1”, “PHQ-2”, “GAD-1”, and “GAD-2”). Next, we developed five new questions as potential single question candidates to replace the original five-question survey and presented them to a second cohort. This process led to the identification of the “Feelings Merged” question, which was then validated with a third cohort alongside a new question designed specifically to elicit responses that would perform well with the PCL-5 (the “Stressful” question). Subsequently, one additional cohort was added to assess the Spanish version of the question. All cohorts were recruited at various times between 2022 and 2024 and are listed in Table 8. The exact wording of questions and their associated name in the results below is included in Table 1.Table 8Cohort dates and questions administered across study phasesCohortDatesQuestions askedCohort #106/2021 to 08/2021, 02/2022 to 04/2022Feelings General, PHQ 1 & 2, GAD 1 & 2Cohort #206/2022 to 07/2022Feelings 1-3, Big Events, YesterdayCohort #310/2022 to 11/2022Feelings Merged, StressfulCohort #402/2023Feelings Merged (Spanish)Cohorts were recruited across various dates, with each cohort receiving a unique set of questions that were iteratively designed using learnings from earlier cohorts and sometimes focused on assessing different aspects of mental health. Question details correspond to specific timeframes for targeted data collection. Cohort #1 originated from data in previous research11. Table 8Cohort dates and questions administered across study phasesCohortDatesQuestions askedCohort #106/2021 to 08/2021, 02/2022 to 04/2022Feelings General, PHQ 1 & 2, GAD 1 & 2Cohort #206/2022 to 07/2022Feelings 1-3, Big Events, YesterdayCohort #310/2022 to 11/2022Feelings Merged, StressfulCohort #402/2023Feelings Merged (Spanish)Cohorts were recruited across various dates, with each cohort receiving a unique set of questions that were iteratively designed using learnings from earlier cohorts and sometimes focused on assessing different aspects of mental health. Question details correspond to specific timeframes for targeted data collection. Cohort #1 originated from data in previous research11. Cohort dates and questions administered across study phases Cohorts were recruited across various dates, with each cohort receiving a unique set of questions that were iteratively designed using learnings from earlier cohorts and sometimes focused on assessing different aspects of mental health. Question details correspond to specific timeframes for targeted data collection. Cohort #1 originated from data in previous research11. Not all participants were administered the PCL-5 assessment. Cohort #1 data originated from prior research that did not include trauma questions. The streamlined approach aimed to build participant comfort and promote conversational flow, ensuring that individuals were more likely to provide detailed, reflective answers. We acknowledge that this iterative design process introduces potential bias when comparing the “Feelings Merged” question to earlier formulations. The final question benefited from insights gained from analyzing prior cohorts, creating an advantage not available during initial question development. However, this iterative refinement mirrors real-world clinical tool development, where instruments are refined based on empirical performance. To partially address this concern, we validated the “Feelings Merged” question on an independent cohort (Cohort 3) not used in its development. Our primary question includes symptom-related terms such as “depressed” and “anxious” to clearly communicate the assessment focus to participants, consistent with established screening instruments like the PHQ-9 and GAD-7. The multimodal model analyzes both semantic content and prosodic features, capturing not only which terms participants use but also the surrounding context, elaboration, and emotional expression in their responses. In the final cohort, participants responded to the “Feelings Merged” question to determine whether similar predictive performance could be achieved when the question and response were provided in Spanish instead of English, with responses being translated into English for scoring, but no other modifications were made to the model architecture. Model architecture The model architecture is a multimodal design incorporating both text and audio inputs. Transcripts are generated using whisper-large37, with audio extracted from each video. The architecture consists of three components trained separately: (1) a transformer-based text encoder, (2) a transformer-based audio encoder, and (3) a classification model that combines embeddings from both text and audio to predict continuous scores for the PHQ-9, GAD-7, PCL-5, and a quality score. These components are shown visually in Fig. 10.Fig. 10Model pipeline overview.Diagram of the model pipeline used to predict mental health assessment scores and response quality from video responses. Audio is extracted and processed with HuBERT to generate prosodic embeddings, while Whisper transcribes the audio for semantic embedding via MPNet. The resulting audio and text embeddings are concatenated and passed into an XGBoost model to predict PHQ-9, GAD-7, and PCL-5 scores, along with a response quality score. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. Model pipeline overview. Diagram of the model pipeline used to predict mental health assessment scores and response quality from video responses. Audio is extracted and processed with HuBERT to generate prosodic embeddings, while Whisper transcribes the audio for semantic embedding via MPNet. The resulting audio and text embeddings are concatenated and passed into an XGBoost model to predict PHQ-9, GAD-7, and PCL-5 scores, along with a response quality score. PHQ patient health questionnaire, GAD generalized anxiety disorder scale, PCL PTSD checklist. For the text model, we selected MPNet38, a BERT-based model39, which was selected for its strong performance in text classification and semantic analysis. Given the complexity of predicting mental health scores from text, MPNet’s capacity to capture nuanced linguistic features was essential to the model’s success. A paraphrasing variant of MPNet40 was used to comprehend participants’ sometimes lengthy responses at both sentence and paragraph levels. We fine-tuned this model on our prediction task using 5-fold cross-validation, ensuring no data leakage across folds. For each fold, MPNet was fine-tuned on the training data, and the resulting embeddings were used as inputs to downstream classifiers. These classifiers had separate heads per modality and were trained only on their respective training splits. For deployment, a final model would be fine-tuned on all available data and used in inference-only mode without retraining on incoming responses. The final prediction head was removed to allow class embeddings to be used in subsequent modeling steps41. To handle multiple question types, each question was prepended with five new language tokens: one unique token for the specific question asked, and four additional tokens representing each target score (PHQ-9, GAD-7, PCL-5, and Quality Score). These tokens are the “class” embeddings mentioned throughout the paper, and are learned at training time. These tokens were designed to encourage separate representations for each prediction target, further reinforced by a custom loss function. A similar approach was utilized for the audio model. For audio, we selected HuBERT42 due to its effectiveness in learning hidden speech representations, particularly prosodic features like tone, pitch, and rhythm which are key to understanding emotion43. Since MPNet captured the content of speech, HuBERT’s focus on prosody complemented the text model well. Initial tests showed that HuBERT outperformed other audio models in predicting mental health scores, validating its use here. We fine-tuned HuBERT on our training data using a random 10-second clip from each response per epoch, then utilized a 10-s windowed average across the full response duration in the final classification model. Text and audio were trained separately due to memory constraints, though they could be optimized jointly within a unified multimodal architecture. Both embedding models were trained with a custom loss function that summed MSE losses21 for each target, weighted by the number of valid samples, since the PCL-5 was not present in all samples. A separate cross-entropy loss was used for the quality score. To prevent joint losses from causing excessive correlations among targets, a penalty was applied to each target pair if their correlation exceeded the original dataset’s cross score correlations (0.80 R)44. This approach encouraged the model to differentiate between score types, improving overall performance. The text and audio embeddings were processed independently, then concatenated for a final training step. Consistency was ensured across training and validation folds throughout the process. A final XGBoost45 model was trained on these combined embeddings to predict scores. This two-stage training process, first fine-tuning embedding models, then training XGBoost on those embeddings, used consistent fold assignments throughout to prevent data leakage between training and validation sets. We selected XGBoost based on preliminary comparisons with other algorithms, as it effectively handles heterogeneous features and captures non-linear interactions between text and audio modalities. The model applies a post-processing adjustment to align the prediction distribution with the target distribution observed during training46,47. This shift, based on a consistent offset identified between training predictions and targets, reduces the model’s tendency to regress toward the mean, which can be common with MSE loss20. In clinical deployment, both the text and audio models operate in inference-only mode. New patient responses are processed through the trained models to generate predictions without any model updates. This ensures consistent performance, computational efficiency, and eliminates the risk of model drift from continuous retraining. Model updates would only occur through formal revalidation cycles with new labeled datasets, following standard medical device software update procedures. Spanish language processing The “Feelings Merged” question was translated into a variant we describe as “Feelings Merged (Spanish)”. This was done by a native Spanish speaker to ensure linguistic and cultural appropriateness: “¿En las últimas dos semanas, te has sentido triste, nervioso, deprimido, ansioso, sin esperanza o al límite? Si es así, por favor explica con detalle cómo eso ha alterado o impactado tu vida.” This translated version was presented only to participants who had identified Spanish as their primary language. For Spanish language responses, we maintained the same multimodal approach as English responses, with one key processing difference, the Spanish verbal responses were transcribed and translated into English using a variant of Whisper with minor modifications to reduce the number of hallucinations caused by the model. After translation, the transcripts were processed through the exact same MPNet pipeline as English, and the audio was processed through the exact same HuBERT pipeline without any architectural modifications. Model validation The model was validated using 5-fold cross-validation with all responses from a given participant kept within the same fold to prevent overfitting48. The same folds were preserved consistently across text, audio, and multimodal models to ensure comparability. We further evaluated performance across multiple independent cohorts and demographic subgroups, including a Spanish-language cohort, to assess robustness beyond cross-validation splits. Splits were created based on two criteria. First, since some participants answered multiple questions in a single session, all responses from any given participant were assigned to the same fold to prevent overfitting to individual participants. Second, we stratified the data by combining target scores (PHQ-9, GAD-7, PCL-5) into a single composite score. To create this composite, we normalized each score, took the maximum of the three normalized values, and used this maximum for stratified splitting across folds49,50. Confidence intervals are estimated using n-out-of-n bootstrapping with replacement51; this process is repeated 1000 times and the predictions and scores from these bootstrapped runs used to estimate 95% confidence intervals for each metric. The model outputs predictions for the PHQ-9, GAD-7, and PCL-5 scores, along with a quality class (invalid, low, high). Quality prediction enhances model robustness by flagging cases where responses lack sufficient detail, indicating to providers when follow-up questioning may be needed for accurate assessment. This quality class can also signal a conversational agent to prompt follow-up questions. The quality score serves as an engagement indicator to identify when responses lack sufficient detail for accurate assessment. Annotators assessed whether a response provided adequate information for scoring, recognizing that this threshold varies by content. For instance, “I’m great, thanks for asking” may suffice if it reflects genuine well-being, while “I’m having the worst year of my life” requires elaboration to accurately quantify symptom severity. The goal is identifying when follow-up prompting would improve assessment accuracy. This capability addresses a limitation of traditional surveys, where minimal-effort responses (e.g., selecting the same response for all items) cannot be distinguished from genuine response patterns. Each score was evaluated using several metrics. First, we assessed the correlation between predicted and actual scores with Pearson’s R correlation52,53. Second, we measured the area under the curve (AUC) for the receiver operating characteristic (ROC) curve54. Because AUC requires a binary target, we created three binary splits based on established threshold values for PHQ-9, GAD-7, and PCL-5 (low, moderate, high), as detailed in Table 5. Lastly, we measured sensitivity and specificity54,55. which involved further binarizing the continuous predictions, using the same threshold values to ensure comparable distributions between predicted and actual scores. For demographic and question-specific performance, we scored the model at each threshold (mild, moderate, severe) and averaged these to produce a composite score. This approach facilitates easier interpretation and enables consistent comparisons across demographic and question categories. Ethics statement This study adhered to ethical guidelines governing participant privacy and data security, in line with HIPAA (Health Insurance Portability and Accountability Act) regulations and Videra Health’s data governance policies. All participants provided informed consent, and data collection was conducted following best practices for privacy and confidentiality in compliance with federal standards. As this was a privately conducted industry research study using the Prolific participant pool, formal institutional review board (IRB) approval was not required. No ethics committee review was sought as the study fell outside traditional academic oversight requirements. Nevertheless, the study followed rigorous ethical standards, including secure data handling, anonymization protocols, voluntary participation, and participants’ right to withdraw at any time. Data collection followed best practices for privacy and confidentiality in compliance with federal standards. Supplementary information Supplementary information Supplementary information