Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches ∗ These authors contributed equally to this work.
Obed Junias ∗, Prajakta Kini ∗, Theodora Chaspari · 2025
Read Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches ∗ These authors contributed equally to this work. by Obed Junias ∗, Prajakta Kini ∗, Theodora Chaspari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25795.txt.
Introduction
Obed Junias, Prajakta Kini, and Theodora Chaspari compare two kinds of models for detecting depression from language, with a focus on bias, in a 2025 paper. They ask whether scores differ by gender and by race or ethnic group. The data are transcripts of clinical interviews from a known corpus.
Deep neural models are compared with large language models used with a few examples. The language models did better overall, above all for groups often underrepresented, such as Hispanic participants. They also showed less gender bias, though racial gaps remained. The team test fixes on both sides, from fair loss functions for the deep models to varied prompts for the language models.
Catalog record
| CREATOR | Obed Junias ∗, Prajakta Kini ∗, Theodora Chaspari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches ∗ These authors contributed equally to this work. Obed Junias ∗, Prajakta Kini ∗, Theodora Chaspari Abstract This paper investigates algorithmic bias in language-based models for automated depression detection, focusing on socio-demographic disparities related to gender and race/ethnicity. Models trained using deep neural networks (DNN) based embeddings are compared to few-shot learning approaches with large language models (LLMs), evaluating both performance and fairness on clinical interview transcripts from the Distress Analysis Interview Corpus/Wizard-of-Oz (DAIC-WOZ). To mitigate bias, fairness-aware loss functions are applied to DNN-based models, while in-context learning with varied prompt framing and shot counts is explored for LLMs. Results indicate that LLMs outperform DNN-based models in depression classification, particularly for underrepresented groups such as Hispanic participants. LLMs also exhibit reduced gender bias compared to DNN-based embeddings, though racial disparities persist. Among fairness-aware techniques for mitigating bias in DNN-based embeddings, the worst-group loss, which is designed to minimize loss for the worst-performing demographic group, achieves a better balance between performance and fairness. In contrast, the fairness-regularized loss minimizes loss across all groups but performs less effectively. In LLMs, guided prompting with ethical framing helps mitigate gender bias in the 1-shot setting. However, increasing the number of shots does not lead to further reductions in disparities. For race/ethnicity, neither prompting strategy nor increasing N N in N N -shot learning effectively reduces disparities. I Introduction Advances in natural language processing (NLP) have opened promising avenues for diagnosis and treatment monitoring of depression through identifying linguistic cues in spoken or written language [ 1 ] . Prior work suggests that individuals with depression tend to use more first-person singular pronouns, negative emotion words, and absolutist terms [ 2 , 3 , 4 ] . Their speech may also exhibit greater abstraction, redundancy [ 5 ] , and lexical repetition [ 6 ] . Early approaches to detecting depression from text relied on lexicon-based features [ 7 , 8 ] and embeddings derived from Word2Vec [ 9 , 10 ] combined with traditional models [ 11 ] . Since these methods often fail to capture contextual semantics, later work adopted deep contextual embeddings from models such as Bidirectional Encoder Representations from Transformers (BERT) and RoBERTa [ 12 , 13 ] . More recently, large language models (LLMs) have been explored under zero- and few-shot settings for depression detection [ 14 , 15 , 16 , 17 , 18 ] . While advances in NLP have enhanced depression detection, an increasing number of studies highlight persistent demographic biases in such models. Studies analyzing social media content reveal that linguistic features used for depression detection vary across demographic groups [ 19 ] . LLMs also exhibit differential performance in detecting depression from spoken data across dimensions such as age, gender, and language [ 20 ] , as well as in clinical decision-making tasks [ 21 ] . These disparities may reflect underlying differences in depression prevalence and the severity of symptoms. For example, there is systematic evidence regarding higher rates of depression among women compared to men [ 22 , 23 ] , and greater severity among Black and Hispanic individuals compared to non-Hispanic White individuals [ 24 ] . Disparities may also stem from gender and race-related differences in language use, including differences in vocabulary, grammar, turn-taking, and interruption [ 25 , 26 , 27 ] . Despite these concerns, most algorithmic fairness assessments in depression detection focus on audio or multi-modal systems [ 28 , 29 ] , with limited attention to text-only models trained on clinical interviews. The widespread adoption of LLMs further complicates this problem, as it remains unclear whether out-of-the-box LLMs amplify existing socio-demographic disparities in the focal task. Research on bias mitigation in ML models for depression detection has primarily focused on tabular data, such as electronic health records [ 30 ] and lifestyle or environmental factors [ 31 ] . Relatively few studies have examined bias mitigation techniques applied to text data from social media posts [ 32 ] or to multimodal data from clinical interviews [ 33 ] . While recent work has explored debiasing strategies, including in-context learning, in general-purpose LLMs [ 34 , 35 , 36 ] , the evaluation of these methods in clinical contexts remains unexplored. This paper investigates algorithmic bias in language-based models for depression detection. It compares trade-offs between depression detection performance and algorithmic bias on conventional models trained using DNN-based embeddings with few-shot learning approaches using large language models (LLMs). The study further explores fairness-aware modeling constraints in mitigating bias in language-based depression detection models trained with DNN-based embeddings, and in-context learning for mitigating algorithmic bias in LLM-based depression detection. Experiments are conducted on the Distress Analysis Interview Corpus/Wizard-of-Oz (DAIC-WOZ) dataset [ 37 ] , with gender and race/ethnicity analyzed as the primary sources of bias. We structure our investigation around the following research questions: • RQ1: How do DNN-based embeddings compare to LLMs in language-based depression detection performance and algorithmic bias? • RQ2: How do different loss functions, implemented as fairness-aware modeling constraints, affect algorithmic bias and depression detection in DNN-based models? • RQ3: How does prompt framing and number of examples in N -shot learning influence algorithmic bias and depression detection performance in LLMs? This paper makes the following contributions to the existing body of knowledge: (1) We compare DNN-based embeddings and LLMs for language-based depression detection, not only in terms of performance, as explored in prior work [ 14 , 15 , 16 , 17 , 18 ] , but also in terms of algorithmic fairness; (2) We examine various fairness-aware modeling constraints as bias mitigation strategies for text-based depression detection, a task relatively unexplored, with prior work mostly focusing on tabular or multimodal data [ 30 , 31 , 33 ] ; and (3) We investigate in-context learning techniques using different prompt framings to mitigate bias in LLMs, an approach that, to our knowledge, has not yet been studied for depression detection. II Related Work Research on text-based automated depression detection has focused primarily on social media posts and clinical interviews [ 38 ] . Literature reviews by William & Suhartono and Babu & Kanaga concluded that early depression detection via social media is feasible, given identifiable linguistic and behavioral patterns [ 39 , 40 ] . Liu et al. reviewed ML techniques for depression detection, highlighting the use of features such as word frequencies, psycholinguistic and emotional terms, temporal references, and topic modeling, used with supervised and unsupervised algorithms [ 41 ] . Husseini et al. found that convolutional neural networks (CNNs) with max-pooling achieved better performance and generalization on unseen data when compared to recurrent neural networks (RNNs) [ 42 ] , while Sadeque et al. observed better performance from RNNs over non-sequential models [ 43 ] . Pre-trained mental health specific models like MentalBERT and MentalRoBERTa have shown further gains in performance [ 12 ] . Recent work has begun to explore the capabilities of LLMs for this task. Zero-shot settings show mixed results: some studies suggest that these struggle with domain-specific subtleties [ 15 ] , while advanced architecture-based models like GPT-4 perform well [ 14 ] .