Analyzing LLM Reasoning to Uncover Mental Health Stigma

Sreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz · 2026

Read Analyzing LLM Reasoning to Uncover Mental Health Stigma by Sreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25053.txt.

Introduction

Sankar and ten colleagues analyse large language model thought to uncover mental health stigma. Existing stigma evaluations rely on choice questions, which miss biases hidden in the models' logic. The authors analyse intermediate thought steps to uncover hidden stigmatizing language plus the internal reasons driving it. Clinical experts sort common patterns of stigmatizing language, and the framework tags problematic statements in model thought and rates their severity. Evaluating thought exposes substantially more stigma than choice methods and helps identify flaws in model logic and understanding of mental health.

Catalog record

CREATORSreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56310
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analyzing LLM Reasoning to Uncover Mental Health Stigma Sreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz Abstract While large language models (LLMs) are increasingly being explored for mental health applications, recent studies reveal that they can exhibit stigma toward individuals with psychological conditions. Existing evaluations of this stigma primarily rely on multiple-choice questions (MCQs), which fail to capture the biases embedded within the models’ underlying logic. In this paper, we analyze the intermediate reasoning steps of LLMs to uncover hidden stigmatizing language and the internal rationales driving it. We leverage clinical expertise to categorize common patterns of stigmatizing language directed at individuals with psychological conditions and use this framework to identify and tag problematic statements in LLM reasoning. Furthermore, we rate the severity of these statements, distinguishing between overt prejudice and more subtle, less immediately harmful biases. To broaden the reasoning domain and capture a wider array of patterns, we also extend an existing mental health stigma benchmark by incorporating additional psychological conditions. Our findings demonstrate that evaluating model reasoning not only exposes substantially more stigma than traditional MCQ-based methods but it helps to identify the flaws in the LLMs’ logic and their understanding of mental health conditions. 1 Introduction Limited access to therapists has led to an increase in the implementation of large language models for mental health applications, ranging from supportive chatbots to systems positioned as replacements for human therapists ( De Choudhury et al. 2023 ; Na et al. 2025 ) . Commercially available therapy bots already interact with millions of users ( De Freitas and Cohen 2024 ) . This is happening despite mounting evidence that they respond inappropriately to serious mental health conditions ( Moore et al. 2025 ; Grabb et al. 2024 ) . A critical concern in this space is stigma ( Corrigan and Watson 2002 ; Thornicroft 2007 ) , which in clinical settings leads to lower-quality care and misdiagnosis ( Shefer et al. 2014 ) . Using a Multiple Choice Question (MCQ) survey from the U.S. National Stigma Studies ( Pescosolido et al. 2021 ) , Moore et al. 2025 demonstrated that LLMs exhibit significant stigma toward individuals with mental health conditions. However, MCQ-based evaluation leaves the LLM’s underlying reasoning unexamined. Furthermore, LLM chain-of-thought (CoT) reasoning can be unfaithful, or models may arrive at correct answers through flawed logic ( Turpin et al. 2024 ; Lanham et al. 2023 ) . Figure 1: Example of mental health vignettes, survey questions to capture stigma and LLM responses with the stigmatizing traces of the reasoning shown in bold font. The left panel shows a model contradicting its reasoning and advising caution against a healthy individual. The middle panel is stigmatizing because it shows a model accepting someone solely because their condition is treatable and also continues the reasoning by using performative language to justify its final answer. Finally, the right panel shows a case of overt stigma successfully captured by the MCQ format, where analyzing the generated text reveals the model’s underlying logic. To address these limitations, we move beyond MCQ evaluation to analyze the reasoning traces of LLMs when responding to mental health stigma scenarios. As shown in Figure 1 , examining these traces exposes instances of unfaithful or subtly harmful reasoning. For instance, in one example, a model selects the non-stigmatizing answer but simultaneously advises caution regarding the individual, which is an inherently paradoxical response on the surface. In another, the model accepts the person because their condition is treatable, with the stigmatizing implication that a chronic condition warrants social rejection. Finally, even in cases where overt and clear stigma is successfully captured by the MCQ format (Figure 1 , right), analyzing the generated text provides critical insight into the underlying logic driving the model’s choice. We prompt various models, including Anthropic’s Claude ( Anthropic 2024 ) , Meta’s Llama ( Grattafiori et al. 2024 ) , and DeepSeek ( DeepSeek-AI 2024 ) , to produce CoT explanations alongside their answers. In this work, we leverage clinical expertise to create a taxonomy for mental health related stigma and use it to annotate these reasoning traces for stigmatizing language and logical inconsistencies. Additionally, we incorporate additional psychological conditions that were not covered in prior work but carry substantial clinical significance. This results in a rich taxonomy of common stigma patterns observed across various LLMs and insight into their stigmatizing logic. Our contributions are as follows: • We conduct the first analysis of stigmatizing language in LLM reasoning with regards to mental health, demonstrating that models exhibit substantially more latent stigma than is captured by MCQ-based evaluations. • We introduce a clinically grounded taxonomy of recurring stigmatizing reasoning patterns, providing a structured framework for identifying both overt and subtle stigmatizing logic in mental health scenarios. • We release an extended benchmark covering additional psychological conditions, alongside a rich, annotated dataset of reasoning traces to support future mitigation efforts 1 1 1 Dataset and code will be released upon acceptance. . 2 Related Work Stigma toward mental health conditions is a well-documented barrier to care. In clinical settings, provider stigma leads to lower-quality treatment, and care avoidance ( Shefer et al. 2014 ; Corrigan and Watson 2002 ; Thornicroft 2007 ) . The measurement of mental health stigma has a rich tradition in social science, with instruments such as the U.S. National Stigma Studies within the General Social Survey providing longitudinal data on public attitudes ( Pescosolido et al. 2021 ) . In the context of AI systems, bias and fairness concerns have been extensively studied, though less so for mental health specifically. Poulain et al. 2024 documented bias patterns in LLMs used for clinical decision support. Aleem et al. 2024 found that ChatGPT exhibits poor multicultural awareness in therapeutic settings. Cross et al. 2024 reviewed broader implications of bias in medical AI for clinical decision-making. Tamkin et al. 2023 developed methods for evaluating and mitigating discrimination in language model decisions. Most directly relevant to our work, Moore et al. 2025 adapted the stigma measurement framework from Pescosolido et al. 2021 to evaluate LLMs. They presented models with vignettes describing individuals meeting criteria for schizophrenia, major depression, alcohol dependence, and a “daily troubles” control, then asked social association questions (e.g., willingness to work closely, be friends, be neighbors). They found that models including GPT-4o and the Llama family showed significant stigma, particularly toward schizophrenia and alcohol dependence, and that scaling up model size did not consistently reduce stigma. They also found that models responded inappropriately to stimuli indicating delusions, suicidal ideation, hallucinations, mania, and obsessive-compulsive behavior, performing significantly worse than human therapists. However, their evaluation was limited to MCQ responses. The present work extends this line of research by examining model reasoning , where we find that stigma is substantially more pervasive than final answers alone suggest. 3 Evaluation Framework Following Moore et al. 2025 , who adapted vignettes from the U.S. National Stigma Studies ( Pescosolido et al. 2021 ) , we present LLMs with vignettes about fictitious individuals with mental health conditions and evaluate their responses to standardized stigma-related questions, with the difference that our work evaluates the model’s entire reasoning process, such as the examples shown in Figure 1 ,

Read the complete text