A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG
Arshia Kermani, Veronica Perez-Rosas, Vangelis Metsis · 2025
Read A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG by Arshia Kermani, Veronica Perez-Rosas, Vangelis Metsis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24307.txt.
Introduction
Kermani, Perez-Rosas, and Metsis compared three ways to put language models to work on mental health text, in a study published in 2025. One route is clever prompting. Another fetches relevant documents to ground the model. The third retrains the model on the task itself. Using one open model, they tested all three on sorting emotions and detecting mental health conditions in two datasets.
Retraining won clearly: 91 percent accuracy on emotions and 80 percent on conditions. The lighter methods managed between 40 and 68 percent, though they deploy more flexibly. Retraining's edge comes at a price in computing power and training data. The trade-off map is the paper's gift to practitioners. Clinics and researchers choosing a stack now have honest numbers for each route's cost and yield.
Catalog record
| CREATOR | Arshia Kermani, Veronica Perez-Rosas, Vangelis Metsis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-24307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG Arshia Kermani, Veronica Perez-Rosas, Vangelis Metsis Abstract This study presents a systematic comparison of three approaches for the analysis of mental health text using large language models (LLMs): prompt engineering, retrieval augmented generation (RAG), and fine-tuning. Using LLaMA 3, we evaluate these approaches on emotion classification and mental health condition detection tasks across two datasets. Fine-tuning achieves the highest accuracy (91% for emotion classification, 80% for mental health conditions) but requires substantial computational resources and large training sets, while prompt engineering and RAG offer more flexible deployment with moderate performance (40-68% accuracy). Our findings provide practical insights for implementing LLM-based solutions in mental health applications, highlighting the trade-offs between accuracy, computational requirements, and deployment flexibility. 1 Introduction The increasing prevalence of mental health conditions, coupled with limited access to mental health professionals, has created an urgent need for scalable approaches to mental health assessment and support. Traditional diagnostic methods in this area often rely heavily on clinical interviews and self-reported questionnaires, which can be time-consuming, subject to human bias, and limited in their reach Chung and Teo 2023 . Recent advances in large language models (LLMs) present promising opportunities to enhance mental health assessment through automated analysis of text-based data. LLMs have demonstrated remarkable capabilities in understanding and generating human language, with recent models like GPT-4, LLaMA 2 Touvron et al. 2023 , and their derivatives achieving unprecedented performance across various natural language processing tasks Brown et al. 2020 . In the medical domain specifically, LLMs have shown potential in tasks ranging from clinical decision support to patient education and medical documentation Thirunavukarasu et al. 2023 . However, their application to mental health assessment presents unique challenges due to the nuanced nature of emotional expression and the critical importance of accuracy in clinical contexts. Previous research has explored various approaches to leverage LLMs for mental health applications. Studies have investigated the use of zero-shot and few-shot prompt strategies for mental health text classification Lamichhane 2023 , achieving moderate success in tasks such as detecting stress and depression. Other work has examined the potential of fine-tuned models for specific mental health tasks Ezerceli and Dehkharghani 2024 , demonstrating improved performance through domain adaptation. However, there remains a significant gap in understanding the relative efficacy of different LLM deployment strategies for mental health assessment tasks. Our study addresses this gap by conducting a systematic comparison of three distinct approaches for mental health text classification: prompt engineering (including both zero-shot and few-shot variants), retrieval augmented generation (RAG), and fine-tuning. We evaluated these approaches using two complementary datasets: the DAIR-AI Emotion dataset, comprising 20,000 tweets labeled with six basic emotions, and the Reddit SuicideWatch and Mental Health Collection (SWMH), which contains 54,412 posts related to various mental health conditions. This work makes several key contributions to the field: 1. We provide the first comprehensive comparison of prompt engineering, RAG, and fine-tuning approaches for mental health text classification, offering insights into their relative strengths and limitations. 2. We demonstrate the effectiveness of LLaMA 3-based models for mental health assessment tasks, achieving accuracy rates of up to 91% on emotion classification and 80% on mental health condition classification through fine-tuning. 3. We present practical insights into the implementation challenges and resource requirements of each approach, informing future applications in clinical settings. Our findings have important implications for the development of automated mental health assessment tools, suggesting that while fine-tuning achieves the highest accuracy, both prompt engineering and RAG offer viable alternatives with different trade-offs in terms of computational resources and deployment flexibility. These results contribute to the broader goal of developing reliable, scalable tools to support mental health professionals and improve access to mental health assessment. In the following sections, we present related work and background (Section 2 ), detail our methodology (Section 3 ), present our experimental results (Section 4 ), discuss their limitations (Section 7 ), and conclude in (Section 5 ). 2 Related Work The intersection of large language models (LLMs) and mental health assessment represents a rapidly evolving field with significant potential for improving healthcare delivery. This section examines the current state of LLMs in healthcare applications and their specific developments in mental health contexts. 2.1 Large Language Models in Healthcare Recent advances in LLMs have transformed their potential applications in healthcare He et al. 2023 . These models have demonstrated capabilities ranging from clinical decision support and medical documentation to patient education and healthcare communication Thirunavukarasu et al. 2023 . The emergence of domain-specific medical LLMs, such as Med-PaLM 2 and Clinical-Camel, has further enhanced their utility in healthcare settings by incorporating specialized medical knowledge and terminology Singhal et al. 2025 . The use of LLMs in healthcare applications typically follows three main strategies: fine-tuning existing models, prompt engineering, and retrieval-augmented generation (RAG). Fine-tuning has shown particular promise in specialized medical tasks, with models achieving performance comparable to healthcare professionals in diagnostic scenarios Singhal et al. 2025 . Prompt engineering approaches have shown effectiveness in zero-shot and few-shot learning contexts, allowing flexible deployment without extensive retraining Liu et al. 2023 . RAG methods have emerged as a promising approach for grounding LLM responses with domain knowledge, thereby reducing hallucination and improving reliability Lewis et al. 2020 ; Gao et al. 2023 . 2.2 Mental Health Text Analysis Mental health assessment presents unique challenges for automated analysis due to the subtle nature of emotional expression and the critical importance of accurate interpretation. Traditional approaches to the analysis of mental health text have relied on rule-based systems and classical machine learning techniques, often struggling to capture the nuanced context necessary for an accurate assessment Kazdin 2011 . Recent work has begun to explore the potential of LLMs for mental health applications. Studies have shown promising results in the detection of signs of depression, anxiety, and suicidal ideation from social media posts Ma et al. 2024 . The evolution of LLM capabilities has particular relevance for mental health research. Recent studies have shown that advanced LLM versions can provide human-level interpretations in qualitative coding tasks Dunivin 2024 and achieve accuracy comparable to mental health professionals in certain diagnostic contexts Kim et al. 2024 . When applied to qualitative analysis, LLMs have demonstrated the ability to perform various analytical approaches, including thematic analysis, content analysis, and grounded theory, as validated by human experts Xiao et al. 2023b ; Rasheed et al. 2024 . This suggests potential for enhancing, rather than replacing, traditional qualitative analysis methods in mental health research. However, these applications pose important challenges, including the need for accurate predictions given the critical nature of mental health assessment, concerns about privacy and data security, and the importance of maintaining therapeutic alliance in clinical settings Byers et al. 2023 . Figure 1: Overview of our experimental framework comparing three LLM deployment approaches (fine-tuning, prompt engineering, and RAG) for mental health text analysis. Each approach processes the same input data through distinct pipelines, enabling systematic comparison of their effectiveness. 2.3 Evaluation Frameworks The evaluation of LLMs in mental health applications requires careful consideration of both model performance and clinical utility. Although traditional metrics such as accuracy and F1 scores