TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes
Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade · 2025
Read TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes by Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20648.txt.
Introduction
Raj Sanjay Shah and five co-authors present TN-Eval in 2025, a way to judge the quality of behavioral therapy notes. Clear standards for such notes lag behind those for physical health notes, even though the notes matter for care and for the law. Working with licensed therapists, the team built a scoring guide that rates notes on completeness, conciseness, and faithfulness.
They applied the guide to notes written by therapists and notes written by language models. The guide gave steadier, clearer results than simple rating scales. Models matched human judges on two of the three scores but struggled with faithfulness. Therapist notes often missed content, model notes sometimes invented it, yet blinded therapists still preferred the model notes.
Catalog record
| CREATOR | Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade Abstract Behavioral therapy notes are important for both legal compliance and patient care. Unlike progress notes in physical health, quality standards for behavioral therapy notes remain underdeveloped. To address this gap, we collaborated with licensed therapists to design a comprehensive rubric for evaluating therapy notes across key dimensions: completeness , conciseness , and faithfulness . Further, we extend a public dataset of behavioral health conversations with therapist-written notes and LLM-generated notes, and apply our evaluation framework to measure their quality. We find that: (1) A rubric-based manual evaluation protocol offers more reliable and interpretable results than traditional Likert-scale annotations. (2) LLMs can mimic human evaluators in assessing completeness and conciseness but struggle with faithfulness. (3) Therapist-written notes often lack completeness and conciseness, while LLM-generated notes contain hallucination. Surprisingly, in a blind test, therapists prefer and judge LLM-generated notes to be superior to therapist-written notes. 1 Introduction Automated medical note generation using large language models (LLMs) has the potential to enhance clinicians’ efficiency by reducing the time spent on electronic health records, allowing them to focus more on patient care. However, applying LLMs to behavioral health notes presents unique challenges ( Hua et al., 2024 ) . In therapy, the conversation itself is the treatment; therefore, techniques like motivational interviewing used in a session may not be explicitly stated. Furthermore, sessions cover various topics, making it crucial to discern significant details from less relevant information. Given the high-stakes nature of behavioral health, using LLMs to generate notes must be rigorously evaluated to ensure they capture key information at an appropriate level of detail. Evaluating the quality of talk therapy notes, however, is not straightforward. Traditionally, human evaluation has been the primary method for assessing their quality, making it resource intensive and costly. Moreover, a lack of standardized reference notes and the limited literature on what constitutes an effective behavioral health therapy note further complicates the evaluation process. Therapists and healthcare providers often have their own styles and preferences, leading to subjective assessments and considerable variation. Without clear standards and evaluation protocols, it becomes difficult to determine the quality of LLM-generated therapy notes. In this work, we focus on the SOAP (Subjective, Objective, Assessment, Plan) format of therapy notes and propose an evaluation framework for notes (TN-Eval). The framework includes (1) a comprehensive, fine-grained, section-wise rubric that outlines the key components and characteristics of a therapy note and (2) both human and automatic evaluation protocols. The rubric, which we co-designed with 5 licensed therapists, details the relevant items for each of the four SOAP sections and their respective levels of importance (Section 3 ) . We then design a human evaluation protocol – TN H -Eval – in which 9 licensed behavioral health therapists from diverse backgrounds assess notes along three dimensions: completeness, conciseness, and faithfulness (Section 4.1 ) . The completeness and conciseness are scored with reference to the rubric to improve the consistency of the evaluation, while faithfulness is evaluated at the sentence level with source attribution ( Rashkin et al., 2023 ) . Finally, we explore the potential of LLMs to emulate expert evaluations, introducing an automatic evaluation protocol called TN A -Eval (Section 4.2 ) . Our experimental results show that our proposed human evaluation protocol – TN H -Eval achieves higher IAA ( IAA ) compared to conventional Likert-scale human evaluation, making it more reliable. We additionally show that using the automatic evaluation protocol – TN A -Eval, we can achieve a better correlation with TN H -Eval on completeness and conciseness evaluation when compared to N-gram-based metrics like ROUGE Lin (2004) or conventional LLM-as-a-Judge Zheng et al. (2023) , making it a quick and cost-effective solution for evaluation. When compared to expert-written notes, we find that LLM-generated notes achieve around 10% higher scores in completeness and conciseness but show relatively lower faithfulness. Deployment considerations: Our TN A -Eval is a deployable and scalable framework for assessing therapy notes with fine-grained, human-like judgments, which is designed by domain experts and has been evaluated on available datasets. Integrating this evaluation into clinical workflows and EHR systems enables: (1) Automated review that flags low-quality notes; (2) Automated scoring systems that assist therapists in refining notes before submission, reducing post-session documentation workload; and (3) Cost-effective, scalable quality assessments in standardized documentation practices. Refer appendix section I for workflow integration suggestions. 2 Related Work AI in Mental Health Care: Recently, interest in using LLMs for mental health care has grown ( Greer et al., 2019 ; Peng et al., 2020 ; Srivastava et al., 2022 ; Luo et al., 2025 ) , with research focusing on three main directions. First, to classify therapeutic methods used by clinicians, assess the effectiveness of treatments, and predict the quality of service ( Saha and Sharma, 2020 ; Chikersal et al., 2020 ; Liu et al., 2021 ; Shah et al., 2022 ) . Second, virtual counselors emulate human behavior in chatbot-like environments ( Shen et al., 2020 ; O’neil et al., 2023 ) , but ethical and legal concerns ( Woodnutt et al., 2024 ; Stade et al., 2024 ) have shifted research toward augmenting therapists with suggestions to enhance their responses ( Saha et al., 2022 ; Sharma et al., 2023a ) . Third, AI tools train novice counselors by providing automatic feedback ( Chaszczewicz et al., 2024 ; Lin et al., 2024 ) and simulating client personas for role-play ( Chen et al., 2023 ; Stapleton et al., 2023 ; Wang et al., 2024 ; Louie et al., 2024 ) . Despite growing interest in AI for mental health support, LLMs for behavioral therapy note generation remain underexplored. Automated clinical note generation: improves clinician efficiency ( Joshi et al., 2020 ) , with research primarily focused on physical health using role-play or anonymized conversations and human-written notes ( Papadopoulos Korfiatis et al., 2022 ; Ben Abacha et al., 2023 ; Yim et al., 2023 ) . Early work fine-tuned lightweight transformer models ( Sharma et al., 2023b ; Michalopoulos et al., 2022 ; Milintsevich and Agarwal, 2023 ; Yuan et al., 2024 ) , while recent studies explore LLM prompting for summarization ( Ben Abacha et al., 2023 ; Mathur et al., 2023 ) . Automatic evaluations for summarization: Reference-based metrics like ROUGE ( Lin, 2004 ) , BLEU ( Papineni et al., 2002 ) , and BERTScore ( Zhang et al., 2020 ) measure lexical similarity between generated and reference summaries while fact-checking evaluators assess faithfulness to the source ( Honovich et al., 2022 ; Zha et al., 2023 ; Laban et al., 2022 ) . Recently, LLMs have been increasingly used as evaluators, known as LLM-as-a-Judge ( Zheng et al., 2023 ; Wang et al., 2023 ) . However, these methods are usually developed for general text summarization tasks and do not account for the challenges of therapy notes, where obtaining high-quality reference summaries is complex, and evaluations require substantial domain knowledge. Our TN A -Eval builds upon existing methodologies, adapting them to address these nuances. Figure 1: The TN H -Eval human evaluation