Assessment of AI-Generated Pediatric Rehabilitation SOAP-Note Quality

Solomon Amenyo, Maura R. Grossman, Daniel G. Brown, Brendan Wylie-Toal · 2025

Read Assessment of AI-Generated Pediatric Rehabilitation SOAP-Note Quality by Solomon Amenyo, Maura R. Grossman, Daniel G. Brown, Brendan Wylie-Toal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15526.txt.

Introduction

Solomon Amenyo and co-authors assess notes made by AI tools for child rehab care, in a 2025 study. Clinics write these notes in a fixed form with four parts, and the work takes much time. The team compares a broad commercial tool and a tool tuned for one Ontario centre for child growth.

The study set held 432 notes, split in equal parts among human written, and the two AI tools. Skilled staff judged the notes blind, with a custom score sheet. Both AI tools made notes close in quality to human notes. The authors also test how much human editing helps, and they point to gains from pairing AI with staff skill.

Catalog record

CREATORSolomon Amenyo, Maura R. Grossman, Daniel G. Brown, Brendan Wylie-Toal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15526

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessment of AI-Generated Pediatric Rehabilitation SOAP-Note Quality Solomon Amenyo, Maura R. Grossman, Daniel G. Brown, Brendan Wylie-Toal Abstract This study explores the integration of artificial intelligence (AI) or large language models (LLMs) into pediatric rehabilitation clinical documentation, focusing on the generation of SOAP (Subjective, Objective, Assessment, Plan) notes, which are essential for patient care. Creating complex documentation is time-consuming in pediatric settings. We evaluate the effectiveness of two AI tools; Copilot, a commercial LLM, and KAUWbot, a fine-tuned LLM developed for KidsAbility Centre for Child Development (an Ontario pediatric rehabilitation facility), in simplifying and automating this process. We focus on two key questions: (i) How does the quality of AI-generated SOAP notes based on short clinician summaries compare to human-authored notes, and (ii) To what extent is human editing necessary for improving AI-generated SOAP notes? We found no evidence of prior work assessing the quality of AI-generated clinical notes in pediatric rehabilitation. We used a sample of 432 SOAP notes, evenly divided among human-authored, Copilot-generated, and KAUWbot-generated notes. We employ a blind evaluation by experienced clinicians based on a custom rubric. Statistical analysis is conducted to assess the quality of the notes and the impact of human editing. The results suggest that AI tools such as KAUWbot and Copilot can generate SOAP notes with quality comparable to those authored by humans. We highlight the potential for combining AI with human expertise to enhance clinical documentation and offer insights for the future integration of AI into pediatric rehabilitation practice and other settings for the management of clinical conditions. K eywords SOAP Notes, LLM, AI-Generated, Pediatric Rehabilitation, Quality Assessment 1 Introduction 1.1 SOAP Notes in Pediatric Rehabilitation Care The importance of clinical documentation in healthcare, particularly in pediatric rehabilitation, cannot be overstated. SOAP notes, which stands for Subjective, Objective, Assessment, and Plan, provide a structured framework for clinicians to systematically record patient interactions and treatment plans [ 1 , 2 ] . These notes are not merely bureaucratic exercises; they are vital tools that inform clinical decision-making, ensure continuity of care, and facilitate effective communication among healthcare providers [ 1 ] . In pediatric rehabilitation, where children’s health status can change rapidly, accurate and timely documentation is even more important [ 3 ] . Clinicians must navigate child development, varying treatment modalities, and precise tracking of progress and outcomes, all of which emphasize the critical nature of high-quality SOAP notes in this field [ 3 ] . Accurate documentation is essential not just for individual patient care but also for broader treatment planning, and quality assurance within healthcare systems [ 4 ] . Inaccuracies or omissions in clinical notes can lead to miscommunication, which may result in diagnostic errors and inappropriate treatment, potentially jeopardizing patient safety [ 4 ] . Unfortunately, the increasing burden of documentation on healthcare providers has been linked to clinician burnout, a pressing issue worsened by the ongoing shortage of primary care physicians and the growing demands placed on them [ 5 ] . Effective clinical documentation practices are, therefore, vital not only for enhancing patient-care outcomes but also for alleviating some of the administrative pressures faced by healthcare professionals. 1.2 The Role of AI in Healthcare Documentation The dawn of artificial intelligence (AI) technologies has introduced new paradigms in clinical documentation practices [ 6 ] . AI-driven systems, particularly those using natural language processing (NLP) and machine learning algorithms, are being developed to automate the generation of clinical notes [ 7 ] , including SOAP notes. Such technologies offer the promise of improving documentation efficiency, reducing the time clinicians spend on paperwork, and potentially increasing the quality of the notes produced. The introduction of automated systems like CliniKnote [ 8 ] highlights the potential for AI to enhance the training and evaluation of models in clinical note generation, thereby simplifying workflows in healthcare settings. However, the integration of AI into clinical documentation also raises significant questions regarding the quality and reliability of AI-generated content. Assessing the quality of AI-generated SOAP notes is of paramount importance. Despite the potential advantages, there are concerns regarding the accuracy, completeness, and adherence of these automated notes to clinical guidelines. Recent studies have shown the challenges associated with AI-generated documentation, revealing a range of errors, including high rates of omissions and inaccuracies [ 9 , 10 ] . For example, a study exploring the capabilities of ChatGPT-4 in generating SOAP notes found that the model produced an average of 23.6 errors per clinical case, with omissions constituting the majority of these errors [ 11 ] . Such findings highlight the need for rigorous evaluation of AI-generated notes, particularly in sensitive areas such as pediatric rehabilitation, where the implications of documentation errors can be profound. 2 Background In this study, Microsoft’s Copilot tool, a commercially available AI system (chatbot), was used to generate SOAP notes for pediatric occupational therapy sessions. These notes were then reviewed and edited by occupational therapists (OTs) to ensure accuracy and completeness. Similarly, KAUWbot, a custom-built AI tool developed in collaboration between KidsAbility and the University of Waterloo, was also used to generate SOAP notes. This tool offered the unique opportunity to evaluate both unedited and edited versions of the SOAP notes, allowing the research team to assess the baseline quality of AI-generated notes and to measure the added value of human editing. The central problem this study aims to address is to assess the quality and reliability of AI-generated SOAP notes in pediatric rehabilitation. While AI tools like Copilot and KAUWbot hold promise for reducing clinician workload [ 12 ] and improving efficiency, it is important to evaluate whether these tools produce SOAP notes that meet the rigorous quality standards required in healthcare documentation. We focus on two research questions: (i) How does the quality of AI-supported SOAP notes compare to that of human-authored SOAP notes in pediatric rehabilitation settings? and (ii) To what extent is human editing necessary for refining AI-generated SOAP note drafts, and how does it impact their overall quality? We explore the role AI can play in clinical documentation and the degree to which human oversight remains necessary to ensure the quality and accuracy of SOAP notes in pediatric rehabilitation. We contribute to the broader discussion on integrating AI into clinical workflows and offer practical recommendations for using AI in pediatric rehabilitation documentation. 2.1 Pediatric Occupational Therapy Clinics Pediatric OT aims to help children and youth who experience challenges in activities of daily life due to physical, cognitive, or developmental disabilities or delays [ 13 ] . Common case types include developmental delays, autism spectrum disorder, sensory processing disorders, cerebral palsy, and traumatic brain injury [ 14 ] . Clinics serve children across a broad age range, from infants to adolescents, with interventions tailored to each developmental stage. Infants may receive early intervention for conditions like cerebral palsy, while older children and teens may work on skills needed for school tasks or independent living [ 15 ] . Sessions address fine and gross motor skills, sensory processing, cognitive and social development, and self-care skills, often through engaging, play-based activities [ 15 ] . Therapists in OT clinics use a variety of approaches, such as strengthening exercises, sensory integration therapy, adaptive equipment training, and activities

Read the complete text