Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework

Sadia Kamal, Tim Oates, Joy Wan · 2025

Read Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework by Sadia Kamal, Tim Oates, Joy Wan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10328.txt.

Introduction

Sadia Kamal, Tim Oates, and Joy Wan built a system in twenty twenty-five that drafts clinical SOAP notes for skin cancer visits. SOAP notes — subjective, objective, assessment, and plan — are the standard record of a patient visit, but writing them by hand eats clinician time and feeds burnout. Skin cancer is the most common cancer worldwide, so the note load is huge.

Their framework learns from weak supervision, needing few hand-labeled examples. It takes a lesion image plus sparse clinical text and produces a structured note. Scores on clinical relevance metrics matched those of top commercial models. The authors add two new metrics of their own, checking alignment with expert medical concepts and coherence with the input. The goal is documentation that scales without draining clinicians.

Catalog record

CREATORSadia Kamal, Tim Oates, Joy Wan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework Sadia Kamal, Tim Oates, Joy Wan Abstract Skin carcinoma is the most prevalent form of cancer globally, accounting for over $8 billion in annual healthcare expenditures. In clinical settings, physicians document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, manually generating these notes is labor-intensive and contributes to clinician burnout. In this work, we propose a weakly supervised multimodal framework to generate clinically structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Our approach reduces reliance on manual annotations, enabling scalable, clinically grounded documentation while alleviating clinician burden and reducing the need for large annotated data. Our method achieves performance comparable to GPT-4o, Claude, and DeepSeek Janus Pro across key clinical relevance metrics. To evaluate clinical quality, we introduce two novel metrics MedConceptEval and Clinical Coherence Score (CCS) which assess semantic alignment with expert medical concepts and input features, respectively. 1 Introduction Skin cancer remains one of the most common and deadliest cancers in the United States, with approximately 9,500 new cases diagnosed daily Rogers et al. 2015 , highlighting the crucial role of clinical documentation as the foundation for effective communication, accurate diagnosis, and informed treatment planning. Structured formats like SOAP (Subjective, Objective, Assessment, Plan) notes are widely adopted in the United States to ensure consistency in recording patient encounters and minimizing communication errors among healthcare professionals Schloss and Konam 2020 . However, generating these notes is labor-intensive and time-consuming, which reduces direct patient interaction time and significantly contributes to physician burnout Li et al. 2024 ; Biswas and Talukdar 2024 . Automating the generation of SOAP notes presents a promising solution to reduce administrative burden, improve documentation consistency, and allow clinicians to focus more on patient centred care. Recent advances in large language models (LLMs) have enabled impressive progress in medical natural language processing tasks, including clinical summarization, question answering Singhal et al. 2023 , lab report interpretation He et al. 2024 , and deidentification of sensitive information Yang et al. 2023 . These models can produce coherent and fluent clinical narratives, making them useful tools for medical documentation. However, general purpose LLMs often lack the domain-specific reasoning required for clinical settings, struggle to understand subtle medical context, and are generally limited to text based inputs. Their performance is further constrained in tasks such as structured note generation, especially when applied to domains like dermatology. Figure 1 : Overview of the proposed framework, consisting of pre-training, fine-tuning, and inference phases. Existing methods for automated SOAP note generation, such as K-SOAP Li et al. 2024 , rely heavily on extensive doctor-patient dialogues and large-scale annotated datasets resources that are particularly limited in dermatology and skin lesion documentation Wei et al. 2024 . Moreover, capturing both the visual features of skin conditions and the underlying clinical reasoning in a structured format remains a major challenge. To address these limitations, we propose a novel weakly supervised multimodal framework that generates structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Unlike prior approaches focused solely on text-based SOAP generation or dermatologic diagnosis, our method uniquely integrates retrieval-augmented clinical knowledge, weak supervision, and multimodal synthesis to enable domain-aligned documentation without requiring large-scale annotations. We also introduce two novel metrics to evaluate clinical and semantic quality: MedConceptEval and Clinical Coherence Score (CCS), which go beyond traditional NLP metrics by assessing alignment with defined concepts and feature consistency. By leveraging domain-guided retrieval and pseudo-labeling, our framework produces clinically relevant SOAP notes with minimal supervision, offering a scalable solution for dermatology and broader healthcare applications. Our main contributions are: • We developed a weakly supervised multimodal framework for structured SOAP note generation from lesion images and limited clinical text. • We introduced two novel evaluation metrics: MedConceptEval and Clinical Coherence Score (CCS) to evaluate semantic alignment with clinical concepts and input consistency. • We performed statistical analysis using one-way ANOVA to quantify the effects of SOAP sections and lesion types on semantic similarity scores. • We conducted qualitative evaluation using an LLM-as-a-Judge framework (Flow-Judge-v0.1) to assess structure, readability, completeness, and clinical relevance. 2 Methodology We propose a three-phase weakly supervised multimodal framework for generating clinically structured SOAP notes from limited dermatologic inputs. As illustrated in Fig. 1 , our method consists of : (1) a pre-training phase to synthesize weakly supervised SOAP notes using generative captioning and retrieval-augmented knowledge integration, (2) a fine-tuning phase to adapt a vision-language model using the synthesized notes, and (3) an inference phase to generate high-quality structured SOAP notes from new patient data. 2.1 Pre-Training Due to the limited availability of large-scale annotated SOAP notes in dermatology, we employed a weak supervision strategy to synthesize training data. Each sample consists a lesion image paired with structured clinical features, including lesion diameter, biopsy status, and symptom descriptors.We first utilize GPT-3.5 Brown et al. 2020 to generate a clinical caption summarizing these structured attributes into a coherent description of the lesion. To improve the clinical relevance and factual reliability of the generated notes, we design a retrieval augmented generation framework Lewis et al. 2020 . The generated caption is used as a query to retrieve semantically relevant passages from a curated vector database. This database indexes document chunks extracted from authoritative medical sources, including the South Texas Skin Cancer Institute Skin Cancer Specialists 2024 , The National Cancer Institute National Cancer Institute 2024 , The American Cancer Society American Cancer Society 2024 , and The UK’s National Health Service NHS 2024 , covering lesion types, diagnostic criteria, symptomatology, and treatment guidelines. The retrieved context is concatenated with the original caption and provided as input to the pre-trained Vision-LLaMA 3.2 model, guided by a structured prompting template that encourages SOAP format outputs. This design addresses common limitations of pre-trained language models, such as outdated knowledge and hallucinated reasoning, and enables the model to generate reliable and clinically grounded, weakly supervised SOAP notes. 2.2 Fine-Tuning We fine-tuned the Vision-LLaMA 3.2 model using the synthesized weakly supervised dataset, where the lesion image and generated caption are treated as multimodal inputs, and the weakly supervised SOAP note serves as the training target. The model is optimized to produce structured outputs following the standard SOAP note format illustrated in Figure 2 in Section 5 of the Appendix. To improve training efficiency and maintain performance, we apply two fine-tuning strategies: Parameter-Efficient Fine-Tuning (PEFT) using QLoRA, and Supervised Fine-Tuning (SFT) using weakly supervised input-output pairs. Detailed training setup and method variants (PEFT and SFT) are provided in the Appendix 7 . 2.3 Inference At inference time, the fine-tuned Vision-LLaMA model receives a lesion image along with its corresponding clinical features, which were first converted into a clinical caption. The model then generates a structured SOAP note. Because the model has been fine-tuned on weakly supervised yet clinically reliable data, it generalizes effectively to new cases. This enables scalable and structured generation of dermatology SOAP notes, even in environments where expert annotations are limited or unavailable. 3 Evaluation We evaluated the generated SOAP notes using both quantitative and qualitative methods. For quantitative evaluation we compare against annotated ground truth using standard and clinical domain NLP metrics. Additionally, we introduced

Read the complete text