Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise

Hanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun · 2024

Read Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise by Hanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12583.txt.

Introduction

Hanyin Wang and co-authors train a step by step verifier for AI written clinical notes in a 2024 paper. Reward models that grade each reasoning step shine in math, where answers check out, but clinic notes have no answer key. The team defines meaningful note steps, injects realistic expert style errors, and lets models mass produce the grading data.

Their verifier, built on an open eight billion parameter model, beat proprietary rivals at two jobs. It spotted which notes held planted errors with near ninety nine percent accuracy. It also picked the note doctors preferred at fifty six percent. A reader study with physicians mapped what makes the verifier useful downstream. The recipe, they argue, travels to other fields without ground truth.

Catalog record

CREATORHanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12583

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise Hanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun Abstract Process-supervised reward models (PRMs) excel at providing step-by-step verification for large language model (LLM) outputs in domains like mathematics and coding. However, their application to fields lacking ground-truth answers, such as clinical note generation, poses significant challenges. We introduce a novel framework for training PRMs to deliver step-level reward signals for LLM-generated clinical notes. By precisely defining meaningful “ steps ,” injecting realistic “ errors ” informed by domain expertise, and leveraging LLMs to generate process supervision data at scale, we overcome previous limitations. Our PRM, built on LLaMA-3.1 8B, consistently outperforms proprietary reasoning and non-reasoning models , achieving state-of-the-art (SOTA) performance on two key evaluations: (1) distinguishing gold-standard from error-containing samples with 98.8% accuracy, and (2) selecting physician-preferred clinical notes with 56.2% accuracy. We investigate critical components for effective PRM training, including optimal loss functions and data selection strategies, and present a comprehensive physician reader study identifying predictors of downstream Best-of-N performance. Our study sheds light on unlocking the potential of PRMs for diverse generative tasks across domains. 1 1 1 Our code is available at https://github.com/hanyin88/prm-clinic . 1 Introduction Figure 1: Overview of study design. We train a PRM to step-by-step verify clinical notes generated by LLMs from patient-doctor dialogues. Step 1 : Clinical notes are transformed into a hierarchical structure of steps, designed based on domain expertise to capture the key considerations in clinical documentation. Step 2 : Gemini Pro 1.5 is utilized to generate synthetic errors from predefined categories. These errors are systematically swapped with the original steps to create negative samples. Step 3 : The PRM is trained using LLaMA-3.1 8B instruct, followed by ablation studies and a physician reader study. LLMs show promise in generating clinical notes Van Veen et al. (2024) , but their outputs may contain errors and misalign with physician preferences Omiye et al. (2024) ; Jin et al. (2024) . Currently, no automated, scalable method exists to evaluate the quality of LLM-generated clinical notes, leaving manual evaluation the gold standard. Consequently, costly clinician reader studies are required to validate LLM-based products, such as ambient AI scribes Liu et al. (2024) . This challenge is amplified by the rapid adoption of ambient scribing technology in the real world Tierney et al. (2025) , with projections indicating that 30% of the healthcare market will use LLM-generated clinical notes by 2025 Beavins (2024) . A potential solution to this challenge is to verify LLM outputs using a reward model (RM). RMs are integral to reinforcement learning for LLM post-training Ouyang et al. (2022) ; Stiennon et al. (2020) . When employed as verifiers, RMs are categorized into two types: outcome-supervised reward models (ORMs) Cobbe et al. (2021) and, more recently, process-supervised reward models (PRMs) Lightman et al. (2023) . ORMs evaluate an entire generation and assign a single reward score at the end, whereas PRMs provide reward scores at each step of the generation, offering several notable advantages. PRMs enable fine-grained verification of LLM outputs, improving explainability by pinpointing errors at their exact location. This step-wise reward signaling also facilitates inference-time scaling, such as Monte Carlo Tree Search (MCTS), and supports step-by-step reinforcement learning Wang et al. (2023b) ; Snell et al. (2024) . However, despite significant success in mathematical and coding tasks Lightman et al. (2023) ; Wang et al. (2023b) ; Ma et al. (2023) , the application of PRMs to other domain remains underexplored Beeching et al. () . This gap arises from several key challenges: (1) Verification of correctness : Unlike mathematical or coding problems, which have objective ground-truth answers, verifying correctness in open-ended generative tasks is inherently more complex. (2) Process-supervised data collection : Training PRMs requires datasets with annotations for each step of the answer, which makes it difficult to scale when relying on human annotators. To address these challenges, we introduce a novel framework for training PRMs to verify open-ended text generation in the clinical domain (Figure 1 ). We specifically target the rapidly growing use case of ambient scribing , where LLMs generate clinical notes from patient-doctor dialogues Barr et al. (2024) . Drawing on domain expertise, we carefully design clinically meaningful “steps” and inject realistic “errors” to reflect authentic documentation practices. To scale supervision, we employ an LLM to generate process-labeled data by systematically introducing errors into high-quality reference notes. Our main contributions are as follows: 1. Our PRM, built on LLaMA-3.1 8B, significantly outperforms both proprietary reasoning and non-reasoning models for clinical note verification. It achieves 98.8% accuracy in detecting erroneous samples and 56.2% accuracy in selecting physician-preferred notes, setting a new SOTA in this critical domain. 2. We broaden the scope of PRMs beyond domains with clear ground-truth answers, introducing a generalizable framework that integrates LLMs with domain knowledge to generate process-supervised data at scale. 3. To support future research, we release our full dataset, including process supervision annotations and physician preference labels. 2 Background and Related Work Verify LLM Outputs Step-by-Step Recent research highlights a growing interest in process supervision of LLMs to improve their reasoning capabilities. In contrast to earlier work on ORMs Cobbe et al. (2021) ; Yu et al. (2024) , PRMs assess the correctness of LLM outputs step-by-step Lightman et al. (2023) ; Uesato et al. (2022) . One of the core applications of PRMs is Best-of-N sampling, where N complete solutions are generated, and the most suitable one is selected based on the PRM’s reward signal. This method has proven exceptionally effective in mathematical reasoning and code generation tasks, frequently surpassing ORMs Wang et al. (2023b) ; Ma et al. (2023) . Furthermore, PRMs can facilitate step-level reinforcement learning and guide inference-time scaling through techniques such as MCTS Wang et al. (2023b) ; Ma et al. (2023) ; Snell et al. (2024) . They are considered crucial to the success of advanced reasoning models Jiang et al. (2024) ; Qin et al. (2024) ; Wang et al. (2024b) . Process-Supervised Data Acquiring human-labeled annotations for each step of PRM training data poses a prohibitive scaling cost, a challenge central to our work. To address this limitation, several studies have proposed methods to automatically generate process supervision data, using techniques such as Monte Carlo estimation Wang et al. (2023b) ; Luo et al. (2024) ; Wang et al. (2024c) . However, these approaches are primarily confined to the mathematical domain, where they are effective due to the unambiguous and easily verifiable nature of mathematical solutions. Our exploration of PRM is also related to prior work on evaluating step-by-step reasoning chains Hao et al. (2024) , though our focus is on open-ended text generation tasks, which present unique challenges. Generate Clinical Note From Dialogue Ambient scribe systems that transcribe patient-provider conversations into clinical notes are among the most impactful applications of LLMs in healthcare, offering substantial reductions in documentation burden Barr et al. (2024) ; Tierney et al. (2025) . This task was explored at the 2023 ACL ClinicalNLP and CLEFImage workshops, where the best results were

Read the complete text