Learning Preference-Based Objectives from Clinical Narratives for Sequential Treatment Decision-Making
Daniel J. Tan, Kay Choong See, Mengling Feng * · 2026
Read Learning Preference-Based Objectives from Clinical Narratives for Sequential Treatment Decision-Making by Daniel J. Tan, Kay Choong See, Mengling Feng * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10783.txt.
Introduction
Tan and five colleagues propose CN-PR, learning reward functions for sepsis treatment from clinical narratives rather than sparse outcome metrics. Discharge notes encode clinician views of progression, treatment success, and recovery, giving trajectory supervision. A large language model scores trajectory quality and builds pairwise preferences between patient paths, with a relevance signal weighting supervision by task. In offline reinforcement learning for sepsis treatment, the learned reward tracks trajectory quality and yields policies with more organ support-free days and faster shock resolution, while mortality stays comparable to baselines.
Catalog record
| CREATOR | Daniel J. Tan, Kay Choong See, Mengling Feng * |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77647 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10783
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Preference-Based Objectives from Clinical Narratives for Sequential Treatment Decision-Making Daniel J. Tan, Kay Choong See, Mengling Feng * Abstract Designing reward functions remains a central challenge in reinforcement learning (RL) for healthcare, where outcomes are sparse, delayed, and difficult to specify. While structured data capture physiological states, they often fail to reflect the overall quality of a patient’s clinical trajectory, including recovery dynamics, treatment burden, and stability. Clinical narratives, in contrast, summarize longitudinal reasoning and implicitly encode evaluations of treatment effectiveness. We propose Clinical Narrative-informed Preference Rewards (CN-PR), a framework for learning reward functions directly from discharge summaries by treating them as scalable supervision for trajectory-level preferences. Using a large language model, we derive trajectory quality scores (TQS) and construct pairwise preferences over patient trajectories, enabling reward learning via a structured preference-based objective. To account for variability in narrative informativeness, we incorporate a confidence signal that weights supervision based on its relevance to the decision-making task. The learned reward aligns strongly with trajectory quality (Spearman ρ = 0.63 \rho=0.63 ) and enables policies that are consistently associated with improved recovery-related outcomes, including increased organ support–free days and faster shock resolution, while maintaining comparable performance on mortality. These effects persist under external validation. Our results demonstrate that narrative-derived supervision provides a scalable and expressive alternative to handcrafted or outcome-based reward design for dynamic treatment regimes. 1 Introduction Clinical decision-making in intensive care units (ICUs) involves complex, time-dependent treatment strategies that must balance competing objectives under uncertainty. Management of critically ill patients often requires sequential adjustments to interventions such as fluid therapy, vasopressors, mechanical ventilation, sedation, and renal support. These decisions are inherently dynamic, motivating the use of data-driven approaches for learning dynamic treatment regimes (DTRs) from electronic health record (EHR) data. Reinforcement learning (RL) provides a principled framework for modeling such sequential decision-making problems, where policies are optimized to maximize long-term outcomes. Prior work has demonstrated the potential of RL in critical care settings, including applications in sepsis management, ventilation strategies, and hemodynamic control [ 16 , 24 , 25 ] . However, a central challenge remains reward specification, which defines the objective optimized by the learned policy. Most clinical RL approaches rely on simplified reward functions based on terminal outcomes (e.g., mortality) or a limited set of physiological proxies. While informative, these signals provide only a partial view of patient trajectories and fail to capture important aspects such as recovery dynamics, treatment burden, and clinical stability. Handcrafted intermediate rewards partially address this limitation but introduce heuristic assumptions and are highly sensitive to design choices, often leading to inconsistent or poorly aligned policy behavior [ 20 , 17 ] . More broadly, specifying reward functions in complex clinical environments remains fundamentally challenging. An alternative is to infer reward functions from preferences rather than explicitly defining them. Preference-based approaches learn from comparative evaluations of trajectories [ 4 ] , allowing reward functions to reflect relative outcome quality instead of fixed endpoints. However, existing methods typically rely on manually collected annotations, which are costly and difficult to scale in healthcare settings. In contrast, routine clinical practice produces a rich and largely untapped source of implicit preference signals: unstructured clinical narratives. Discharge summaries and clinical notes synthesize clinician assessments of treatment response, disease progression, and recovery [ 13 ] , providing high-level evaluations of patient trajectories that are not captured in structured data alone. These narratives can be viewed as naturally occurring supervision, where relative outcome quality is embedded in how patient courses are described. Recent advances in large language models (LLMs) enable the extraction of structured signals from clinical text at scale, making it feasible to operationalize such narratives for reward learning. Rather than relying on handcrafted objectives or explicit annotation, narrative-derived signals can serve as a scalable proxy for trajectory-level preferences, capturing broader aspects of clinical trajectories beyond predefined endpoints. In this work, we propose Clinical Narrative-informed Preference Rewards (CN-PR), a framework for learning reward functions from clinical narratives. We use an LLM to assign trajectory quality scores (TQS), which are used to construct pairwise preferences over patient trajectories. To account for variability in the informativeness of clinical narratives, we introduce a confidence signal that reflects the relevance of each summary to the target decision-making task, enabling the model to weight supervision by reliability. The resulting reward function is trained using a structured preference-based objective that incorporates both the magnitude of outcome differences and the confidence of the underlying supervision. This allows the learned reward to capture not only relative ordering of trajectories but also the strength and reliability of preference signals. We evaluate CN-PR in the context of sepsis treatment as a representative use case in critical care. Our results show that the learned reward aligns with clinically meaningful outcomes, supports the learning of policies associated with improved recovery metrics, and generalizes across datasets. These findings suggest that leveraging narrative-derived supervision provides a scalable and expressive approach to reward specification for RL-based dynamic treatment regimes. 2 Background and Related Work 2.1 Reinforcement Learning for Dynamic Treatment Regimes Reinforcement learning (RL) has been increasingly applied to dynamic treatment regimes (DTRs), where policies optimize sequential treatment decisions based on patient state [ 16 ] . Applications include sepsis management, mechanical ventilation, and sedation using retrospective EHR data [ 22 , 24 , 7 ] . In these settings, learning is performed offline, making policy behavior highly dependent on the reward function. Because clinical RL is typically performed in an offline setting without the ability to safely explore new actions, the learned policy is highly sensitive to the reward function used during training. As a result, reward specification plays a central role in determining both the behavior and clinical plausibility of learned policies. 2.2 Reward Learning and Preference-Based Methods Designing reward functions for clinical RL is challenging due to the multidimensional nature of clinical outcomes. Most approaches rely on terminal endpoints or handcrafted proxies, which introduce subjective assumptions and can lead to unstable policies [ 20 , 17 ] . In contrast, inverse reinforcement learning (IRL) aims to infer reward functions from observed behavior, providing a data-driven alternative to manual reward design [ 1 ] . In healthcare, IRL has been used to model clinician decision-making in settings such as sepsis treatment and mechanical ventilation [ 38 , 39 ] . However, classical IRL assumes that observed behavior is near-optimal, an assumption that is often violated in clinical practice due to variability in expertise, institutional protocols, and patient heterogeneity [ 34 , 3 ] . Preference-based IRL relaxes this assumption by learning from comparisons between trajectories rather than demonstrations alone, allowing reward learning from imperfect data [ 4 ] . While promising, these approaches require explicit preference annotations, which are difficult to obtain at scale in clinical domains. Conceptually, our work is also related to perspectives that treat reward functions as imperfect observations of underlying objectives rather than definitive specifications [ 8 ] . This view motivates learning reward functions from diverse sources of evidence, including implicit signals embedded in data. 2.3 Learning from Language and Clinical Narratives Recent work has shown that reward functions can be learned from human judgments over text, as in reinforcement learning from human feedback (RLHF) [ 27