CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis

Yunsung Chung, Alex El Darzi, Carlo El Khoury, Han Feng, Nassir Marrouche, Jihun Hamm · 2026

Read CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis by Yunsung Chung, Alex El Darzi, Carlo El Khoury, Han Feng, Nassir Marrouche, Jihun Hamm (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12650.txt.

Introduction

Chung and five colleagues adapt diffusion models to medical imaging with medical reward fit. Base models generate real-looking images, yet the medical move suffers limited tagged data and false making. Metrics like FID miss per-image medical fit. Their Clinical Alignment Score rates made images along matching sides beyond look, using a base model as proxy. Building on it, Clinical Reward-Aligned Finetuning (CRAFT) transfers medical knowledge from language and vision models through prompt boost, medical checklists and smooth reward tuning. It lifts fit and sorting across four types.

Catalog record

CREATORYunsung Chung, Alex El Darzi, Carlo El Khoury, Han Feng, Nassir Marrouche, Jihun Hamm
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82071
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis Yunsung Chung, Alex El Darzi, Carlo El Khoury, Han Feng, Nassir Marrouche, Jihun Hamm Abstract Foundation diffusion models can generate photorealistic natural images, but adapting them to medical imaging remains challenging. In medical adaptation, limited labeled data can exacerbate hallucination-like and clinically implausible synthesis, while existing metrics such as FID or Inception Score do not quantify per-image alignment with pathology-relevant criteria. We introduce the Clinical Alignment Score (CAS), a foundation-model-based proxy for clinical alignment that evaluates generated images along four complementary dimensions beyond visual fidelity. Building on CAS, we propose Clinical Reward-Aligned Finetuning (CRAFT), a reward-based adaptation framework that transfers medical knowledge from multimodal large language models and vision-language models through label-conditioned prompt enrichment, clinical checklists, and differentiable reward optimization. Across four diverse modalities, CRAFT improves CAS and downstream classification performance over strong adaptation baselines. Beyond average CAS gains, CRAFT reduces the empirical low-alignment tail below a real-image reference threshold by 5.5–34.7% points relative to the strongest baseline, corresponding to a 20.4% average relative reduction across datasets. These results indicate fewer hallucination-like generations under CAS, and are corroborated by out-of-family evaluator evaluation, structured checklist auditing, memorization analysis, and a blinded physician preference study on CheXpert. 1 Introduction Figure 1: Melanoma example. The baseline synthesis lacks clear pathology-specific cues, while CRAFT exhibits stronger alignment with the clinical criteria captured by CAS. Deep learning has demonstrated remarkable success in medical decision-making, including dermatology and radiology ( 3 ; 14 ; 7 ; 32 ; 41 ) . However, robust medical model training remains constrained by limited labeled data, privacy barriers, annotation cost, and long-tailed clinical distributions. Synthetic data generation has emerged as a partial remedy to medical data scarcity ( 10 ) . Diffusion models ( 21 ) have been adapted from general ( 16 ; 26 ; 30 ; 37 ; 50 ; 52 ) to medical domains, including chest X-ray, brain MRI, CT, and dermatology ( 2 ; 23 ; 25 ; 34 ; 18 ; 49 ; 45 ) . Synthetic data can also improve downstream utility or mitigate distribution shifts in some settings ( 9 ; 44 ; 28 ) . In medical imaging, visual realism is insufficient: a generated image can look plausible while missing the pathology-specific findings that make it clinically useful. Distribution-level metrics such as Fréchet Inception Distance (FID) ( 20 ) , Inception Score (IS) ( 38 ) , and Kernel Inception Distance ( 5 ) measure set-level visual fidelity, while LPIPS ( 53 ) and CLIPScore ( 19 ) measure perceptual or text-image alignment. None directly evaluates whether a generated image satisfies pathology-relevant criteria for a given clinical prompt (see Figure 1 ). To close this gap, we propose the Clinical Alignment Score (CAS), a foundation-model-based proxy for clinical alignment. CAS combines four primitives: Visual Description Consistency (VDC), Clinical Criteria Satisfaction (CCS), Diagnostic Discriminability (DD), and Semantic Feature Similarity (SFS). Unlike generic perceptual metrics, CAS targets clinically meaningful failure modes via text–image alignment, checklist satisfaction, diagnostic separability, and reference-anchored semantic similarity, and it requires no manual preference labels. CAS remains a proxy, not a substitute for expert review. Building on CAS, we introduce CRAFT (Clinical Reward-Aligned Finetuning), which turns the same clinical dimensions into differentiable training signals. CRAFT adapts a general-purpose diffusion model toward clinically aligned medical synthesis under limited supervision. Specifically, we use MLLMs (e.g., MedGemma ( 40 ) , GPT-4o ( 1 ) ) to expand sparse clinical labels into image-grounded descriptions and create disease-level clinical criteria, and also use VLMs (e.g., MedSigLIP ( 40 ) ) as a differentiable critic that supplies reward feedback during finetuning. Across four medical imaging datasets, CRAFT achieves higher CAS on held-out test cases than the compared adaptation baselines, and also improves real+synthetic downstream classification performance. On CheXpert, CRAFT samples are rated higher by medical experts, supporting improved clinical plausibility beyond simple memorization. While we focus on medical imaging, the framework should extend naturally to other domains with expert-defined semantic criteria—e.g., satellite imagery, materials science, or industrial inspection. Clinical alignment must be evaluated not only by average performance but also by failure tails. In medical image synthesis, even a small fraction of hallucination-like samples can inject misleading pathology cues into downstream training. We therefore define a low-alignment rate using a real-image reference threshold and analyze the lower tail of per-image CAS. Across four datasets, CRAFT reduces this low-alignment rate by 5.5–34.7% points relative to the strongest baseline, indicating that clinical reward alignment improves not only average quality but also tail reliability. Our contributions are threefold. (1) We introduce CAS, a decomposed foundation-model-based proxy for clinical alignment that combines VDC, CCS, DD, and SFS. (2) We propose CRAFT, a reward-aligned finetuning framework that transfers structured medical knowledge from MLLMs and VLMs into a diffusion model through prompt enrichment and differentiable reward optimization. (3) Across dermatology, radiology, histopathology, and retinal fundus imaging, CRAFT improves average CAS, reduces low-CAS failure rates, and improves downstream classification accuracy. We corroborate these gains with out-of-family evaluator analysis, checklist auditing, diversity analysis, and blinded physician preference study on CheXpert. Figure 2: Overview of the CRAFT Framework. The pipeline consists of two stages: (1) Semantic Enrichment uses an MLLM to generate an image-grounded enriched prompt ( t_{ep} ) from each image-label pair ( x_{i},y_{i} ) and a class-level checklist ( t_{c} ) from the label y_{i} . (2) Clinical Reward-Aligned Finetuning optimizes a pretrained diffusion model using a VLM critic, which computes differentiable rewards for visual description consistency ( r_{\textrm{vdc}} ), clinical criteria satisfaction ( r_{\textrm{ccs}} ), diagnostic discriminability ( r_{\textrm{dd}} ), and semantic feature similarity ( r_{\textrm{sfs}} ). Reward gradients are backpropagated through the final K denoising steps, updating only LoRA parameters for efficient training. 2 Related works Medical image synthesis. Latent diffusion models ( 36 ) have been adapted to medical imaging domains such as chest X-rays ( 2 ; 13 ; 23 ) , 3D brain MRI ( 25 ; 34 ) , and CT ( 18 ; 55 ) . Beyond high-fidelity, synthetic data can improve robustness in data-scarce regimes ( 45 ) , and mitigate demographic distribution shifts ( 28 ; 44 ) . To introduce semantic control, prior work adapts pretrained diffusion models using Textual Inversion ( 16 ) and/or LoRA ( 22 ) to inject disease concepts into pre-trained models ( 13 ; 44 ; 15 ) . However, optimizing for distributional realism or reconstruction fidelity alone does not guarantee pathology-specific correctness. Prior work has shown that medical image translation and synthesis systems can introduce hallucinated features when objectives fail to preserve clinically meaningful structure ( 12 ) . Reward and preference alignment. Diffusion model alignment increasingly uses reward or preference signals, extending ideas such as DPO ( 35 ) to images via offline preference learning and reward modeling ( 43 ; 47 ) . Existing methods include classifier/reward-model-based alignment ( 4 ; 31 ; 29 ; 43 ) and direct optimization through the denoising process ( 6 ; 11 ) . In medical imaging, 42 leverages direct pathologist feedback, while subsequent work explores automated feedback: 45 utilizes DPO to align skin lesion synthesis, and 39 optimizes against online classifiers but relies on discrete reward proxies. CRAFT differs

Read the complete text