Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL

Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, Rossella Arcucci · 2025

Read Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL by Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, Rossella Arcucci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17952.txt.

Introduction

Che Liu and eight co-authors ask whether a medical language model can learn to reason without copying worked examples from a closed model. In a 2025 paper they present AlphaMed, a model trained with reinforcement learning alone, guided by simple rule-based rewards on public multiple-choice medical questions.

AlphaMed topped six medical question benchmarks, beating models trained with the usual mix of supervised fine-tuning and reinforcement learning. On the hardest test set it even passed much larger closed models. The team also studied what drives the gains and found that how much a data set teaches matters most. They close by calling for harder medical benchmarks that test reasoning more directly.

Catalog record

CREATORChe Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, Rossella Arcucci
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, Rossella Arcucci Abstract Improving performance on complex tasks and enabling interpretable decision making in large language models (LLMs), especially for clinical applications, requires effective reasoning. Yet this remains challenging without supervised fine-tuning (SFT) on costly chain-of-thought (CoT) data distilled from closed-source models (e.g., GPT-4o). In this work, we present AlphaMed , the first medical LLM to show that reasoning capability can emerge purely through reinforcement learning (RL), using minimalist rule-based rewards on public multiple-choice QA datasets, without relying on SFT or distilled CoT data. AlphaMed achieves state-of-the-art results on six medical QA benchmarks, outperforming models trained with conventional SFT+RL pipelines. On challenging benchmarks (e.g., MedXpert), AlphaMed even surpasses larger or closed-source models such as DeepSeek-V3-671B and Claude-3.5-Sonnet. To understand the factors behind this success, we conduct a comprehensive data-centric analysis guided by three questions: (i) Can minimalist rule-based RL incentivize reasoning without distilled CoT supervision? (ii) How do dataset quantity and diversity impact reasoning? (iii) How does question difficulty shape the emergence and generalization of reasoning? Our findings show that dataset informativeness is a key driver of reasoning performance, and that minimalist RL on informative, multiple-choice QA data is effective at inducing reasoning without CoT supervision. We also observe divergent trends across benchmarks, underscoring limitations in current evaluation and the need for more challenging, reasoning-oriented medical QA benchmarks. The code and pretrained model weights will be publicly released upon acceptance. 1 Introduction Recently, the reasoning capabilities of large language models (LLMs) have advanced significantly, achieving impressive results in tasks requiring complex reasoning, such as mathematical problem solving, code generation, and general-purpose benchmarks [ 1 , 2 , 3 , 4 ] . These developments highlight the potential of LLMs to generalize and perform multi-step reasoning across domains. In the medical domain, reasoning is particularly crucial. Clinical natural language processing (NLP) tasks often require interpreting nuanced patient information, integrating knowledge from diverse sources, and making informed decisions [ 5 , 6 , 7 ] . More importantly, reasoning provides a valuable lens into the model’s decision-making process, allowing researchers and clinicians to examine how conclusions are derived. This improves the interpretability and transparency of AI outputs, which are essential for clinical trust [ 8 , 9 ] . Currently, most medical LLMs acquire reasoning capabilities through supervised fine-tuning (SFT) on chain-of-thought (CoT) datasets, often followed by reinforcement learning (RL) for further refinement. However, this pipeline heavily relies on an initial SFT stage using costly CoT data, which are either manually crafted or distilled from closed-source commercial models such as GPT-4o [ 10 , 11 ] . This dependence not only incurs substantial annotation and distillation costs but also introduces scalability and accessibility challenges, as it ties model development to expensive and external resources. These limitations motivate a critical question: Can we achieve medical reasoning through minimalist rule-based RL without relying on distilled CoT data? To address this question, we propose AlphaMed , the first work designed to incentivize reasoning capability solely through minimalist rule-based RL, going beyond conventional approaches that rely on SFT with CoT data. Instead of depending on distilled CoT data supervision, AlphaMed is trained directly via simple rule-based rewards derived from multiple-choice QA datasets. Our key contributions are as follows: • We show that minimalist rule-based RL can incentivize reasoning ability in medical LLMs without relying on distilled CoT data, achieving superior performance. We further analyze how dataset quantity, diversity, and especially informativeness impact reasoning performance. We empirically find that higher informativeness enhances reasoning performance, while less-informative data limits gains. • We show that reasoning can be incentivized even with lower-difficulty data and further enhanced by harder examples. While high-difficulty samples benefit challenging benchmarks like MedXpert, a mix of difficulty levels is essential for robust generalization. Non-monotonic trends across benchmarks suggest that current evaluations may be insufficient to assess medical LLM reasoning. • Building on these insights, we introduce AlphaMed, a medical LLM trained solely via minimalist rule-based RL without any SFT on distilled CoT data, and demonstrate that it achieves state-of-the-art performance across six mainstream medical QA benchmarks, outperforming models that use complex training strategies with CoT data and even surpassing larger or closed-source models such as DeepSeek-V3-671B and GPT-4o. 2 Related Work Supervised Fine-Tuning for Reasoning in LLMs. Large language models can acquire complex reasoning skills through SFT on CoT data. For example, [ 12 ] showed that training models to generate step-by-step reasoning paths significantly improves performance on math and logic problems. [ 13 ] scaled this approach by incorporating a broad range of CoT examples into instruction tuning across diverse tasks. [ 14 ] proposed STaR, where a model bootstraps its own reasoning traces to reduce reliance on human-annotated CoT. However, recent work [ 15 ] suggests that SFT often encourages memorization of training rationales rather than true reasoning generalization, limiting robustness in out-of-distribution or unfamiliar tasks. Moreover, obtaining high-quality CoT data is costly, requiring either expert annotations or distillation from proprietary models, posing significant challenges to scalability and adaptability [ 16 ] . Reinforcement Learning with Preference Data after SFT. InstructGPT [ 17 ] introduced reinforcement learning with human preferences (RLHF) to align model behavior with user intent. Subsequent research has shown that RL can enhance generalization [ 16 , 18 ] and better capture nuanced human preferences beyond rote memorization [ 15 ] . Among RL algorithms, Proximal Policy Optimization (PPO) is widely used, but it is highly resource-intensive—requiring learned reward models that are often sensitive to noise, difficult to interpret, and occasionally misaligned with intended objectives [ 19 ] . To address these limitations, Direct Preference Optimization (DPO) [ 19 ] eliminates the need for an explicit reward model by directly optimizing over preference pairs. However, DPO still relies on high-quality preference annotations, which are particularly challenging to construct in the medical domain due to clinical ambiguity and a lack of universal agreement on what constitutes a “better” response [ 20 ] . Recently, DeepSeek-R1-Zero [ 21 ] demonstrated that reasoning behavior can be effectively elicited without CoT supervision or preference annotations, instead by leveraging final answers (e.g., multiple-choice accuracy) as rule-based supervision signals [ 16 , 18 , 22 , 23 ] . Open-Source Medical LLMs. Open-source medical LLMs have emerged as promising tools for domain-specific clinical reasoning, yet most remain heavily dependent on supervised data or handcrafted feedback. HuatuoGPT [ 24 ] was instruction-tuned on ChatGPT-distilled medical dialogues. BioMistral [ 25 ] adapted the Mistral architecture to biomedical question answering through continued pretraining [ 26 ] and domain-specific instruction tuning. OpenBioLLM [ 20 ] and UltraMedical [ 27 ] utilized DPO-based preference optimization, but their preference pairs were directly distilled from closed-source models, making supervision ambiguous and potentially inconsistent with expert clinical reasoning. Since human verification of each distilled example is prohibitively costly and impractical, there is no guarantee that the reasoning process reflected in the supervision is valid. HuatuoGPT-o1 [ 28 ] further incorporated PPO using a self-trained 3B reward model and relied on CoT data distilled from OpenAI o1. However, this approach is resource-intensive and tightly coupled to

Read the complete text