Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
Yuxuan Zhou, Yubin Wang, Bin Wang, Chen Ning, Xien Liu, Ji Wu, Jianye Hao · 2025
Read Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning by Yuxuan Zhou, Yubin Wang, Bin Wang, Chen Ning, Xien Liu, Ji Wu, Jianye Hao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10067.txt.
Introduction
Yuxuan Zhou and six co-authors teach medical language models to notice what a patient message leaves out. Models score well on medical benchmarks yet underperform in real consultations, where safe answers depend on context such as identity, history, and risk factors. Their method, MuSeR, trains context-awareness across decision making, communication, and safety through self-evaluation and refinement.
A query generator first simulates varied users by changing role, region, intent, and ambiguity. The model answers, grades its own answers on the three facets, refines them, and then fine-tunes on the improved pairs. On the HealthBench dataset the gains were largest exactly on context-awareness, and with added knowledge distillation a smaller model surpassed its teacher, setting a new best among open-source systems.
Catalog record
| CREATOR | Yuxuan Zhou, Yubin Wang, Bin Wang, Chen Ning, Xien Liu, Ji Wu, Jianye Hao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10067
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning Yuxuan Zhou, Yubin Wang, Bin Wang, Chen Ning, Xien Liu, Ji Wu, Jianye Hao Abstract Large language models (LLMs) have shown great promise in the medical domain, achieving strong performance on several benchmarks. However, they continue to underperform in real-world medical scenarios, which often demand stronger context-awareness, i.e., the ability to recognize missing or critical details (e.g., user identity, medical history, risk factors) and provide safe, helpful, and contextually appropriate responses. To address this issue, we propose Multifaceted Self-Refinement (MuSeR), a data-driven approach that enhances LLMs’ context-awareness along three key facets (decision-making, communication, and safety) through self-evaluation and refinement. Specifically, we first design a attribute-conditioned query generator that simulates diverse real-world user contexts by varying attributes such as role, geographic region, intent, and degree of information ambiguity. An LLM then responds to these queries, self-evaluates its answers along three key facets, and refines its responses to better align with the requirements of each facet. Finally, the queries and refined responses are used for supervised fine-tuning to reinforce the model’s context-awareness ability. Evaluation results on the latest HealthBench dataset demonstrate that our method significantly improves LLM performance across multiple aspects, with particularly notable gains in the context-awareness axis. Furthermore, by incorporating knowledge distillation with the proposed method, the performance of a smaller backbone LLM (e.g., Qwen3-32B) surpasses its teacher model, achieving a new SOTA across all open-source LLMs on HealthBench (63.8%) and its hard subset (43.1%). Code and dataset will be released at https://muser-llm.github.io . 1 Introduction Large language models (LLMs) have witnessed significant advancements in recent years ( 3 ; 4 ; 18 ; 27 ; 42 ) and demonstrated promising capabilities in various domains, including the medical domain. Recent studies ( 37 ; 38 ; 30 ; 34 ; 29 ) indicate that current LLMs (e.g., GPT-4) encode substantial medical knowledge and achieves strong performance on several medical benchmarks. Despite these advancements, LLMs still struggle to meet the demands of real-world medical applications, limiting their practical utility in healthcare settings. One of the decisive differences between medical benchmark questions and real-world scenarios lies in the requirement for stronger context-awareness , namely, the ability to recognize missing or critical details (e.g., medical history, user identity, risk factors) and to provide safe, helpful, and contextually appropriate responses. As illustrated in Figure 1 a, existing medical benchmarks typically adopt question-answering task for evaluation, where questions contain sufficient information for answering, presented in the impersonal tone, and answering errors have limited consequences. In contrast, real-world medical scenarios often omit key details for decision-making, involve diverse user roles (e.g., patients, doctors), and require careful consideration of safety and ethical implications. While current LLMs perform well on exam-style context, they often overlook the contextual factors in real-world medical scenarios, leading to responses that may be inappropriate, unsafe, or unhelpful to the user’s specific situation. Figure 1: (a) Comparison between medical exam questions and real-world medical scenarios. (b) The proposed Multifaceted Self-Refinement learning framework ( MuSeR ) to enhance the medical context-awareness ability of LLMs through data synthesis and self-refinement. In this paper, we aim to enhance the context-awareness of LLMs in the medical domain. A common approach is to collect high-quality real-world medical conversations for supervised fine-tuning (SFT). However, this is often impractical due to high collection costs and ethical concerns. To address this, we explore a cost-effective and scalable alternative: enhancing context-awareness through data synthesis. Specifically, we propose a novel Mu ltifaceted Se lf- R efinement ( MuSeR ) framework. MuSeR improves medical context-awareness by synthesizing simulated real-world medical queries and generating context-aware responses by self-refining the answers of LLMs along three key facets of context-awareness: decision-making, communication, and safety. As shown in Figure 1 b, the proposed framework consists of three main components: (1) a attribute-conditioned query generator that simulates diverse real-world user contexts by varying attributes such as role, geographic region, intent, and degree of information ambiguity; (2) a multifaceted self-refinement module where an LLM responds to the generated queries, evaluates its answers along the three key facets, and refines its responses to better align with the requirements of each facet; and (3) a supervised fine-tuning stage where the generated queries and refined responses are used to reinforce the model’s context-awareness ability. The entire process does not require any external medical corpora or human annotations, making it a cost-effective and scalable solution for enhancing LLMs’ context-awareness in the medical domain. To evaluate the effectiveness of our proposed method, we apply the proposed method on different sizes of LLMs (Qwen3-32B, Qwen3-14B, OpenPangu-7B) and assess their performance on the latest HealthBench dataset ( 5 ) , which focuses on evaluating LLMs performance in real-world medical scenarios. The results demonstrate that our method significantly improves LLM performance on HealthBench, with particularly notable gains in the context-awareness axis. Furthermore, by incorporating knowledge distillation into the proposed framework using a strong teacher model (e.g., GPT-oss-120B), the performance of a smaller backbone LLM (e.g., Qwen3-32B) surpasses that of the teacher model by 6%, achieving a new state-of-the-art result among open-source LLMs on HealthBench ( 63.8 %) and its hard subset ( 43.1 %). Our main contributions are summarized as follows: • We propose a novel Multifaceted Self-Refinement ( MuSeR ) learning framework that enhances LLMs’ context-awareness across three key facets (decision-making, communication, and safety) through self-evaluation and refinement, facilitating their application in real-world medical scenarios. • Extensive experiments on the HealthBench dataset demonstrate the effectiveness of our method in improving LLM performance, particularly in the context-awareness axis. • By incorporating knowledge distillation into our framework, we achieve new state-of-the-art performance among open-source LLMs on the HealthBench dataset ( 63.8 %) and the hard subset (43.1 %) using only 100k generated queries. 2 Related Work LLM Medical Evaluation Most of existing medical benchmarks for LLMs are in the question-answering form, where the questions are sourced from medical exams ( 40 ; 23 ; 33 ; 7 ; 41 ; 34 ; 46 ) , literatures ( 24 ; 26 ) , and healthcare consultations ( 28 ; 1 ; 37 ) . These benchmarks primarily evaluate the LLMs’ medical knowledge and reasoning abilities, and existing LLMs are reported to achieve strong performance on these benchmarks ( 37 ; 38 ; 31 ; 34 ) . For example, GPT-4 achieves over 90% accuracy on the MedQA-USMLE exam dataset, approaching the performance level of human medical experts. Nevertheless, these benchmarks may not fully capture the complexities of real-world medical scenarios, especially in terms of context-awareness. Recently, OpenAI proposed HealthBench ( 5 ) , a new benchmark includes 5,000 realistic health conversations annotated by 262 physicians across 60 countries, evaluating LLMs’ performance as medical assistants in real-world scenarios. In this work, we primarily evaluate the effectiveness of our method on the HealthBench dataset. Medical LLM Training Existing works on training medical LLMs mainly focus on two aspects: (1) continual pre-training on medical corpora to inject domain-specific knowledge into LLMs ( 11 ; 34 ; 44 ) ; (2) post training on downstream tasks to enhance the model’s reasoning and decision-making capabilities ( 37 ; 38 ; 39 ; 13 ; 12 ; 9 ) . For the training data, most works