When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma · 2026

Read When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries by Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28332.txt.

Introduction

Yige Li and seven colleagues built MedHarm, a benchmark that stress-tests language models on dangerous medical questions. Models now answer health questions daily, yet their behavior on high-risk prompts, where a helpful tone could cause real harm, was barely measured. MedHarm holds eleven hundred grounded queries across ten safety-critical categories, from toxicology to fetal harm.

Li's team evaluated fifteen language models plus four guardrail systems. Passing ordinary alignment checks did not guarantee medical safety: aligned models still produced unsafe, actionable answers. Medical fine-tuning sometimes made harmful answers more specific and therefore worse. Guardrails blocked some failures but also refused clumsily and offered weak safe alternatives. Domain-specific stress testing, they conclude, is essential before deploying models in safety-critical medicine.

Catalog record

CREATORYige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28332

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma Abstract Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood. We introduce MedHarm 1 1 1 Code and data will be released upon acceptance. Due to the sensitive nature of high-risk medical queries, data access will be available to qualified researchers upon request. , a high-risk medical safety benchmark with 1,100 medically grounded queries across 10 safety-critical categories, including toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm. Unlike broad medical QA benchmarks, MedHarm targets realistic clinical, educational, and technical prompts that require refusal, caution, or safe redirection rather than direct helpfulness. We evaluate 15 LLMs spanning general-purpose, medical-purpose, closed-source, and downstream SFT models, together with 4 representative guardrail models. Results reveal a substantial gap between apparent alignment and medical safety: aligned models can still produce unsafe or actionable responses, medical fine-tuning can amplify harmful specificity, and external guardrails reduce some failures while introducing brittle blocking and weak safe helpfulness. These findings show that medical safety cannot be inferred from general alignment or medical capability alone, highlighting the need for domain-specific stress testing before deploying LLMs in safety-critical medical applications. Table 1: Positioning relative to representative LLM and medical safety benchmarks. Existing benchmarks primarily test broad safety, medical capability, or adversarial robustness; our benchmark focuses on high-stakes medical prompts where medically actionable details can cause physical harm. Benchmark Primary focus Medical grounding High-risk actionable guidance SafetyBench ( 32 ) General safety understanding No Indirect Do-Not-Answer ( 25 ) General refusal safeguards No Indirect ClinicBench ( 16 ) Clinical decision-making Yes Limited MedSafetyBench ( 9 ) Medical safety principles Yes Medium CARES ( 4 ) Medical adversarial safety Yes Medium–High MedHarm (Ours) Medical safety-alignment stress test Yes Very High 1 Introduction Large language models (LLMs) are increasingly entering medical and health-related settings, where their outputs may influence drug-related decisions, poisoning responses, triage suggestions, and other safety-critical judgments ( 22 ; 14 ; 2 ; 15 ) . This growing adoption amplifies not only their potential utility, but also their failure cost. Prior work shows that LLMs can answer medical licensing questions, support patient-facing communication, and approach expert-level performance on some medical QA tasks. Yet these capabilities also raise concerns about factual reliability, unsafe advice, and clinical over-reliance ( 14 ; 2 ; 22 ; 5 ) . In open-domain applications, unsafe responses are often treated as undesirable model behavior. In medical contexts, however, they can become unsafe guidance, misleading reassurance, or actionable instructions that may directly contribute to physical harm. A detailed answer about dosage, toxic exposure, or drug interactions is not merely a benign mistake when the stakes involve real patients, real emergencies, and real-world consequences ( 15 ; 18 ; 5 ) . Figure 1: Use-case motivation. In doctor-facing or patient-facing LLM use, medically plausible queries can still trigger unsafe, over-specific, or actionable guidance, creating safety risks that generic alignment may not reveal. To mitigate harmful behavior, modern LLM systems commonly rely on instruction tuning, safety alignment, domain-specific fine-tuning, and guardrail-based protection ( 19 ; 3 ; 11 ; 28 ) . These mechanisms are often assumed to make models safer for deployment, and in many settings this assumption appears reasonable. Existing safety evaluations have made substantial progress in measuring general harmfulness, refusal behavior, and adversarial robustness ( 31 ; 24 ; 17 ) . However, they rarely test the setting that is most consequential for medical deployment: whether a model that appears aligned under standard evaluations remains safe when facing medically grounded, high-stakes, and realistically phrased dangerous queries. This leads to a central question: do current alignment, fine-tuning, and guardrail mechanisms truly make LLMs safe in high-risk medical scenarios? This question is challenging because high-risk medical prompts are not simply another subtype of generic unsafe requests. Figure 1 illustrates this deployment risk, where medically plausible queries can lead to unsafe advice or missed guardrail interventions unless the system provides safe refusal and clinical redirection. A query about toxic thresholds, adverse drug interactions, or emergency response may appear medically plausible while still eliciting dangerous and actionable guidance. Such prompts fall between two regimes: they are too medically realistic for simple keyword filtering, yet too safety-critical to be treated as ordinary medical QA. Our results show that even strong aligned LLMs, e.g., GPT-5.5 and DeepSeek-V4-Pro, can still produce unsafe or actionable responses under clinically plausible harmful framing. To study this problem, we introduce MedHarm , a targeted stress-test benchmark for high-risk medical safety rather than a general medical knowledge benchmark. MedHarm contains 1,100 medically grounded queries across 10 clinically relevant risk categories and evaluates whether models can refuse harmful details, avoid actionable instructions, and redirect users toward safe clinical behavior. We apply MedHarm to a broad set of LLM systems, including general instruction-tuned models, medically specialized models, downstream SFT variants, closed-source systems, and guardrail-protected pipelines. Our results reveal three consistent findings. First, general-purpose alignment is not sufficient for medical safety: unsafe response rates vary widely even among instruction-tuned systems. Second, medical specialization does not reliably improve safety and can increase actionable harm when domain expertise is added without equally strong safety constraints. Third, guardrails reduce some failures, but often trade safety for safe helpfulness: generic refusal may be acceptable in broad safety settings, but is insufficient when safe medical advice or clinical redirection is needed. In summary, our main contributions are as follows: • We introduce MedHarm , a 1,100-query benchmark for high-risk medical safety, covering 10 medically grounded categories that require refusal, caution, or safe redirection rather than generic helpfulness. • We evaluate a broad set of model and protection settings on MedHarm , including general instruction-tuned models, medical fine-tunes, downstream SFT variants, closed-source systems, and guardrail models. • We show that alignment, medical fine-tuning, and guardrails exhibit distinct failure modes under realistic high-risk medical prompts, motivating domain-specific safety evaluation before medical deployment. 2 Related Work Medical Safety Benchmarks and Evaluation. Existing LLM safety and medical evaluation benchmarks cover several related but distinct directions. General safety benchmarks evaluate harmfulness, refusal behavior, and safeguard robustness across broad risk categories ( 32 ; 25 ) , while medical LLM benchmarks mainly assess clinical knowledge, reasoning, diagnosis, or decision support ( 26 ; 30 ; 1 ; 27 ; 16 ) . More recent medical safety benchmarks move closer to our setting by evaluating safety principles, adversarial prompts, and safety–effectiveness trade-offs in clinical domains ( 9 ; 4 ; 23 ) . However, these benchmarks generally do not isolate the specific failure mode studied in this work: medically realistic prompts that elicit actionable details capable of causing physical harm. Table 1 summarizes how our benchmark differs from representative prior evaluations. Alignment, Fine-Tuning, and Guardrails for Medical Models. Instruction tuning and reinforcement learning with human feedback (RLHF) have improved the helpfulness and harmlessness of general-purpose assistants ( 19 ; 3 ) , but such alignment does not necessarily transfer to high-risk medical contexts. Prior work shows that downstream fine-tuning can weaken safety alignment even when the training data is not overtly

Read the complete text