This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
Hye Sun Yun, Geetika Kapoor, Michael Mackert, Ramez Kouzy, Wei Xu, Junyi Jessy Li, Byron C. Wallace · 2026
Read This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA by Hye Sun Yun, Geetika Kapoor, Michael Mackert, Ramez Kouzy, Wei Xu, Junyi Jessy Li, Byron C. Wallace (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05051.txt.
Introduction
Yun and six co-authors evaluate whether large language models answer medical questions consistently when patients phrase them differently. Patients ask language models complex questions that are hard to articulate, yet models are sensitive to phrasing and can be swayed by wording. Models should respond consistently regardless of phrasing when grounded in the same evidence. They test this in a retrieval-augmented generation setting for medical QA, using expert documents rather than retrieval. They examine question framing from positive to negative and language style from technical to plain, across 6,614 questions.
Catalog record
| CREATOR | Hye Sun Yun, Geetika Kapoor, Michael Mackert, Ramez Kouzy, Wei Xu, Junyi Jessy Li, Byron C. Wallace |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52194 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA Hye Sun Yun, Geetika Kapoor, Michael Mackert, Ramez Kouzy, Wei Xu, Junyi Jessy Li, Byron C. Wallace Abstract Patients are increasingly turning to large language models (LLMs) with medical questions that are complex and difficult to articulate clearly. However, LLMs are sensitive to prompt phrasings and can be influenced by the way questions are worded. Ideally, LLMs should respond consistently regardless of phrasing, particularly when grounded in the same underlying evidence. We investigate this through a systematic evaluation in a controlled retrieval-augmented generation (RAG) setting for medical question answering (QA), where expert-selected documents are used rather than retrieved automatically. We examine two dimensions of patient query variation: question framing (positive vs. negative) and language style (technical vs. plain language). We construct a dataset of 6,614 query pairs grounded in clinical trial abstracts and evaluate response consistency across eight LLMs. Our findings show that positively- and negatively-framed pairs are significantly more likely to produce contradictory conclusions than same-framing pairs. This framing effect is further amplified in multi-turn conversations, where sustained persuasion increases inconsistency. We find no significant interaction between framing and language style. Our results demonstrate that LLM responses in medical QA can be systematically influenced through query phrasing alone, even when grounded in the same evidence, highlighting the importance of phrasing robustness as an evaluation criterion for RAG-based systems in high-stakes settings. Figure 1: Overview of our evaluation pipeline for assessing LLM sensitivity to patient question framing in a controlled RAG-based medical QA setting. Given paired queries that vary only in phrasing (positive vs. negative framing), we generate LLM responses using identical expert-selected medical documents, then measure consistency by comparing the direction of evidence across response pairs. 1 Introduction LLMs have demonstrated remarkable capabilities in healthcare ( 47 ; 48 ) , internalizing medical knowledge and performing complex clinical reasoning. These models perform well on benchmarks such as MedQA ( 20 ) and those derived from the United States Medical Licensing Examination (USMLE) ( 33 ; 47 ; 42 ; 48 ) . Increasingly, patients are turning to LLMs directly with medical questions ( 9 ; 55 ; 31 ; 36 ; 57 ) —however, they are unlikely to phrase questions in ways that benchmarks test for “medical ability.” Existing benchmarks say little about whether models will safely and reliably respond to real patient queries. Patient questions are often complex, unstructured, and shaped by misunderstandings from limited medical knowledge, unvetted sources, information from friends, or prior conversations with physicians ( 34 ; 14 ) . Further, patient language may feature a variety of dialects, grammatical errors, and emotional expressions ( 26 ; 49 ; 51 ; 37 ) . While LLMs perform well on medical benchmarks, 4 recently found that performance declines when models are used by the general public, likely due to vague or incomplete user queries. 54 showed that LLMs were substantially better at citing relevant medical references for expert queries than for lay ones. Ideally, LLM responses to medical questions would be consistent regardless of how a query is phrased, particularly when grounded in the same underlying evidence. If a patient receives different guidance simply because they framed their question positively rather than negatively (e.g., “ Does this treatment work? ” vs. “ Does this treatment not work? ”), or because they used colloquial (vs. technical) language, there can be real consequences. Retrieval-augmented generation (RAG) systems ( 46 ) , which ground model responses in relevant external sources such as medical literature, are increasingly deployed to improve reliability in healthcare context. However, it remains an open question whether adding relevant documents, in context as RAG, mitigates LLM sensitivity to patient query phrasing, or if such sensitivity persists even when the underlying evidence is held constant. Our work systematically investigates how variations in patient query phrasing affect the consistency of LLM responses in a controlled RAG-based medical QA setting. We compare responses generated from positively- and negatively-framed questions and from technical and plain language questions grounded in the same randomized controlled trial (RCT) abstracts. We also examine whether LLMs are more susceptible to persuasion across multi-turn conversations than in single-turn interactions. To this end, we construct a dataset of paired technical and plain language medical queries with positive and negative framings grounded in RCT abstracts, and conduct a systematic evaluation of LLM response consistency and susceptibility to persuasion across these conditions. Our research questions are: RQ1: Framing Sensitivity. How does positive versus negative framing of patient medical queries affect the consistency of LLM responses when grounded in the same RCT evidence? RQ2: Single-turn vs. Multi-turn Susceptibility. Are LLMs more susceptible to framing effects under repeated persuasive multi-turn conversations than in single-turn interactions? RQ3: Technical vs. Plain Language Susceptibility. Are LLMs more susceptible to framing effects in plain language queries than in technical ones? 2 Related Work Prior work has shown that LLMs exhibit systematic biases in medical QA, such as cognitive biases prevalent in clinical practice that prioritizes short-term benefits over long-term risks ( 22 ) . Research has also shown that query variations from patients significantly affect the quality of LLM responses in healthcare contexts. 12 found that tonal, syntactic, and demographic perturbations shifted treatment recommendations by 7–9% on average, while 21 demonstrated that implicit identity markers can systematically influence medical advice across race. Importantly, biases persist even in RAG systems designed to enhance accuracy and reliability. 18 and 23 highlighted how RAG can inadvertently amplify demographic biases, and 54 found that question type greatly affects the quality of sources provided by LLMs. Findings from 54 showed that queries from medical experts outperformed those from the general public, with only 30% of public queries fully substantiated by RAG-enabled GPT-4 model. This suggests that patients most in need of reliable information are least well-served. 53 identified related issues in commercial RAG systems, including fact decontextualization and reinforcement of patient misconceptions, although their analysis of question framing effect was preliminary. LLM sycophancy—broadly, the tendency to excessively agree with or flatter users—poses additional risks in patient-facing contexts ( 45 ; 8 ; 25 ) . 60 found that frontier LLMs correct false presuppositions (i.e., misconceptions) in cancer-related questions less than half of the time, and 37 documented real-world cases in which leading questions induced sycophantic responses in health chatbots. More broadly, 1 showed that LLMs cherry-pick evidence to make misleading answers appear credible, while 56 demonstrated that sustained persuasion through multi-turn conversations can completely shift LLM beliefs. These existing studies focus on narrow syntactical perturbations, emphasize sociodemographic variations over query-level framing effects, and largely omit RAG contexts. Here we address these gaps by systematically examining how positively (and negatively) framed questions rooted in human cognitive bias affect LLM behavior in a RAG-based medical QA system. 3 Evaluation We evaluate how LLMs respond to positively-framed questions (emphasizing beneficial outcomes) compared to negatively-framed ones (highlighting the negative outcomes or side effects). These questions capture important psychological biases and preconceptions that users bring to treatment decisions. Specifically, we examine how this type of question framing influences the consistency of LLM responses as a function of the following patient and contextual factors that are common among health information seeking LLM