Position: The Pitfalls of Over-Alignment: Overly Caution Health-Related Responses From LLMs are Unethical and Dangerous

Wenqi Marshall GuoYiyang Du, Heidi J.S. Tworek, Shan Du · 2025

Read Position: The Pitfalls of Over-Alignment: Overly Caution Health-Related Responses From LLMs are Unethical and Dangerous by Wenqi Marshall GuoYiyang Du, Heidi J.S. Tworek, Shan Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08833.txt.

Introduction

Wenqi Marshall Guo and 3 co-authors argue, in a 2025 position paper, that over-alignment of health answers from language models can be harmful. Alignment aims to stop harmful output. In health queries, the authors say, overly cautious replies can raise anxiety and cause real risk.

People with anxiety or obsessive-compulsive disorder are singled out as especially affected. Qualitative results show models differ in how cautious they are. The paper calls for stronger reasoning and more tailored, nuanced health responses. A warning notes the paper contains material that could trigger health anxiety. Data and full results are shared through a public repository.

Catalog record

CREATORWenqi Marshall GuoYiyang Du, Heidi J.S. Tworek, Shan Du
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: The Pitfalls of Over-Alignment: Overly Caution Health-Related Responses From LLMs are Unethical and Dangerous Wenqi Marshall GuoYiyang Du, Heidi J.S. Tworek, Shan Du Abstract Large Language Models (LLMs) are usually aligned with “human values/preferences” to prevent harmful output. Discussions around the alignment of Large Language Models (LLMs) generally focus on preventing harmful outputs. However, in this paper, we argue that in health-related queries, over-alignment—leading to overly cautious responses—can itself be harmful, especially for people with anxiety and obsessive-compulsive disorder (OCD). This is not only unethical but also dangerous to the user, both mentally and physically. We also showed qualitative results that some LLMs exhibit varying degrees of alignment. Finally, we call for the development of LLMs with stronger reasoning capabilities that provide more tailored and nuanced responses to health queries. Warning: This paper contains materials that could trigger health anxiety or OCD. 1 Introduction Figure 1: A simplified illustration of our position in this paper. We argued that overly cautious responses could lead to severe outcomes. Large Language Models (LLMs) are becoming increasingly powerful and are now widely used as a daily source of information, particularly for specific and tailored queries. An Ipsos survey found that about 30% of the US consumers are already using generative AI to fill needs between doctor’s appointments for healthcare ( Choy et al. 2024 ) . To prevent LLMs from producing harmful or unsafe advice, they are typically aligned with certain safety preferences. These preferences are generalized and shaped by developers, meaning that they do not represent the full spectrum of real-world issues. Here, we suggest that while literature has focused on the harm of under-cautious responses, overly cautious responses can themselves be harmful, especially for vulnerable individuals ( Dorison et al. 2022 ; Grant et al. 2022 ) such as those suffering from obsessive-compulsive disorder (OCD) and anxiety, particularly in domains such as health and safety, where LLMs tend to be more conservative ( Zeng et al. 2025 ) . While much existing research focuses on improving the safety of LLMs, little attention has been paid to the potential harm caused by excessive caution. To the best of our knowledge, we are one of the first to investigate this problem. We refer to this phenomenon as over-alignment, analogous to overfitting in traditional machine learning. Previous work has advocated for individualized safety alignment to offer greater protection for vulnerable populations ( In et al. 2025 ) , but this has largely addressed under-cautious rather than over-cautious behavior. In this paper, we argue that the safety values underlying models might not be universalizable as they seem, and specifically, over-alignment to such values in health-related questions can be both harmful and unethical. Additionally, we constructed our own small dataset to qualitatively evaluate popular models, and investigate if some models suffer from this over-alignment problem. 2 Related Work 2.1 LLM Alignment and Value Pluralism AI developers often claim that they have aligned their AI with “human values” or “human preferences”, aiming to increase its usefulness and harmlessness, including InsturctGPT and Anthropic AI. ( Ouyang et al. 2022 ; Bai et al. 2022 ; Hendrycks et al. 2023 ) . One such value or preference is safety. However, even something as seemingly universal as safety looks different in different places to different people. Sutrop 2020 concerns that AI developers underestimated the difficulty of the question about which values or whose values the AI should align with. The authors argued that given that our everyday life is full of moral disagreements and the plural nature of values, how can we decide which objectives or values we inject into the AIs? Arzberger et al. 2024 argues that current alignment approaches rely on universal framings of human values, which could be problematic and result in AI systems that are biased, leading to equity and justice issues. Turchin 2019 proposed an even more critical point of view, which argues that “human values” are not an object, “human value system” has flaws, and even “human values” are not good by default. He suggests that “human values” in AI should be replaced with something better, or at least used very cautiously. Existing evaluations have shown that the model could be biased towards different cultural backgrounds, due to either unintentional bias in the training data or intentional bias introduced during alignment. Segerer 2025 finds that DeepSeek (a Chinese LLM) shows more value towards collectivism compared to Western LLMs. Münker 2025 states that their study suggests a concerning reality: “Large Language Models (LLMs) fail to represent diverse cultural moral frameworks despite their linguistic capabilities.” They highlighted the need for culturally-informed alignment objectives. Current approach regresses the model to a “mean moral framework” rather than representing diverse human values. Without cross-cultural evaluation metrics, models may appear well-aligned within the tested context but fail to perform appropriately under alternative moral frameworks. 2.2 Over-alignment The term over-alignment has been used informally before to describe how “AI systems excessively rely on a user’s expertise, perceptions, or hypotheses without sufficient independent validation or critical engagement” ( Fitzgerald 2025 ) . This problem is also sometimes referred to as “AI sycophant” ( Open AI 2025 ; Sharma et al. 2025 ; Chen et al. 2025 ; Arvin 2025 ) . It describes where AI is over-aligned on “helpfulness” or “friendliness”, and thus cannot give meaningful advice. This is different to what we are describing in this paper, which tackle the problems that AI is over-aligned to “harmlessness.” 2.3 AI Risk Preferences A large body of literature examines LLMs’ approach to risk. Ouyang et al. 2025 studied how LLMs’ cautiousness in ethical alignment affects economically valuable risk-taking, which might affect economic forecasts and suppress valuable risk-taking. Zeng et al. 2025 applied DOSPERT ( Blais & Weber 2006 ) to different LLMs and found that they show different risk tolerance in different areas; however, they did not compare with a human baseline. Ray & Bhalani 2024 studied LLMs’ over-refusal in cases like prompts with homonyms (e.g., how to kill a process) or safe context (“how to kill someone in [a video game name]”), etc. They found that many LLMs have problems with over-refusing prompts. Cui et al. 2025 is another benchmark and evaluation for model over-refusal, and they found a positive relationship between over-refusal and safety. In et al. 2025 argued that AI safety should be tailored to individual people. For example, a normal diet question might be harmless for normal people, but be dangerous for people with an eating disorder. However, this work only focuses on how AI should be more “cautious” for certain populations, instead of avoiding being overly cautious. Although we agree with their idea that AI safety is contextual, we disagree with giving a person’s mental health, criminal, and financial details to AI and AI providers, which raises significant privacy, anonymity, and autonomy concerns. 2.4 Health Tools, OCD, and Anxiety Before LLMs became popular tools, individuals, particularly those with OCD or health anxiety, were already turning to resources such as online symptom checkers and nursing helplines for medical reassurance. One study ( Wetzel et al. 2024 ) found that health anxiety (hypochondria) is a reliable predictor of symptom checker application (SCA) use. Over half of the SCA users scored

Read the complete text