Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks
Junjie Chu Yiting Qu Ye Leng Michael Backes Yun Shen Savvas Zannettou Yang Zhang · 2026
Read Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks by Junjie Chu Yiting Qu Ye Leng Michael Backes Yun Shen Savvas Zannettou Yang Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11914.txt.
Introduction
Chu and co-authors ask whether language models refuse benign tasks when the user's material contains harmful content. Alignment work mostly targets directly harmful tasks, not this content-level question. The team builds a harmful knowledge set of 1,357 entries in ten categories, plus nine harmless tasks needing heavy to light use of supplied text. They test nine mainstream models, including GPT-5.2 and Gemini-3-Pro. Even the newest models often keep processing the harmful material instead of refusing, they find. How task and harm category interact shifts the behavior further.
Catalog record
| CREATOR | Junjie Chu Yiting Qu Ye Leng Michael Backes Yun Shen Savvas Zannettou Yang Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87953 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11914
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks Junjie Chu Yiting Qu Ye Leng Michael Backes Yun Shen Savvas Zannettou Yang Zhang Abstract Large Language Models (LLMs) are increasingly trained to align with human values, primarily focusing on task level, i.e., refusing to execute directly harmful tasks. However, a subtle yet crucial content-level ethical question is often overlooked: when performing a seemingly benign task, will LLMs—like morally conscious human beings—refuse to proceed when encountering harmful content in user-provided material? In this study, we aim to understand this content-level ethical question and systematically evaluate its implications for mainstream LLMs. We first construct a harmful knowledge dataset (i.e., non-compliant with OpenAI’s usage policy) to serve as the user-supplied harmful content, with 1,357 entries across ten harmful categories. We then design nine harmless tasks (i.e., compliant with OpenAI’s usage policy) to simulate the real-world benign tasks, grouped into three categories according to the extent of user-supplied content required: extensive, moderate, and limited. Leveraging the harmful knowledge dataset and the set of harmless tasks, we evaluate how nine LLMs behave when exposed to user-supplied harmful content during the execution of benign tasks, and further examine how the dynamics between harmful knowledge categories and tasks affect different LLMs. Our results show that current LLMs, even the latest GPT-5.2 and Gemini-3-Pro, often fail to uphold human-aligned ethics by continuing to process harmful content in harmless tasks. Furthermore, external knowledge from the “Violence/Graphic” category and the “Translation” task is more likely to elicit harmful responses from LLMs. We also conduct extensive ablation studies to investigate potential factors affecting this novel misuse vulnerability. We hope that our study could inspire enhanced safety measures among stakeholders to mitigate this overlooked content-level ethical risk. Disclaimer: Examples of harmful content are included. Reader discretion is recommended. $\clubsuit$ $\clubsuit$ footnotetext: Corresponding author. Introduction Moral human beings not only make moral judgments about whether to perform a given task, but also about how to act when encountering unethical materials in otherwise benign tasks. For example, professional translators, according to the International Association of Professional Translators and Interpreters (IAPTI) Code of Ethics [ 23 ] , are required to refuse to translate material that could harm the public interest, the law, or the profession, such as material related to weapon production, even if the translation itself is morally neutral. Mishandling harmful materials within legitimate tasks can even incur legal liabilities, as seen in real-world prosecutions over the translation of extremist content. 1 1 1 Mehanna was arrested and later convicted of providing material support to Al Qaeda, including translating radical terrorist books and videos into English for the website At Tibyan [ 66 , 55 , 61 , 51 ] . These moral and legal constraints illustrate a fundamental aspect of human ethics: it extends beyond task-level refusal to encompass content-level discernment. In the domain of large language models (LLMs), models [ 44 , 36 , 37 ] are increasingly trained for safety alignment to ensure consistency with human values. Through techniques such as reinforcement learning with human feedback (RLHF) [ 49 ] , rule-based reward modeling [ 48 ] , and red teaming [ 53 ] , LLMs have learned the task-level values and are able to refuse overtly harmful tasks, such as providing instructions for illegal activities. Current safety alignment efforts often overlook the human-like discernment at the content level. It remains unclear whether LLMs possess content-level moral awareness—the ability to recognize and avoid processing user-supplied harmful materials within an otherwise benign task. Consider a scenario in which a user provides a document containing detailed instructions for weapon manufacturing or extremist propaganda and requests the translation of it (see Figure 2 ). A professional human translator would refuse to process such content, adhering to their ethical standards. In contrast, an LLM might faithfully translate it and even expand or elaborate on the harmful material using its pretrained knowledge, because its moral filters might not be activated in the context of a seemingly benign task. Such behavior could result in real-world harm, like information hazards. 2 2 2 Information hazard [ 6 ] : A risk that arises from the (potential) dissemination of (true) information that may cause harm or enable some agent to cause harm. This discrepancy highlights a fundamental gap in the content level between human moral awareness and current paradigms of LLM safety alignment. Bridging this gap is not only a matter of improving model safety but a prerequisite for developing genuinely ethically aligned artificial general intelligence (AGI). Figure 1 : An instance of the in-content harm risk : translating detailed instructions for constructing nuclear bombs [ 60 ] . Figure 2 : Overview of the assessment process for in-content harm risk . We refer to this overlooked aspect of model alignment as in-content harm risk , a complementary alignment challenge requiring LLMs to exercise ethical discernment toward user-provided harmful materials, even when performing benign tasks. In this study, we aim to characterize this risk and comprehensively assess its impact on mainstream LLMs. Methodology. An overview of our assessment framework is shown in Figure 2 . To investigate this new form of risk, we first construct a harmful knowledge dataset consisting of 1,357 entries across ten harmful categories, serving as user-supplied harmful content. We then design ten seemingly harmless tasks, grouped into three categories according to their dependence on user-provided information: user - supplied - knowledge - dependent (extensive), mixed - knowledge (moderate), and pretrained - knowledge - dependent (limited). Importantly, these tasks themselves do not violate model usage policies. To systematically quantify the risk, we employ three metrics: the harmful response number per knowledge piece (K - HRN, ranging from 1.0–9.0, with higher values indicating greater risk), the harmful response rate per task (T - HRR, ranging from 0.0–1.0), and the groundedness score (GS) [ 40 , 64 ] . Evaluation & Findings. We begin by conducting experiments on nine frontier LLMs to measure, analyze, and compare in - content harm risk across models, harmful categories, and tasks. Our findings show that many mainstream LLMs are highly vulnerable to this risk. Among them, even Qwen3 (one of the latest open-source models) exhibits the second-highest vulnerability, with an average K - HRN of 3.942, indicating that, out of nine harmless tasks, about four (on average) elicit harmful responses from LLMs when tested with the given knowledge piece. In contrast, Llama 3 shows the greatest resilience with a low average K - HRN of 0.178. Tasks that rely extensively on user - provided content also prove more susceptible to this risk. For example, the Translation task reaches an average T - HRR of 0.512, meaning that over half of the tested knowledge pieces result in harmful responses from LLMs under this task. We further perform ablation studies to quantify the effects of various factors on in - content harm risk. These factors include the source of harmful content in the generated responses, the status of internal safety checks, and the length, proportion, position, and diversity of the harmful information within user - supplied materials. Our analyses reveal that explicitly instructing LLMs to perform safety checks can substantially reduce the generation of harmful responses. Under the same task conditions,