Why Do Large Language Models Generate Harmful Content?

Rajesh Ganguli (✉), Raha Moraffah · 2026

Read Why Do Large Language Models Generate Harmful Content? by Rajesh Ganguli (✉), Raha Moraffah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11663.txt.

Introduction

Ganguli and Moraffah trace harmful LLM generation to its causal factors with multi-granular mediation analysis across layers, modules, and neurons. Experiments show harmful generation arises in later layers. It stems mainly from MLP block failures rather than attention blocks, and involves neurons acting as gates for harmful generation. Early layers build contextual understanding of harmfulness in a prompt, which propagates to late layers and to a sparse set of last-layer neurons that determine harmful output.

Catalog record

CREATORRajesh Ganguli (✉), Raha Moraffah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42895
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Do Large Language Models Generate Harmful Content? Rajesh Ganguli (✉), Raha Moraffah Abstract Large Language Models (LLMs) have been shown to generate harmful content. However, the underlying causes of such behavior remain underexplored. We propose a causal mediation analysis–based approach to identify the causal factors responsible for harmful generation. Our method performs a multi-granular analysis across model layers, modules (MLP and attention blocks), and individual neurons. Extensive experiments on state-of-the-art LLMs indicate that harmful generation arises in the later layers of the model, results primarily from failures in MLP blocks rather than attention blocks, and is associated with neurons that act as a gating mechanism for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt, which is then propagated through the model, to generate harmfulness in the late layers, as well as a signal indicating harmfulness through MLP blocks. This is then further propagated to the last layer of the model, specifically to a sparse set of neurons, which receives the signal and determines the generation of harmful content accordingly. 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across variety of tasks, leading to their rapid deployment in increasingly safety-critical domains [ 16 , 22 ] . Despite these advances, LLMs remain susceptible to generating harmful content [ 21 , 24 ] . Understanding harmful generation is a fundamental challenge in ensuring reliable and enforceable safety guarantees. Without a principled understanding of the internal mechanisms that cause harmful content generation, safety interventions remain brittle, difficult to generalize, and challenging to verify [ 25 ] . Prior research has explored characterizing harmful behavior in LLMs by identifying internal subspaces associated with harmful representations [ 19 ] , analyzing the structural properties of adversarial prompts [ 3 ] , and performing targeted activation interventions to modulate unsafe outputs [ 9 ] .These efforts are correlational in nature, establishing associations but not demonstrating that the identified features or interventions are the actual causes of the harmful behavior. However, to ensure that robust and principled safety guards are viable for LLMs, it is necessary to identify the causal factors [ 5 ] . In this paper, we propose a Causal Mediation Analysis (CMA)-based framework for identifying the causal factors underlying harmful content generation in LLMs. Specifically, we treat individual model components as candidate mediators and quantify the extent to which the causal effect of the input on harmful output is transmitted through each component. By estimating these mediated effects, our approach enables tracing the flow of harmful information through the model and identifying components that causally contribute to harmful generation. To further investigate the transformation and propagation of the harmful information throughout the LLM architecture, our approach is conducted across different levels of granularity. We analyze the causal contribution of the model’s layers to determine the responsible causal mechanisms. We also conduct finer-grained analysis of modules (MLP and Attention blocks) to determine the components within a layer that cause the generation. Finally, we extend our analysis to individual neurons to establish the causal flow of harmful generation. This hierarchical approach provides insight into both coarse architectural roles and fine-grained mechanistic interactions. We conduct extensive experiments over state-of-the-art (SOTA) LLMs. Our results indicate that harmful generation arises in the late and final layers within the model, is a product of failures within the MLP blocks rather than contextual attention routing, and from neurons that serve as mechanisms for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt which is then propagated to the late layers of the model which generate a response as well as a signal indicating the harmfulness utilizing the MLP blocks, which is then further propagated to the final layer of the model for which a sparse set of neurons act as a gating mechanism that functions off of the signal allowing for the generation of the harmful content or the refusal. These findings display the causes of harmful generation within LLMs, providing insights for future research on safety guardrails within these models. Our contributions are summarized as follows: • We propose a causal mediation analysis-based approach that identifies the causal mechanisms responsible for harmful content generation. • We propose a multi-granular approach that allows for further exploration of the flow of information within the model. • Our comprehensive experiments on SOTA LLMs demonstrate that late layers in a model are integral for the generation and signaling of harmful content as a result of failures within the MLP components, which provide a signal to gating neurons. 2 Related Work Harmful Content Generation. Large Language Models remain susceptible to the generation of harmful content [ 24 ] . This generation can be performed through a toxicity present in trained bias [ 13 ] , through adversarial attacks [ 20 ] ,and through jailbreak attacks [ 11 ] . Understanding how these systems allow for harmful content generation allows for improved moderation and guarding of content generation [ 1 ] . Mechanistic Interpretability for Harmful Content Generation. Recent work on understanding harmful content generation applies mechanistic interpretability through manipulation of internal subspaces to bypass safeguards [ 19 , 7 ] , analyzing prompt features [ 3 ] , steering vectors [ 8 ] , and input vectors of refusal and harmfulness prevention [ 2 , 10 ] . Similar works instead employ weak classifiers and logit grafting to modify hidden states [ 29 ] . Other studies [ 4 , 6 ] utilize activation patching to isolate individual safety neurons or specific task vulnerabilities. Furthermore, other efforts propose a dual framework using representation analysis to characterize how jailbreaks alter the model’s perception of harmfulness, coupled with circuit analysis to identify components responsible for the resulting deception [ 7 ] . These prior mechanistic approaches correlate specific internal features or interventions with harmful generations, but do not identify causal mechanisms. Our novel work causally identifies these mechanisms responsible. Causal Mechanistic Interpretability for LLMs. Causal methods in mechanistic interpretability aim to move beyond simple correlation by identifying the causal mechanisms underlying model behavior. Recent work in this area can be broken into the categories of causal abstraction, causal interventions, and causal mediations. Causal abstraction seeks to extract latent causal variables from observational data to better understand internal model states [ 15 , 18 ] . Causal Interventions work traces the causal flow of information across layers through targeted interventions upon the model [ 30 , 14 ] . Causal Mediation Analysis (CMA), works as a robust framework for decomposing the internal decision-making processes of LLMs. By treating internal components of the LLM as a mediator, CMA quantifies the causal contribution of the mediator on the final output through its indirect effect. Existing wors has developed causal mediation analysis-based methods to analyze an LLM’s logical reasoning capabilities [ 17 ] and to evaluate neuron-level security threats [ 26 ] . While prior efforts have shed light on reasoning and guardrail vulnerabilities, CMA in the specific domain of safety alignment remains underexplored. To the best of our knowledge, our work is the first to propose CMA to

Read the complete text