Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?

Zhengyang Shan, Aaron Mueller · 2025

Read Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics? by Zhengyang Shan, Aaron Mueller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20796.txt.

Introduction

Zhengyang Shan and Aaron Mueller ask whether a language model can lose its stereotypes without losing sight of demographics entirely. A model may need to recognize names or groups for fair treatment, yet use the same signals to repeat profession stereotypes. The pair test whether those two mechanisms are truly separate inside the model.

Working with Gemma-2-9B, they switched off single internal features picked out by two different hunting methods. Targeted switches reduced race and gender profession bias while name recognition stayed intact. Education bias responded better to the other method, and one wrong switch actually increased bias. Debiasing, they conclude, must be tailored bias by bias.

Catalog record

CREATORZhengyang Shan, Aaron Mueller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics? Zhengyang Shan, Aaron Mueller Abstract We investigate how independent demographic bias mechanisms are from general demographic recognition in language models. Using a multi-task evaluation setup where demographics are associated with names, professions, and education levels, we measure whether models can be debiased while preserving demographic detection capabilities. We compare attribution-based and correlation-based methods for locating bias features. We find that targeted sparse autoencoder feature ablations in Gemma-2-9B reduce bias without degrading recognition performance: attribution-based ablations mitigate race and gender profession stereotypes while preserving name recognition accuracy, whereas correlation-based ablations are more effective for education bias. Qualitative analysis further reveals that removing attribution features in education tasks induces “prior collapse”, thus increasing overall bias. This highlights the need for dimension-specific interventions. Overall, our results show that demographic bias arises from task-specific mechanisms rather than absolute demographic markers, and that mechanistic inference-time interventions can enable surgical debiasing without compromising core model capabilities. 1 Introduction Large language models (LLMs) are increasingly deployed in socially sensitive domains, raising concerns about how they represent and reason about demographic attributes such as race, gender, and education. While prior research has documented biases in model outputs 4 ; 8 ; 19 , less is known about the internal mechanisms that give rise to these biases. Understanding how demographic information is encoded in model representations, and how these representations are used across different tasks, is crucial for building systems that are both fair and interpretable. Recent advances in mechanistic interpretability, particularly sparse autoencoders (SAEs; 7 ; 5 ), provide new opportunities to move beyond surface-level bias detection and investigate which internal features drive demographic reasoning ( 16 ) . However, current bias mitigation approaches face significant limitations: training-time interventions require expensive model retraining 24 , while complete removal of demographic information can harm performance in legitimate contexts where such information is relevant. Recent work demonstrates that alignment processes can inadvertently amplify implicit bias by reducing awareness of demographic concepts 27 . Most critically, existing work has either focused on output benchmarks 25 ; 13 or interpretability 18 ; 26 in isolation, without systematically characterizing the scope of bias mechanisms across diverse task contexts. We define bias mitigation as selective reduction of a feature’s influence when it is causally irrelevant. This requires context-sensitive interventions. Given tasks T_{j}\in\mathcal{T} and representations \mathbf{h} , we seek to identify features f_{i}\in\mathbf{f} , where \mathbf{f} is derived from \mathbf{h} (e.g., via autoencoders) such that their indirect effect IE (T,f_{i}) on task performance satisfies: • IE (T_{j},f_{i})=0 when f_{i} is causally irrelevant to T_{j} • IE (T_{r},f_{i})\neq 0 when f_{i} is causally relevant to T_{r} where IE (T_{j},f_{i}) represents the causal influence of feature(s) f_{i} on task performance. The challenge lies in finding features that can be universally ablated to reduce performance only on causally irrelevant associations (harmful stereotypes) while preserving performance on causally relevant associations (legitimate demographic recognition). In this work, we introduce an automated pipeline designed to systematically characterize bias mechanisms across diverse task contexts and demographic dimensions. Rather than seeking universal solutions, our approach measures trade-offs between debiasing and general performance via multi-task demographic reasoning evaluations. We evaluate three core task categories: name–demographic associations (where demographic reasoning is causally relevant), profession–demographic stereotyping (where it is irrelevant), and profession–education requirements (where it is sometimes relevant). Multi-task evaluation enables us to assess whether biases arise from shared underlying mechanisms or distinct representational pathways for each prediction direction. We compare four ways of collecting features to ablate: gradient attribution, activation correlations, the intersection of the attribution and correlation sets, and the difference of attribution and correlation sets. This allows us to test whether input features or output features ( 2 ) achieve the best trade-off between debiasing and general performance. Our contributions are threefold: 1. We investigate how independent bias mechanisms are from general demographic detection using a multi-task setup that disentangles necessary recognition from stereotype-driven associations. 2. We compare attribution- and correlation-based methods for finding features, as well as intersection and non-overlapping variants. These have distinct effects on bias mitigation across demographic dimensions. 3. We evaluate the robustness of our findings with bidirectional prompt formats (Demo-L vs. Demo-R), providing evidence that our conclusions hold across alternative task framings. Figure 1: The pipeline implements bidirectional evaluation using two prompt formats: Demo-R where demographic labels appear after items (“Jack -White”), and Demo-L where labels precede items (“Asian -Ming”). Following structured prediction generation, we apply attribution analysis and correlation analysis using sparse autoencoder (SAE) features to identify neural representations responsible for demographic associations. These features inform targeted ablation experiments to test their causal roles in both legitimate recognition and stereotyping behaviors. 2 Related Work Feature Attribution and Steering Recent advances in mechanistic interpretability have enabled decomposition of neural network representations into interpretable features through sparse autoencoders (SAEs) ( 7 ; 15 ; 5 ) . While activation steering techniques demonstrate effective inference-time control ( 14 ; 34 ; 30 ; 22 ; 16 ) , identifying the most effective features remains challenging. Methods based on integrated gradients ( 28 ) are currently the most common ( 12 ; 18 , e.g.,) . Our work compares gradient attribution–based methods like integrated gradients with correlation-based methods, revealing that each can be effective for different types of bias. Bias Mitigation Existing bias mitigation approaches including training-time interventions ( 24 ) , post-processing corrections, and representation learning modifications, often lack proper controls to distinguish legitimate demographic recognition from harmful stereotypical associations. As discussed by 11 , an absence of controls can lead to not entirely removing the targeted bias. However, there is also a risk of non-selectivity that is less commonly discussed: if bias is entangled with a model’s ability to recognize a demographic, then removing it could impair performance in contexts requiring demographic awareness or fail to target specific harmful biases ( 17 ) . Complete removal of demographic information risks eliminating necessary recognition capabilities, while uncontrolled interventions may introduce unintended side effects 32 . Our cross-task evaluation framework enables us to investigate this limitation by using name-demographic associations as a control condition. Localization of Bias Prior work has identified specific model components encoding bias, from attention heads and MLP layers ( 31 ) to localized transformer components ( 10 ; 6 ) . However, shared pathways between bias and other tasks complicate targeted mitigation efforts. Our work extends this mechanistic understanding through evaluation across multiple demographic categories, revealing how different feature identification methods capture distinct aspects of bias-encoding representations and their downstream effects. 3 Method 3.1 Pipeline Overview Figure 1 illustrates our pipeline. We assess three phenomena: (1) bidirectional associative accuracy between names and demographic categories; (2) systematic biases in prediction outcomes and internal feature representations; and (3) the causal influence of SAE feature ablations on both associative accuracy and bias. Prompt Format We construct prompts designed to elicit implicit associations between demographics and attributes. The experiment with placing the demographic on the right-hand side (Demo-R) format draws inspiration from the word-association paradigm proposed by 3 , while we introduce the left-hand side (Demo-L) format to assess asymmetric performance patterns. Each prompt presents a batch of eight items and instructs the model to generate pairings using consistent syntax. Demo-R

Read the complete text