GKnow: Measuring the Entanglement of Gender Bias and Factual Gender
Leonor Veloso, Hinrich Schütze · 2026
Read GKnow: Measuring the Entanglement of Gender Bias and Factual Gender by Leonor Veloso, Hinrich Schütze (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12299.txt.
Introduction
Veloso and Schütze built a test set, called GKnow, that separates two kinds of gendered language-model output. Factual gender follows a word that truly carries gender as part of its meaning, while biased gender rests on a stereotype instead. Probing circuits and single neurons, they find the two are deeply tangled, so deleting neurons meant to remove bias also harms factual gender handling. Ablation, they argue, is therefore an unreliable way to debias a model.
Catalog record
| CREATOR | Leonor Veloso, Hinrich Schütze |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64054 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GKnow: Measuring the Entanglement of Gender Bias and Factual Gender Leonor Veloso, Hinrich Schütze Abstract Recent works have analyzed the impact of individual components of neural networks on gendered predictions, often with a focus on mitigating gender bias. However, mechanistic interpretations of gender tend to (i) focus on a very specific gender-related task, such as gendered pronoun prediction, or (ii) fail to distinguish between the production of factually gendered outputs (the correct assumption of gender given a word that carries gender as a semantic property) and gender biased outputs (based on a stereotype). To address these issues, we curate GKnow, a benchmark to assess gender knowledge and gender bias in language models across different types of gender-related predictions. GKnow allows us to identify and analyze circuits and individual neurons responsible for gendered predictions. We test the impact of neuron ablation on benchmarks for disentangling stereotypical and factual gender (DiFair and the test set of GKnow), as well as StereoSet. Results show that gender bias and factual gender are severely entangled on the level of both circuits and neurons, entailing that ablation is an unreliable debiasing method. Furthermore, we show that benchmarks for evaluating gender bias can hide the decrease in factual gender knowledge that accompanies neuron ablation. We curate GKnow as a contribution to the continuous development of robust gender bias benchmarks. 1 Introduction Thanks to recent developments in the field of mechanistic interpretability , we have a growing understanding of why and how a large language model (LLM) produces a certain output ( 11 ) . Mechanistic analysis methods have been applied to the production of socially biased outputs. For the specific case of gender bias, techniques such as causal mediation analysis ( 43 ; 6 ; 10 ) have been employed to locate relevant components for the production of gender biased outputs. More interpretable debiasing techniques, such as probing ( 38 ) , concept erasure ( 2 ) , and neuron ablation ( 29 ; 23 ; 7 ) are gaining traction due to their lack of reliance on costly curation of datasets and expensive fine-tuning. An often overlooked side effect of gender debiasing is damage to what can be referred to as the model’s factual gender knowledge ( 48 ) , i.e., the lexical correspondence between nouns that possess the semantic property of female or male and their respective pronouns or determiners (e.g., woman/she and man/he ). This phenomenon is a symptom of the entanglement of gender bias and factual gender signals in the inner model representations. Previous work has focused on the development of debiasing methods that keep the factual gender signal ( 28 ; 26 ) , but the mechanistic story behind this entanglement on a circuit level remains mostly unexplained. Our contributions 1 1 1 https://github.com/leonorv/gknow are as follows: (i) We curate GKnow, closing the gap in available benchmarks for evaluating the entanglement of gender bias and gender knowledge in English, for different types of gender-related tasks; (ii) Employing the state-of-the-art circuit analysis technique EAP-IG, we find evidence of entanglement between gender bias and factual gender on a circuit-level. This circuit-level analysis is described in Section 4 . (iii) We employ the interpretability/debiasing technique of neuron ablation for Llama-3.1-8b and Olmo-7b on the test set of GKnow, StereoSet ( 34 ) , and DiFair ( 48 ) , a benchmark for assessing the entanglement of gender bias and factual gender. We detail our neuron-level analysis and experiments in Section 5 . We observe that circuits responsible for gender bias and factual gender knowledge have a high overlap, and a low degree of separation when measuring cross-task faithfulness (i.e., the ability of one circuit to solve another’s task ( 19 ) ). Since this entanglement is also present at the level of neurons, simple ablation of the most relevant gender bias neurons causes a sharp decrease in the model’s linguistic ability to correctly predict factual gender. However, we also find that ablation of stereotypical gender neurons has results that can be interpreted as positive in biased settings, masking the decrease in factual gender knowledge. This highlights the need to develop resilient gender bias evaluation datasets that take factual gender into account. Based on these findings, we caution against unfiltered neuron ablation as the sole method of gender debiasing. 2 Related Work 2.1 Linguistic Gender in English Although English lacks grammatical gender, it possesses lexically gendered nouns and socially gendered nouns ( 32 ) . Lexical and social gender are linguistic gender categories: lexical gender refers to the semantic association between a noun and its gender ( woman/she and man/he ), while social gender conveys stereotypical femaleness or maleness and can differ across time and cultures ( nurse/she and pilot/he ). Following other NLP works ( 48 ; 28 ) , we will refer to social gender as stereotypical gender , and to lexical gender as factual gender . 2.2 Circuit Discovery and Neuron Attribution A circuit is defined in the field of mechanistic interpretability as a computational subgraph with distinct functionality ( 44 ) . Circuits can be interpreted at different levels of granularity and across different components. The nodes of the subgraph represent model components, including neurons, attention heads, and embeddings, while the edges symbolize interactions between these components, such as residual connections, projections, or attention mechanisms ( 45 ; 11 ) . Previous works have identified the circuits relevant for the production of gendered outputs, using methods such as causal mediation analysis and activation/attribution patching ( 10 ; 31 ; 7 ) . Recent efforts at benchmarking and standardizing mechanistic interpretability methods have led to a predominant interest in edge-based circuits ( 33 ; 15 ; 20 ) . Building on this work, we apply circuit analysis to interpret gender. An adjacent form of interpretability work comes in the form of neuron attribution – i.e., identifying task-relevant hidden FFN neurons. Several works have pinpointed the importance of neurons in storing knowledge ( 12 ; 17 ; 47 ) . Different methods have been proposed to calculate the importance of a neuron to a prediction, from gradient-based ( 12 ) to attribution-based (or static) methods ( 46 ) . A small subset of neurons can play a critical role in several model capabilities, such as language competence ( 13 ) , factual knowledge ( 12 ; 9 ) , and linguistic phenomena ( 35 ) . These works also show that simple ablation of subsets of relevant neurons impacts model behavior. Of note to the present work, 29 ; 23 ; 47 identify neurons relevant for the production of socially and gender biased outputs (respectively), and evaluate neuron ablation as a debiasing technique. In our work, we use neuron ablation both to test for gender debiasing and to further our understanding of gender circuits. 2.3 Impact of Gender Debiasing on Factual Gender The entanglement of gender bias and factual gender can be observed in model representations ( 28 ) and feature vectors ( 14 ) . This entanglement raises the concern that gender debiasing methods might negatively affect a model’s knowledge of factual gender. 48 introduce a language modeling dataset that aims to measure performance on gendered instances, and find that bias mitigation methods