Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts
Huanping Xiao · 2026
Read Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts by Huanping Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30152.txt.
Introduction
Huanping Xiao, writing alone, separates two kinds of gender signal inside contextual language models. In languages like Spanish, nouns carry grammatical gender by rule. Models also absorb social gender stereotypes from their training text. The two signals become tangled in the learned representations. Debiasing methods built for English ignore that tangle entirely.
Xiao builds balanced datasets of inanimate nouns for the study. Both controlled sentence templates and natural Wikipedia contexts are used. Three estimators then extract a grammatical gender direction from embeddings. They are a centroid method, a support vector machine, and linear discriminant analysis. New dual-objective metrics weigh two goals against each other. One goal suppresses grammatical gender leaking into inanimate nouns. The other preserves real semantic gender for occupation terms. Unweighted controlled contexts yield the purest gender direction. The simple centroid estimator beats the trained rival methods.
Catalog record
| CREATOR | Huanping Xiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30152
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts Huanping Xiao Abstract Abstract Contextual language models conflate grammatical gender and social semantic bias in gendered languages such as Spanish. Existing gender debiasing approaches only operate on static word embeddings leaving contextual representations unexplored for this two dimensional gender disentanglement. To address the this issue, we make the first attempt to disentangle grammatical gender from semantic contamination for contextual embeddings. We construct both controlled templates and natural Wikipedia contexts to build balanced datasets of inanimate nouns, and design a framework equipped with centroid, Support Vector Machine (SVM) and Linear Discriminant Analysis (LDA) gender direction estimators as well as contamination-aware weighting strategies. A set of dual-objective evaluation metrics is proposed to balance the suppression of grammatical gender leakage on inanimate nouns and the preservation of semantic gender distinctions for occupation terms. The results reveal that unweighted controlled contexts yield the purest grammatical gender direction, and the centroid estimator achieves better performance than discriminative baselines. 1 Introduction Pre-trained contextual language models have become foundational tools for multilingual natural language processing [ 1 ] . Extensive research [ 2 , 3 , 4 ] has confirmed that these models encode pervasive social gender stereotypes within their word and sentence representations, motivating a wide range of embedding debiasing techniques based on orthogonal projection [ 2 , 5 , 6 ] . Most existing bias mitigation approaches are designed for English [ 7 , 8 , 9 ] , a language without grammatical gender, and treat all gender-related embedding signals as unwanted social bias [ 5 ] . This design creates critical limitations when applied to grammatically gendered languages [ 10 ] like Spanish and French, where two distinct gender signals coexist: mandatory morphosyntactic grammatical gender [ 11 ] and socially constructed semantic gender bias. Grammatical gender is an inherent linguistic property of inanimate nouns in gendered languages. Nouns are fixed as masculine or feminine and require consistent agreement with articles, adjectives and pronouns. Distributional co-occurrence patterns between nouns and their modifiers make grammatical gender cues automatically detectable in contextual embeddings. Unfortunately, grammatical gender signals are inherently entangled with social semantic gender associations learned from real-world corpora [ 12 ] . Prior psycholinguistic [ 13 , 14 ] and computational [ 15 , 16 , 17 , 18 ] studies further prove that even semantically neutral inanimate nouns absorb contextual gendered descriptions from natural text, leading to severe semantic contamination. While a small body of literatures [ 5 , 19 ] have proposed disentanglement pipelines to separate the two gender dimensions, all these existing methods [ 5 , 19 , 20 ] rely exclusively on static word embeddings trained from unconstrained natural corpus contexts, and no prior work has attempted such disentanglement within contextual pre-trained language models. To address these research gaps, we systematically investigate how to construct the purest grammatical gender directions within contextual word embeddings, and further establish a unified analytical framework for disentangling grammatical gender signals from semantic gender bias. We first build balanced datasets of Spanish and French inanimate nouns, paired with two types of context resources: controlled templates for grammatical signal extraction, and natural sentences for contamination quantification. To generate robust grammatical gender axes, we implement centroid-based [ 21 ] , SVM-based [ 22 ] and LDA-based [ 5 ] estimators, and further introduce mean-normalized and median-normalized weighting strategies to down-weight semantically contaminated nouns. We design three complementary evaluation metrics to balance two conflicting goals: suppressing redundant grammatical gender leakage on inanimate nouns while preserving semantic gender distinctions for occupation vocabulary. We adopt the harmonic trade-off score as our primary comprehensive metric to compare all model variants. Experiments on four monolingual and multilingual pre-trained encoders yield consistent conclusions: controlled unweighted contexts produce the purest grammatical gender directions, and the simple centroid estimator outperforms discriminative alternatives. Our orthogonal projection only removes components aligned with grammatical gender axes without disrupting general lexical semantic structure, as validated by supplementary semantic stability tests. The main contributions of this paper are summarized as follows: 1. We pioneer grammatical-semantic gender disentanglement for contextual language models, while all prior comparable work is limited to static word embeddings. We build paired controlled template and natural Wikipedia datasets for Spanish/French inanimate nouns to verify that controlled templates eliminate corpus noise and extract cleaner grammatical gender signals. 2. We are the first to formalize semantic contamination in inanimate noun representations for contextual language models and design contamination-aware strategies to suppress biased distributional artifacts. 3. Drawing on WEAT [ 4 ] , SEAT [ 23 ] and hard debiasing [ 21 ] , we propose a novel dual-objective evaluation framework with leakage reduction, semantic preservation and harmonic trade-off metrics to resolve the lack of balanced evaluation standards for gendered language debiasing. In the following sections, we first review related work on embedding bias, gender signal disentanglement and multilingual bias evaluation. Next, we detail our datasets, context construction and pre-trained models. We then formalize our method including gender direction estimation, contamination weighting and evaluation metrics. After presenting experimental results and ablation analyses, we discuss the core findings, limitations and future research directions. Fig 1 visualizes our unified disentanglement framework. Fig 1: Overall framework of our grammatical-semantic gender disentanglement method. 2 Related Work 2.1 Gender Bias in Embeddings Early works [ 21 , 24 ] firstly revealed that static word embedding techniques such as Word2Vec and GloVe implicitly encode human-like social gender stereotypes within dense vector representations. These classical studies observed that occupation-related words like engineer , doctor were geometrically closer to masculine pronouns in the embedding space, while words such as nurse , secretary were tightly clustered with feminine lexical items [ 25 ] , which replicated the long-standing gender prejudice inherited from training corpora. To mitigate such unwanted bias, researchers pioneered projection-based debiasing pipelines [ 6 ] : these methods first identify a low-dimensional gender subspace via principal component analysis from gender-pair word vectors, then project all word embeddings onto the orthogonal complement of this bias subspace to erase gender directional information while striving to preserve the original semantic utility of word vectors. Despite their wide adoption, these early static debiasing strategies suffer from inherent limitations: they apply a globally fixed bias elimination transformation to every token, failing to distinguish between contextually essential gender attributes (e.g., gender information for nouns referring to specific individuals) and harmful stereotypical bias, which inevitably leads to over-debiasing and sacrifices downstream task performance to a certain extent. With the prevalence of pre-trained contextual language models, later studies [ 3 , 23 , 26 ] extended gender bias investigation from static word embeddings to contextual token embeddings and sentence-level semantic representations. Different from fixed vector outputs of static embedding methods, BERT-style models generate dynamic token representations conditioned on surrounding textual context, which introduces more complicated bias distribution patterns. Prior studies [ 27 , 28 ] have found evidence that gender bias persists in transformer-based contextual encoders such as BERT, while contextualized bias measurements can vary substantially depending on sentence context and evaluation method. 2.2 Disentanglement of Grammatical and Semantic Gender in Word Representations Gendered languages require a strict distinction between grammatical gender and semantic gender, as these two types of gender signals are inherently entangled