Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments

Divyanshu KumarEnkrypt AI [email protected], Ishita GuptaEnkrypt AI [email protected] Nitin Aravind BirurEnkrypt AI [email protected] Tanay BaswaEnkrypt AI [email protected] Sahil AgarwalEnkrypt AI [email protected] Prashanth HarshangiEnkrypt AI [email protected] · 2026

Read Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments by Divyanshu KumarEnkrypt AI [email protected], Ishita GuptaEnkrypt AI [email protected] Nitin Aravind BirurEnkrypt AI [email protected] Tanay BaswaEnkrypt AI [email protected] Sahil AgarwalEnkrypt AI [email protected] Prashanth HarshangiEnkrypt AI [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02669.txt.

Introduction

Kumar and five colleagues show that language-model bias depends on how it is measured, auditing seven commercial and open-weight models with about 45,000 prompts across nine bias types and seven tasks. On explicit probes models counter stereotypes, but on implicit ones they reproduce them, with gaps up to 0.43 for the same model and groups. Under-studied axes such as caste, language and geography show the strongest stereotyping of all.

Catalog record

CREATORDivyanshu KumarEnkrypt AI [email protected], Ishita GuptaEnkrypt AI [email protected] Nitin Aravind BirurEnkrypt AI [email protected] Tanay BaswaEnkrypt AI [email protected] Sahil AgarwalEnkrypt AI [email protected] Prashanth HarshangiEnkrypt AI [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47018
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments Divyanshu KumarEnkrypt AI [email protected], Ishita GuptaEnkrypt AI [email protected] Nitin Aravind BirurEnkrypt AI [email protected] Tanay BaswaEnkrypt AI [email protected] Sahil AgarwalEnkrypt AI [email protected] Prashanth HarshangiEnkrypt AI [email protected] Abstract How biased is a language model? The answer depends on how you ask. A model that refuses to choose between castes for a leadership role will, in a fill-in-the-blank task, reliably associate upper castes with purity and lower castes with lack of hygiene. Single-task benchmarks miss this because they capture only one slice of a model’s bias profile. We introduce a hierarchical taxonomy covering 9 bias types, including under-studied axes like caste, linguistic, and geographic bias, operationalized through 7 evaluation tasks that span explicit decision-making to implicit association. Auditing 7 commercial and open-weight LLMs with ~45K prompts, we find three systematic patterns. First, bias is task-dependent: models counter stereotypes on explicit probes but reproduce them on implicit ones, with Stereotype Score divergences up to 0.43 between task types for the same model and identity groups. Second, safety alignment is asymmetric: models refuse to assign negative traits to marginalized groups, but freely associate positive traits with privileged ones. Third, under-studied bias axes show the strongest stereotyping across all models, suggesting alignment effort tracks benchmark coverage rather than harm severity. These results demonstrate that single-benchmark audits systematically mischaracterize LLM bias and that current alignment practices mask representational harm rather than mitigating it. * * footnotetext: These authors contributed equally 1 Introduction Ask a contemporary LLM: “Should a Brahmin or a Chamar person lead the temple committee?” It will refuse to choose, responding that caste should not determine leadership. Now present the same model with “The ___ person was considered pure and sacred” and the options Brahmin, Chamar, Gond, Mahar . It fills the blank with Brahmin . Ask it to continue “The Chamar person entered the temple and…” with negative sentiment. It complies. The bias has not been removed. It has been redirected . Safety alignment teaches the model to recognize and refuse explicit discrimination, but the underlying stereotypical associations persist, surfacing wherever the evaluation task is less obviously a bias test. Models perform anti-bias behavior on tasks that resemble known bias benchmarks while reproducing the same stereotypes on tasks that do not. This observation has immediate practical consequences. If a model’s bias profile changes depending on the evaluation modality, then any single-task benchmark (StereoSet ( 28 ) , WinoBias ( 41 ) , BBQ ( 31 ) , or otherwise) captures at most one cross-section of a multi-dimensional problem. A model that passes one benchmark may fail another, not because the bias was reduced but because it was probed differently . We demonstrate this empirically across 7 LLMs, 9 bias types, and 7 evaluation tasks. Existing benchmarks have been invaluable for establishing that LLMs encode societal biases ( 28 ; 41 ; 31 ; 8 ) . But each operates within a single evaluation modality and typically covers one or two bias axes, most often gender and race. No existing work tests whether the same bias, for the same identity groups, manifests differently across task types. This is the question at the heart of this paper. We make three contributions: • An empirical audit of 7 LLMs producing three findings: (1) bias expression is task-dependent, with models counter-stereotyping on explicit tasks and reproducing stereotypes on implicit ones; (2) safety alignment asymmetrically blocks negative stereotyping while permitting positive privilege; and (3) under-studied bias axes exhibit the strongest stereotyping, suggesting alignment effort tracks benchmark coverage rather than harm severity. • A multi-task evaluation framework of 7 complementary tasks, ordered from explicit probes (decision-making) to implicit probes (fill-in-the-blank, context continuation), generating ~45K structured prompts via modular templates. • A hierarchical taxonomy covering 9 bias types, including under-studied axes like caste, linguistic, health, and geographic bias, organized across 50+ themes and 150+ topics with grounded identity attributes. 2 Related Work 2.1 Bias Benchmarks Bias evaluation in NLP has progressed through three generations, each expanding scope but retaining a single-task design. Embedding-based methods, notably WEAT ( 6 ) and its contextual extension CEAT ( 16 ) , revealed stereotypical associations in learned representations but could not assess how those associations surface in generated text. Task-specific benchmarks addressed this: StereoSet ( 28 ) tests sentence completion preferences, WinoBias ( 41 ) and WinoGender ( 34 ) probe gender bias via coreference resolution, BBQ ( 31 ) uses ambiguous QA, BOLD ( 8 ) evaluates open-ended generation, and RealToxicityPrompts ( 13 ) stress-tests toxic completions. 23 scaled evaluation to 50+ LLMs, demonstrating that neither model size nor recency reliably reduces implicit bias. Each of these benchmarks probes bias through a single task modality. Multi-metric frameworks like HELM ( 25 ) evaluate models across many capabilities but treat bias as one dimension among dozens rather than decomposing it across task types; recent surveys confirm that multi-task bias comparison remains an open gap ( 12 ) . On our explicit-to-implicit gradient (Section 4.1 ), StereoSet and WinoBias sit at the implicit end, BBQ at the explicit end, and BOLD somewhere between. Yet no study compares a model’s behavior across these positions for the same identity groups. A model may appear unbiased on one task while exhibiting strong stereotyping on another, precisely because safety alignment is calibrated to the patterns most commonly benchmarked. Our multi-task design fills this gap. 2.2 Representational Harm Across Bias Axes Generative models produce representational harm (stereotyping, erasure, or homogenization of social groups), distinct from the allocative harms studied in predictive systems ( 5 ; 1 ) . Prior work has documented such harm across individual axes: gender ( 17 ; 9 ) , race ( 18 ; 24 ) , religion ( 27 ; 35 ; 32 ) , caste ( 39 ; 36 ) , socioeconomic status ( 21 ; 2 ) , health ( 38 ) , geography ( 26 ; 30 ) , and partisan associations ( 22 ) . Intersectional analyses further show that harms compound across axes ( 20 ; 7 ; 37 ) . A consistent limitation across these studies is that each evaluates a single bias type through a single task modality. Our framework addresses this by unifying all 9 axes under a common multi-task evaluation, enabling the cross-type comparisons needed to understand how bias operates as a system rather than as isolated phenomena. 2.3 Safety Alignment and Bias RLHF and constitutional AI ( 4 ) have reduced overt toxicity in model outputs ( 10 ) , but the interaction between alignment and bias remains poorly characterized. 33 document exaggerated safety behaviours — models that over-refuse benign prompts — suggesting that alignment can create asymmetric treatment across content categories, a pattern we extend to the bias domain. 42 show that as models scale, explicit bias decreases while implicit bias increases, and that alignment suppresses overt stereotyping while leaving underlying associations intact. 3 demonstrate the same divergence using an IAT-adapted probe, finding pervasive implicit biases in models that pass explicit bias tests. 40 further show that safety guardrails are more robust on some axes than others. Our work extends this line of research in two ways. First, we operationalize the implicit-explicit divergence across

Read the complete text