A Scalable Entity-Based Framework for Auditing Bias in LLMs
Akram Elbouanani, Aboubacar Tuo, Adrian Popescu · 2026
Read A Scalable Entity-Based Framework for Auditing Bias in LLMs by Akram Elbouanani, Aboubacar Tuo, Adrian Popescu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12374.txt.
Introduction
Akram Elbouanani, Aboubacar Tuo, and Adrian Popescu take on the task of bias checks in large language models. Past checks face a hard trade off. Fake prompts give tight control but do not match real use. Real tasks match real use but lack scale and rigor. The field needs a way to get both at once.
The team builds a framework that uses named entities as probes. Names of people, places, and groups can be swapped in a planned way. That swap lets them test how model behavior shifts across groups. The design aims to be large in scale and firm in method. It seeks gaps in how a model treats distinct sets of entities.
Catalog record
| CREATOR | Akram Elbouanani, Aboubacar Tuo, Adrian Popescu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Scalable Entity-Based Framework for Auditing Bias in LLMs Akram Elbouanani, Aboubacar Tuo, Adrian Popescu Abstract Existing approaches to bias evaluation in large language models (LLMs) trade ecological validity for statistical control, relying on artificial prompts that poorly reflect real-world use, or on naturalistic tasks that lack scale and rigor. We introduce a scalable bias-auditing framework using named entities as probes to measure structural disparities in model behavior. We show that synthetic data reliably reproduces bias patterns observed in natural text, enabling large-scale analysis. Using this approach, we conduct the largest bias audit to date, comprising 1.9 1.9 billion data points across multiple entity types, tasks, languages, models, and prompting strategies. Our results reveal systematic biases: models penalize right-wing politicians, favor left-wing politicians, prefer Western and wealthy nations over the Global South, favor Western companies, and penalize firms in the defense and pharmaceutical sectors. While instruction tuning reduces bias, increasing model scale amplifies it, and prompting in Chinese or Russian does not attenuate Western-aligned preferences. These results indicate that LLMs should undergo rigorous auditing before deployment in high-stakes applications. 1 Introduction LLMs are now integral to decision-making tools across diverse sectors 26 . While their predictive capabilities enable rapid deployment with minimal domain expertise 39 , internalized biases that manifest in varying downstream tasks 21 ; 59 can compromise their reliability. Addressing these fairness issues is critical for high-stakes domains such as politics 31 , geopolitics 41 ; 44 , and economics 3 . Figure 1: Entity-level audit results aggregated across tasks/languages/models/prompts that reflect LLM outputs under sentence templates with light supporting evidence. They should not be interpreted as ground-truth properties or normative judgments about entities. We use cross-task aggregation for compact visualization. Finer-grained results (§ 4 ) represent the primary evidence of the study. However, quantifying bias in realistic environments remains an open and evolving challenge. Prior analyses emphasize that “bias” in NLP is a multifaceted phenomenon tied to power, social meaning, and measurement choices 5 , and that large-scale text corpora can encode and amplify social and representational patterns 4 . In practice, auditing methods prioritize experimental control over ecological validity. Political compasses and template-based benchmarks 33 force models into artificial, constrained interaction formats that do not reflect practical LLM use 22 . These abstractions imply that bias is a stable property, overlooking its contextual dependence. Such evaluations may capture surface-level preferences rather than behavior that meaningfully impacts downstream applications, limiting their external validity 42 . At the same time, auditing bias in real-world data is challenging because few datasets exist that allow systematic variation of entities, making it difficult to isolate bias from factors like topic, syntax, or narrative context 19 . This trade-off between control and realism underscores the need for audit methods that maintain task fidelity while enabling the systematic isolation of bias 40 . We propose a flexible, entity-based framework for bias analysis to reconcile statistical rigor with ecological validity. Following 5 , we operationalize entity bias as systematic, evidence-light differences in model outputs attributable to entity identity under controlled templates. Our measurements capture model priors and associations rather than ground-truth properties of entities. Therefore, they should not be interpreted as objective assessments of entities. By systematically varying entities within consistent task contexts, we enable generalizable bias measurements across domains and LLM configurations. We rely on data generation to boost scalability. Although synthetic data may abstract away some linguistic nuance nadua\vtop{\halign{#\cr\hbox{}\crcr}} , its generation allows us to construct sentence templates that can be instantiated with different entities, isolating bias-related factors without introducing counterfactual or semantically inconsistent content. We compare bias measurements derived from synthetic texts with those from real datasets, and we find a strong correlation. Supported by this validation, we conduct a large-scale audit comprising approximately $1.9$ billion generated data points. It covers three entity types, twelve tasks, three languages, sixteen models, and four prompting variants. This setting allows us to move beyond anecdotal evidence of bias to an understanding of its origins and mechanisms. We address three research questions: • RQ1: Can synthetic data serve as a valid proxy for auditing bias in realistic downstream tasks? We examine the correlation of structural bias signals detected in the synthetic and natural data. • RQ2: How does bias manifest across different domains and tasks? Using a large-scale dataset, we investigate the variation of bias across applications and contexts. • RQ3: How stable are these disparities across languages, model architectures, and prompting variations? We quantify the robustness and generality of bias patterns at scale. Figure~ 1 provides an illustrative snapshot of our audit, visualizing aggregated entity priors for countries under standardized, evidence-light templates. The map is a descriptive summary that highlights a salient pattern: country identity alone can systematically modulate model predictions in otherwise comparable inputs. These tendencies motivate us to test whether such variations are robust across settings and to examine why auditing is critical before deploying LLMs. 2 Related Work Bias evaluation in NLP has traditionally relied on fixed benchmarks designed to test specific social stereotypes. Datasets such as CrowS-Pairs~ 33 , StereoSet~ 32 , BOLD~ 14 , HolisticBias~ 49 , and BBQ~ 35 use paired sentences or question-answering templates to detect performance disparities in sensitive contexts~ 47 ; 48 ; 34 . While these benchmarks were instrumental in highlighting the existence of bias, their reliance on rigid, predefined templates limits their ability to capture the complexity of bias in varied contexts. Furthermore, critics suggest that template-based methods can suffer from poor construct validity and may not predict model behavior in real-world scenarios~ 6 ; 46 . Other studies use psychometric-style evaluations, such as questionnaires or political compass tests, to map LLMs onto ideological spectra~ 22 ; 31 ; 36 . While they provide high-level insights into model alignment, they have a reduced practical utility ~ 42 . A model's response to a political survey does not necessarily predict how it will summarize news articles or assess financial risks related to companies~ 50 ; 15 . This gap underscores the need for evaluation frameworks that directly measure bias in downstream tasks that use LLMs. Entity-centric analysis offers a pathway to such task-specific evaluation. Prior work has observed that replacing entity names in a prompt can drastically alter model outputs~ 7 ; 43 . For instance, toxicity detection systems~ 20 and sentiment classifiers are sensitive to specific names or geopolitical entities~ 52 ; 10 . However, these studies have been constrained by data availability and have focused on single domains or small entity sets. Large-scale, statistically robust studies have been infeasible due to the lack of extensive, annotated counterfactual datasets~ 42 ; 45 . Generation offers a solution to data scarcity. Recent work has employed synthetic text to study social stereotyping and political framing~ 53 ; 9 . However, because synthetic texts lack the semantic richness of human language, they may lead to oversimplified or unrepresentative findings~ 54 ; 28 ; 24 , thereby questioning the validity of the approach. We distinguish our work by validating the fidelity of synthetic bias measurements against real-world benchmarks. 3 Methodology We introduce a scalable, generalizable, and realistic framework for auditing LLMs. It leverages a robustness requirement: an LLM's prediction in a specific context should be consistent across