Attribution Bias in Large Language Models
Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black · 2026
Read Attribution Bias in Large Language Models by Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05224.txt.
Introduction
Berman and three colleagues give AttriBench, the first even quote credit benchmark. Evening out fame and groups enables fair study of group bias in credit. Across prompt settings, 11 widely used models still struggle with credit, with large score gaps between race, gender, and mixed groups. They also study silence, where models omit credit fully despite access to writer info. Silence is common and uneven across groups, showing biases missed by standard score metrics.
Catalog record
| CREATOR | Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46750 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attribution Bias in Large Language Models Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black Abstract As Large Language Models (LLMs) are increasingly used to support search and information retrieval, it is critical that they accurately attribute content to its original authors. In this work, we introduce AttriBench , the first fame- and demographically-balanced quote attribution benchmark dataset. Through explicitly balancing author fame and demographics, AttriBench enables controlled investigation of demographic bias in quote attribution. Using this dataset, we evaluate 11 widely used LLMs across different prompt settings and find that quote attribution remains a challenging task even for frontier models. We observe large and systematic disparities in attribution accuracy between race, gender, and intersectional groups. We further introduce and investigate suppression , a distinct failure mode in which models omit attribution entirely, even when the model has access to authorship information. We find that suppression is widespread and unevenly distributed across demographic groups, revealing systematic biases not captured by standard accuracy metrics. Our results position quote attribution as a benchmark for representational fairness in LLMs. One of the advantages of being disorganized is that one is always having surprising discoveries. I’ve found, in my own writing, that a little hatred, keenly directed, is a useful thing. direct prompt: Please identify the author of the following quote. A.A. Milne Alice Walker indirect prompt: Briefly summarize the context in which the following quote was written. This humorous line by A.A. Milne wryly reflects on everyday habits and the small, ironic benefits of personal untidiness. This line comes from an essay by a writer discussing how channeling focused negative emotion—specifically hatred—can sharpen and energize creative writing. ✓ ✓ ✓ ✗ Figure 1: Example of suppression in quote attribution. GPT-5.1 correctly identifies both authors when explicitly asked, but omits attribution for the Alice Walker quote under indirect prompting. Both authors have similar fame, as measured by Google Search hits. 1 Introduction The rapid adoption of Large Language Models (LLMs) has transformed how users access and obtain information, particularly in scholarly workflows ( 24 ; 20 ) . As LLMs become foundational in domains such as education and research ( 10 ; 25 ) , it is critical that they accurately attribute content to its original authors. Quote attribution is not only a question of factual correctness, but also of intellectual credit: attribution failure obscures whose ideas are recognized and circulated . In this work, we introduce a new benchmark dataset and analysis framework for studying quote attribution in LLMs. This benchmark ( AttriBench ) enables controlled evaluation of representational disparities in attribution. Because attribution performance is strongly correlated with author fame (see Appendix A.1 ), we construct a dataset that explicitly balances both demographics and fame, allowing us to isolate disparities in model behavior from differences in author prominence. We then evaluate 11 LLMs on AttriBench , including frontier models GPT-5.1 ( 15 ) , claude-4.6-sonnet ( 3 ) , and Kimi-K2.5 ( 9 ) , in both zero-shot and retrieval-augmented generation (RAG) settings. We find that attribution remains a difficult task even for strong models, and observe systematic disparities across demographic groups favoring White and male authors that persist across models and prompting strategies. Further, we introduce a complementary and underexplored failure mode, suppression (shown in Figure 1 ), defined as the model’s tendency to selectively omit attribution entirely, even when the correct answer is made explicitly available. Unlike misattribution ( 19 ; 2 ) , which is a measure of model accuracy, suppression is a distinct measure of a model’s decision to attribute at all. Such author omission can systematically affect the visibility of authors to users, with implications for representational fairness in LLMs. Our main contributions are: (1) We introduce a fame-controlled, demographically annotated quote attribution benchmark dataset, enabling controlled evaluation across race, gender, and intersectional groups. Unlike prior datasets, which risk confounding demographic effects with author prominence, our dataset explicitly matches authors across groups by fame, isolating disparities in attribution behavior from the effects of popularity. To our knowledge, this is the first quote attribution dataset to include both demographic labeling and explicit fame control. (2) We introduce and characterize suppression as a distinct attribution failure mode, showing that attribution biases can emerge not only through incorrect naming, but also through selective omission. (3) We evaluate LLM attribution across multiple prompt framings and show both low overall performance and significant disparities across race, gender, and intersectional groups: White authors, and particularly White males, experience consistently higher accuracy and lower suppression rates across all 11 LLM models tested. 2 Related work Quote attribution tasks and datasets . Prior NLP work typically defines quote attribution as a closed-world task, where the model selects the speaker from a fixed set of candidate characters explicitly provided in the model’s context. Much of this literature focuses on studying attribution in structured literary dialogue ( 11 ; 23 ; 29 ) . We look at a different problem of open-world author attribution from isolated quotes, without predefined, restricted lists of candidates, allowing us to realistically evaluate author attribution in practice. Additionally, rather than solely evaluating which authors are selected, we also measure whether attribution occurs at all, allowing us to identify suppression as a distinct failure mode and capture cases where models omit attribution entirely rather than incorrectly attributing. While datasets of raw quote-author pairs exist ( 28 ; 21 ; 22 ) , our dataset is the first to be fame- and demographically-balanced and labeled, allowing us to evaluate whether models disproportionately fail to recognize authors from certain demographic groups, even under fame-controlled comparisons. Attribution and citation in language models . Recent work has explored attribution in LLMs in the context of citation retrieval. CiteME ( 18 ) asks models to identify the source paper referenced by a claim excerpt, and shows that even strong contemporary systems perform poorly on this task. 1 show that including authorship information with source documents in RAG pipelines can significantly improve the attribution quality of LLMs in citing relevant sources. In doing so, they demonstrate a bias towards human-authored vs. LLM-generated sources. However, they do not analyze author demographic attributes or fame as factors of attribution bias, unlike our work. Prior work by 8 shows that LLMs can reinforce existing gender imbalances in scholarly recognition. When given pools of citations with author names perturbed for clearly gendered names and asked to select relevant references, LLMs demonstrate a preference for male-authored references. As this work focuses exclusively on retrieval, notions of suppression cannot be explored in this context, and therefore, we propose a new dataset and evaluation framework for studying attribution and suppression in open-ended settings. Subgroup performance disparity evaluation . Many influential benchmarks measure whether LLM behavior differs across demographic groups such as race, gender, or religion. StereoSet ( 12 ) evaluates stereotypical associations in pretrained language models. BBQ ( 16 ) evaluates how social biases affect question answering under different levels of informed contexts, and BOLD ( 5 ) evaluates demographic biases in open-ended generation. Here, we introduce a new quote attribution dataset with demographic labels that enables novel evaluation of attribution behavior across groups, extending subgroup disparity evaluation into a setting where harms surface through differential