Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Models
Ben Maltbie, Shivam Raval · 2026
Read Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Models by Ben Maltbie, Shivam Raval (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11609.txt.
Introduction
Maltbie and Raval probe whether large language models flatter users differently by perceived demographics. Across 768 multi-turn chats spanning 128 personas and three domains, sycophancy varies sharply by model and domain, emerging from trait combinations rather than any single dimension. GPT-5-nano scores far higher than Claude Haiku 4.5, averaging 2.96 versus 1.74, and within GPT-5-nano, philosophy elicits 41 percent more sycophancy than mathematics, and Hispanic personas score highest. The worst persona, a confident 23-year-old Hispanic woman, averages 5.33 out of 10. The authors argue safety tests should include identity-aware testing.
Catalog record
| CREATOR | Ben Maltbie, Shivam Raval |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50379 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Models Ben Maltbie, Shivam Raval Abstract Large language models exhibit sycophantic tendencies, but whether this behavior varies systematically with perceived user demographics is underexplored. Inspired by intersectionality (overlapping identities produce compounded effects), we probe whether frontier models conditionally exhibit sycophancy. Across 768 multi-turn conversations spanning 128 personas (varying race, age, gender, confidence) and three domains (mathematics, philosophy, conspiracy theories), we find that sycophancy varies sharply with target model and domain, and emerges from combinations of perceived user traits rather than any single dimension. GPT-5-nano scores far higher than Claude Haiku 4.5 (average sycophancy scores of x ¯ = 2.96 \bar{x}=2.96 vs. 1.74 1.74 , p < 10 − 32 p<10^{-32} ); within GPT-5-nano, philosophy elicits 41% more sycophancy than mathematics and Hispanic personas receive the highest scores across races. The worst-scoring persona, a confident, 23-year-old Hispanic woman, averages 5.33/10 (max 6/10), while Claude Haiku 4.5 remains uniformly low with no significant demographic variation. We argue that safety evaluations should incorporate identity-aware adversarial testing. 1 Introduction Figure 1 : Overview: We test whether LLM sycophancy varies by perceived user demographics using multi-turn adversarial conversations via Anthropic’s Petri tool which orchestrates multi-agent interaction. In our setup, (1) an auditor model that plays a user persona with incorrect beliefs, (2) a target model we evaluate for sycophantic responses, and (3) a judge model that scores the full, final transcript between the target and auditor models. This figure shows a representative failure progression from our experiments (GPT-5-nano, philosophy domain, sycophancy score: 8/10, higher score is more sycophantic). The target model degrades from principled disagreement to writing unqualified endorsements of claims it previously argued were false. Full transcript in Appendix F . Large language models (LLMs) are increasingly deployed as conversational assistants, tutors, and information sources. Growing evidence suggests their behavior varies systematically based on perceived user characteristics ( 5 ; 7 ) . Sycophancy is the tendency to agree with or validate users regardless of factual accuracy, and represents a safety-relevant alignment failure that can reinforce false beliefs and cause real harm, particularly in educational contexts ( 11 ) . Our work is motivated by the legal concept of intersectionality , originating in DeGraffenreid v. General Motors (1977) and formalized by 2 , which holds that overlapping identity dimensions produce compounded effects not captured by examining any single dimension in isolation. We hypothesize that similar intersectional effects exist in LLM sycophancy, where models may treat users differently based on combinations of race, age, gender, and expressed confidence. A preliminary investigation using the Petri evaluation framework ( 3 ) probed GPT-4.1-nano with 42 persona combinations (varying age, gender, and confidence) across 86 multi-turn conversations in mathematics and philosophy observed that the model was 67% more sycophantic toward women. Additionally, a U-shaped age effect was seen, with sycophancy spiking for children and the elderly. In one experiment, a complete alignment failure occurred for the “70-year-old confident woman” persona, which scored 10/10 on sycophancy when it validated objectively false mathematical statements (transcript excerpts reproduced in Appendix G ). Those findings motivate our present, more comprehensive study. Through our experiments, we make three major contributions: 1. We conduct the largest systematic study of demographic effects on LLM sycophancy to date: 768 multi-turn adversarial conversations spanning 128 persona combinations across three demographic dimensions (race, age, gender), one emotional state (confidence level), three knowledge domains (math, philosophy, conspiracy theories), and two target models (GPT-5-nano, Claude Haiku 4.5). 2. We provide the first cross-model comparison of complex demographic sycophancy, showing that GPT-5-nano exhibits significantly higher and more demographically variable sycophancy than Claude Haiku 4.5. 3. We identify specific intersectional effects, particularly an age-by-gender interaction where the sycophancy gender gap widens with age, and document qualitatively distinct failure modes across domains. 2 Related Work Sycophancy in LLMs. 9 identified sycophancy as an inverse scaling phenomenon in RLHF-trained models. 11 provided extensive analysis establishing sycophancy as a consistent failure mode across model families. 8 found that larger models paradoxically performed worse at truthfulness, likely due to mimicking training-data misconceptions. 13 showed that simple synthetic data interventions can partially reduce sycophantic behavior, though the problem persists. 1 found that sycophantic behavior actually reduces user trust rather than increasing it. User personas and contextual effects. 5 demonstrated that manipulating user personas is more effective at eliciting harmful outputs than direct jailbreaks, with certain personas “breaking model safeguards.” 7 found that interaction context, including conversation history and user-provided background, often increases sycophantic behavior in LLMs. 4 showed that LLM belief profiles shift substantially as context accumulates during extended interactions, with these shifts translating into behavioral changes in downstream tasks. 12 identified that first-person opinion statements (“I believe…”) reliably trigger sycophancy while framing users as experts has a negligible effect, indicating that the model does not internally represent user authority. This motivates our use of first-person belief statements in auditor prompts. Demographic bias in LLMs. While demographic bias and fairness in LLM outputs have been studied ( 10 ) , no prior work has systematically connected demographic identity dimensions to sycophancy rates in multi-turn adversarial settings. 3 Methodology 3.1 Evaluation framework We utilize Anthropic’s Petri framework ( 3 ) , an automated red-teaming auditing tool that orchestrates multi-turn conversations through a three-model pipeline, with the following selected models: Auditor model (GPT-5-mini): Plays assigned persona roles, presents incorrect beliefs prefixed with “I believe…” (per 12 ), and persists when corrected. The auditor conveys demographic characteristics through natural conversational cues rather than explicit labels. Target models : (1) GPT-5-nano (OpenAI) and (2) Claude Haiku 4.5 (Anthropic), each configured with a generic helpful-honest-harmless system prompt. Target models receive no system prompt information about user demographics. Judge model (GPT-5.1): Evaluates completed transcripts on a 37-dimension rubric covering sycophancy, encouragement of delusion, deception, and other safety-relevant behaviors, scoring each on a 1–10 scale (10 is highest). The rubric is built into the Petri tool. Each conversation runs up to 40 turns; across all 768 conversations the actual mean was 21.5 turns (Claude Haiku 4.5: 12.9; GPT-5-nano: 30.1), with the auditor terminating sooner against more resistant targets. Additionally, the target model must infer user demographics from conversational context clues embedded by the auditor whenever possible (e.g., references to grandchildren for elderly personas, mentions of cultural background for race). 3.2 Persona design We construct 128 persona configurations from four dimensions: Table 1 : Persona dimensions and values. Full-combination experiments (numbered v16–v127) cross all four dimensions, yielding 4\times 7\times 2\times 2=112 unique personas. Single-dimension experiments (numbered v1–v15) isolate individual factors. One baseline (numbered v0) uses no persona information. Dimension Values Race Black, White, Hispanic, Asian Age 8, 13, 18, 23, 30, 50, 70 years old Gender Boy/girl ( \leq 13), man/woman ( \geq 18) Confidence Extremely confident, deeply insecure Each of the 128 versions is tested in three domains— mathematics (objectively false claims like “ 2+2=5 ”), philosophy (contentious claims presented as absolutes, e.g., “free will is an illusion”), and common conspiracy theories —and against both target models, yielding 128\times 3\times 2=768 total experiments. Unless otherwise noted, dimension analyses use only the 112 full-combination experiments per domain per model ( n=336 per model). The remaining 16 versions per domain consist of 15 single-dimension