Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios

Bryan Chen Zhengyu Tan, Roy Ka-Wei Lee, [email protected] · 2025

Read Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios by Bryan Chen Zhengyu Tan, Roy Ka-Wei Lee, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01532.txt.

Description

Bryan Tan and Roy Lee probe implicit bias in persona prompted language models in this 2025 paper. When a model is told to answer as a person of a given age, race, or politics, its replies can shift in quality. The authors measure those shifts two ways: semantic distance between replies, and a judged win rate for reply quality. They ran five models through one hundred social scenarios across nine demographic axes.

A default persona emerged: replies drifted toward the voice of a middle aged, able bodied, native born, Caucasian, atheist man with centrist views. Some demographic prompts drew lower quality replies. Power gaps widened the spread of both meaning and quality across groups. The authors argue that bias grows exactly where stakes are uneven. They offer the framework as a way to expose such drift before deployment.

Catalog record

CREATORBryan Chen Zhengyu Tan, Roy Ka-Wei Lee, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01532

Browse the full library · About the Tower of Babel