IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

Paul Röttger Musashi Hinck Valentin Hofmann, Kobi Hackenburg Valentina Pyatkin Faeze Brahman Dirk Hovy · 2025

Read IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance by Paul Röttger Musashi Hinck Valentin Hofmann, Kobi Hackenburg Valentina Pyatkin Faeze Brahman Dirk Hovy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-08395.txt.

Description

Paul Röttger and six colleagues wrote a 2025 paper on bias in writing help from large language models. Millions of people now use these tools to draft text on hot topics. If a model leans to one side, it may shape how users think about that topic.

The team built IssueBench, a set of 2.49 million realistic prompts for testing this lean. The prompts come from 3.9 thousand templates and 212 political issues drawn from real use. Tests on 10 top models show that one-sided answers are common and stick around. The data set gives labs a shared way to measure the problem. It may help push models toward fairer writing help.

Catalog record

CREATORPaul Röttger Musashi Hinck Valentin Hofmann, Kobi Hackenburg Valentina Pyatkin Faeze Brahman Dirk Hovy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-08395

Browse the full library · About the Tower of Babel