Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations
Jiho Jin Woosung Kang Junho Myung Alice Oh KAIST { jinjh0123 , wskang , junho00211 }@kaist.ac.kr, [email protected] · 2025
Read Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations by Jiho Jin Woosung Kang Junho Myung Alice Oh KAIST { jinjh0123 , wskang , junho00211 }@kaist.ac.kr, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06987.txt.
Description
Jiho Jin and co-authors build a benchmark for social bias in long-form text generation, in this 2025 paper. Most bias tests use short question answering, which can miss bias that seeps into stories. Their benchmark, BBG, adapts a known question-based set by asking models to continue story prompts instead. It runs in English and Korean across ten large language models.
The team counts neutral versus biased continuations and compares scores with the multiple-choice original. The two methods often disagreed, sometimes sharply. A model that looked fair on questions could still drift into biased story lines. The authors argue that generation and question tests measure different things. Bias audits, they say, need long-form checks before deployment claims are trusted.
Catalog record
| CREATOR | Jiho Jin Woosung Kang Junho Myung Alice Oh KAIST { jinjh0123 , wskang , junho00211 }@kaist.ac.kr, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06987