JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs

Taihei Shiotani, Masahiro Kaneko, Ayana Niwa, Yuki Maruyama, Daisuke Oba, Masanari Ohi, Naoaki Okazaki · 2026

Read JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs by Taihei Shiotani, Masahiro Kaneko, Ayana Niwa, Yuki Maruyama, Daisuke Oba, Masanari Ohi, Naoaki Okazaki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20581.txt.

Description

Shiotani and co-authors build JUBAKU, a test for bias in Japanese language models. Most bias tests are mere translations of English ones. They miss local norms on rank, dialect, and gender roles. Native speakers hand wrote dialogue scenes across ten cultural themes. Nine Japanese models scored an average of 23 percent. That sits well below the 50 percent random mark. Human readers scored 91 percent, which shows the test is sound.

Catalog record

CREATORTaihei Shiotani, Masahiro Kaneko, Ayana Niwa, Yuki Maruyama, Daisuke Oba, Masanari Ohi, Naoaki Okazaki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26022
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20581

Browse the full library · About the Tower of Babel