Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition
Kyuhee Kim, Sangah Lee · 2025
Read Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition by Kyuhee Kim, Sangah Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04014.txt.
Description
Kyuhee Kim and Sangah Lee published a research paper in 2025 on a test bench that grades language models on cultural sense, using Korean superstitions as the probe. Models now advise people across cultures, so how they handle local beliefs matters. Their bench, Nunchi-Bench, holds 247 questions across thirty-one topics.
Questions check bare facts, sound advice, and reading of social scenes. Models were tested in Korean and in English with custom scoring for cultural nuance. They mostly knew the facts but fumbled when asked to apply them in real scenes. Naming the cultural frame outright helped more than just asking in Korean. The bench and its scoreboard are public for other teams.
Catalog record
| CREATOR | Kyuhee Kim, Sangah Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04014