Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition

Kyuhee Kim, Sangah Lee · 2025

Read Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition by Kyuhee Kim, Sangah Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04014.txt.

Description

Kyuhee Kim and Sangah Lee published a research paper in 2025 on a test bench that grades language models on cultural sense, using Korean superstitions as the probe. Models now advise people across cultures, so how they handle local beliefs matters. Their bench, Nunchi-Bench, holds 247 questions across thirty-one topics.

Questions check bare facts, sound advice, and reading of social scenes. Models were tested in Korean and in English with custom scoring for cultural nuance. They mostly knew the facts but fumbled when asked to apply them in real scenes. Naming the cultural frame outright helped more than just asking in Korean. The bench and its scoreboard are public for other teams.

Catalog record

CREATORKyuhee Kim, Sangah Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04014

Browse the full library · About the Tower of Babel