JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh · 2026

Read JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors by Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26955.txt.

Introduction

Jin and seven colleagues present JuICE, a benchmark for testing whether language-model judges can spot cultural errors in long-form responses. The dataset holds 7,470 span-level annotations across 1,050 query-response pairs from four countries, in English and local languages. Even the strongest judge model scores only 0.52 F1 on erroneous span detection and routinely misses thick cultural errors that locals catch at once. Cultural evaluation, they argue, must go beyond surface detection toward depth and situated meaning.

Catalog record

CREATORJiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85526
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh Abstract As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctively, meaning that a response can be factually plausible yet unmistakably wrong to a local reader. Existing cultural benchmarks have treated culture as a flat set of facts via fact verification or norm entailment methods, and have adopted LLM-as-a-Judge without examining whether they can capture such thick cultural errors. To address this gap, we present JuICE (Benchmark for LLM- Ju dge in I dentifying C ultural E rrors), a multilingual dataset of 7,470 span-level annotations of cultural and linguistic errors in long-form LLM responses. It covers 1,050 query-response pairs from four countries (the United States, South Korea, Indonesia, and Bangladesh), in both English and their countries’ main languages. Using JuICE, we find that even the strongest LLM-judge achieves only an F1 of 0.52 in the erroneous span detection task. Furthermore, LLM-judges consistently miss thick cultural errors that local residents readily identify. Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning. 1 1 1 Our dataset, annotations, code, and prompts are available at https://jinjh0123.github.io/JuICE . 1 Introduction Figure 1: Examples of cultural errors in long-form LLM responses, organized from thinner to thicker categories in our taxonomy. While LLM-judges often detect surface-level or verifiable errors, they frequently miss thicker cultural errors such as connotation, specificity, and missingness. Globally deployed large language models (LLMs) are increasingly adopted for everyday, culturally embedded tasks, such as writing personal messages, generating ideas, and crafting stories [ 6 ] . Fundamentally, these tasks require cultural knowledge that goes beyond facts. Producing an appropriate response often depends on social norms, tacit expectations, and locally shared meanings within a community [ 29 , 11 ] —the kind of thick cultural understanding that native speakers draw on instinctively. As a result, a response can be factually plausible yet feel subtly but unmistakably wrong: a story set in Bangladesh that places the character at a coffee shop when any local reader would expect a roadside tea stall ( tong er dokan ), or one set in Indonesia that evokes the scent of jasmine ( melati ) as a pleasant city smell—when to an Indonesian reader, that scent carries the weight of funerals and ceremonies ( Figure 1 ). Evaluating such responses is inherently difficult: LLM outputs are long-form and open-ended, errors are distributed across the response, and there is no single ground-truth answer. The LLM-as-a-Judge paradigm has emerged as the dominant scalable approach to this challenge [ 39 , 19 ] , and recent cultural benchmarks use it to evaluate cultural norms or factual knowledge [ 3 , 31 ] . However, thick cultural failures may not appear as explicit norm violations or factual errors, but as subtle incoherence in setting, connotation, specificity, or missing context. Whether LLM-judges can reliably detect such thick, localized cultural errors remains an open question. To address this, we present JuICE (Benchmark for LLM- Ju dge in I dentifying C ultural E rrors), a multilingual and multicultural benchmark of 7.5k span-level annotations of cultural and linguistic errors in long-form LLM responses to everyday tasks. JuICE covers three task types—story generation, creative ideation, and how-to advice—in four languages (English, Korean, Indonesian, and Bengali) across four countries (the United States, South Korea, Indonesia, and Bangladesh). The dataset is constructed through a human–LLM collaborative annotation pipeline involving 44 native speakers, where each response is independently annotated by two native speakers and two frontier LLM-judges, with all identified spans subsequently cross-validated by human annotators. Using JuICE, we evaluate LLM-as-a-Judge reliability on two tasks: span-level error detection and sentence-level error classification. In both tasks, we measure the alignment between LLM judgments and human-validated errors. We find that LLM-judges systematically miss thick cultural errors that native speakers readily recognize, including failures of connotation, specificity, coherence, and missing cultural context. Even the strongest model achieves an F1 score of only 0.52 on span-level detection, while at most 50% of human-annotated errors are captured across all evaluated models. Furthermore, we demonstrate that providing LLM-judges with a thick-evaluation taxonomy adapted from Qadri et al. 2025 , along with category-level few-shot examples, improves detection of thick cultural errors, suggesting a practical path toward more culturally grounded evaluation paradigms. Our contributions are as follows: • We propose a novel task to evaluate the cultural competence of LLMs through their judging capabilities, specifically their ability to detect and localize cultural errors in generated text. • We present JuICE, a multilingual and multicultural dataset of 7.5k span-level annotations of cultural and linguistic errors in long-form LLM responses to everyday tasks validated by native speakers across Korean, Indonesian, Bengali, and English. • We show that current LLM-judges systematically miss thick cultural errors, achieving an F1 score of only 0.52 on span-level error detection and a recall of at most 0.50 specifically for human-detected errors. 2 Related work Table 1: Feature comparison of JuICE against existing cultural benchmarks ( ✓ : supported, ✗ : not supported). JuICE is the first benchmark to jointly support long-form evaluation, thick cultural taxonomy, large-scale span-level annotation, and LLM-as-a-judge evaluation for cultural error detection. Benchmark Long-form Thick Span-level LLM-judge Primary evaluation task evaluation taxonomy annotation evaluation BLEnD [ 21 ] ✗ ✗ ✗ ✗ Short-answer question & MCQ CultureAtlas [ 10 ] ✗ ✗ ✗ ✗ Cultural assertion verification CaLMQA [ 3 ] ✓ ✗ ✗ ✗ Atomic fact verification CultureBank [ 31 ] ✓ ✗ ✗ ✗ Norm entailment LLM-GLOBE [ 16 ] ✓ ✗ ✗ ✗ Value dimension scoring Extrinsic Eval. [ 5 ] ✓ ✗ ✗ ✗ Lexical variance analysis MAKIEval [ 38 ] ✓ ✗ ✗ ✗ Cultural entity specificity scoring CURE [ 36 ] ✓ ✓ ✗ ✗ Response-level rubric scoring TALES [ 4 ] ✓ ✓ ✓ (2.9k) ✗ Human evaluation & MCQ JuICE (Ours) ✓ ✓ ✓ (7.5k) ✓ Erroneous span detection & sentence classification 2.1 Evaluating cultural competence of LLMs As LLMs are deployed across diverse cultural contexts, evaluating their cultural understanding has become crucial [ 28 ] . Early work targeted factual cultural knowledge via closed-ended formats such as multiple-choice and short-answer questions [ 21 , 22 , 10 ] . However, real-world usage often involves open-ended, situational queries that require sensitivity to cultural nuance and context [ 40 , 23 ] . Recent work has shifted toward evaluating long-form, open-ended generation. CaLMQA [ 3 ] and CultureBank [ 31 ] use LLM-judges to assess cultural knowledge through atomic fact verification and entailment against gold-standard norms. Karinshak et al. 2024 adopted an LLMs-as-a-Jury protocol to score responses along GLOBE cultural value dimensions. Other approaches analyze outputs more directly by examining how explicit cultural cues affect open-ended QA and story generation through lexical variance [ 5 ] , or linking generated text to Wikidata

Read the complete text