KAIO : A Collection of More Challenging Korean Questions

Nahyun Lee, Guijin Son, Hyunwoo Ko, Kyubeen Han, [0.5em] Chung-Ang University, OneLineAI, Konkuk University, MODULABS · 2025

Read KAIO : A Collection of More Challenging Korean Questions by Nahyun Lee, Guijin Son, Hyunwoo Ko, Kyubeen Han, [0.5em] Chung-Ang University, OneLineAI, Konkuk University, MODULABS (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14752.txt.

Introduction

Nahyun Lee and three co-authors introduce KAIO in 2025, a harder Korean benchmark for frontier language models. Korean benchmarks are fewer, often translated or narrow, and saturate quickly as models improve. No current Korean test could rank frontier systems, the authors argue.

KAIO is a Korean, maths centred collection stressing long chain reasoning, built to resist saturation and contamination longer than legacy suites. Original Korean questions replace translated imports, cutting both cultural mismatch and leakage from English training data. Frontier progress in Korean needs a yardstick that does not top out within a product cycle. A durable benchmark also protects buyers in Korea from English only capability claims.

Catalog record

CREATORNahyun Lee, Guijin Son, Hyunwoo Ko, Kyubeen Han, [0.5em] Chung-Ang University, OneLineAI, Konkuk University, MODULABS
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14752

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KAIO : A Collection of More Challenging Korean Questions Nahyun Lee, Guijin Son, Hyunwoo Ko, Kyubeen Han, [0.5em] Chung-Ang University, OneLineAI, Konkuk University, MODULABS Abstract With the advancement of mid/post-training techniques, LLMs are pushing their boundaries at an accelerated pace. Legacy benchmarks saturate quickly (e.g., broad suites like MMLU over years, newer ones like GPQA-D even faster), which makes frontier progress hard to track. The problem is especially acute in Korean : widely used benchmarks are fewer, often translated or narrow in scope, and updated more slowly, so saturation and contamination arrive sooner. Accordingly, at this moment there is no Korean benchmark capable of evaluating and ranking frontier models. To bridge this gap, we introduce KAIO , a Korean, math-centric benchmark that stresses long-chain reasoning. Unlike recent Korean suites that are at or near saturation KAIO remains far from saturated: the best-performing model, GPT-5, attains 62.8 followed by Gemini-2.5-Pro (52.3). Open models such as Qwen3-235B and DeepSeek-R1 cluster falls below 30 demonstrating substantial headroom, enabling robust tracking of frontier progress in Korean. To reduce contamination, KAIO will remain private and be served via a held-out evaluator until the best publicly known model reaches at least 80 % accuracy, after which we will release the set and iterate to a harder version. 1 Introduction Traditionally, major LLM revisions were slow and costly: advancing a model typically required collecting and cleaning massive corpora ( Bai et al. 2023 ; Gao et al. 2020 ) followed by billion-dollar–scale pre-training ( OpenAI 2025a ) . Recent mid- and post-training methods (e.g., continued pre-training, SFT, preference optimization, RL/distillation) ( Guo et al. 2025 ) now enable frequent, lower-cost updates. With faster updates, however, benchmarks saturate more often. For example, MMLU (released 2020/09) ( Hendrycks et al. 2020 ) took \approx 31 months to see the first >80\% result (GPT-4, 2023/03), whereas GPQA-Diamond (released 2023/11) ( Rein et al. 2024 ) crossed 80\% in \approx 15 months (2025/02) despite being substantially harder. The problem is sharper in Korean: benchmark releases are scarce, and modern multilingual models now perform in Korean nearly as well as in English, rapidly saturating existing suites. As Figure 1 illustrates, the math benchmark KSM (Jan 2025) ( Ko et al. 2025 ) and the general-knowledge KMMLU-Pro (May 2025) ( Hong et al. 2025 ) already have models exceeding 80%, for KMMLU-Pro , within roughly three months of release. This pace highlights the need for harder, contamination-resistant Korean benchmarks. Figure 1: Accuracy of DeepSeek-R1 (light purple) and Gemini-2.5-Pro (dark) on KSM, KMMLU-Pro, and KAIO. To this end, we introduce KAIO , a Korean mathematics benchmark designed to stress top-tier LLMs. We manually curated 331 candidate problems from publicly available, high-difficulty sources (e.g., contest and olympiad preparation sites). We then applied multi-stage adversarial filtering, de-duplication, ambiguity and leakage checks, and model-in-the-loop screening—retaining only items that consistently defeat strong frontier models. The final set contains 44 problems selected for maximal difficulty. Our evaluation shows that, the best-performing model, GPT-5, attains 62.8, followed by Gemini-2.5-Pro (52.3) and open models clustering below 30. On efficiency, GPT-5 sits on the accuracy–cost frontier at 7.8k tokens per query. Finally, we observe that output length generally does not predict correctness (Welch’s t -test, \alpha{=}0.05 ). To prevent contamination, KAIO will remain private and be served via a held-out evaluator until the best publicly known model reaches at least 80 % accuracy, after which we will release the set and iterate to a harder version. 2 Preliminary: Saturation of Korean Benchmarks Benchmark Release date \geq 80% reached Ko-HellaSwag Nov 2023 ✓ Ko-TruthfulQA Feb 2024 ✓ KMMLU Feb 2024 ✓ KSM Jan 2025 ✓ KMMLU-Pro May 2025 ✓ Table 1: Saturation status of major Korean-language benchmarks. Each row lists the benchmark, its first public release date, and whether any language model has achieved 80% accuracy In recent years, LLMs have rapidly ascended toward high-level performance, demonstrating impressive proficiency not only in English but also in Korean. When benchmark performance transcends the 80% threshold, its interpretive value often diminishes due to inherent limitations in dataset curation and evaluation methodology Kiela et al. 2021 ; Gema et al. 2024 . In this context, distinctions such as 85% vs. 90% are likely influenced more by dataset artifacts or annotation noise than by genuine differences in model capability. At these elevated performance levels, the remaining errors frequently stem from ambiguous problem statements, annotation inconsistencies, or subjective interpretation differences rather than genuine model capabilities ( Northcutt et al. 2021 ; Bowman and Dahl 2021 ; Kovatchev and Lease 2024 ) . Earlier Korean benchmarks ( Park et al. 2024 ) such as Ko-HellaSwag (Nov. 2023) and the factuality set Ko-TruthfulQA (Feb. 2024)—saturated rapidly, with leading systems exceeding 80% accuracy within \sim 25 weeks of release. More recent datasets have converged even faster: the mathematics benchmark KSM (Jan. 2025) ( Ko et al. 2025 ) reached saturation in about six months, while the general-knowledge suite KMMLU-Pro (May 2025) ( Hong et al. 2025 ) crossed the 80% threshold in roughly three months. To address these fundamental limitations, a reorientation of benchmark design philosophy is essential. Instead of pursuing comprehensive coverage through large-scale datasets, rigorous evaluation of frontier models necessitates meticulously curated problems that maintain discriminative power even at high performance levels. This methodology emphasizes depth over breadth , concentrating on issues that persistently challenge state-of-the-art systems. 3 KAIO 3.1 Dataset Collection We systematically collected 331 candidate problems from mathematics contests and olympiad preparation websites. Problems were digitized via a YOLO-based OCR pipeline and subsequently validated by five human reviewers using a structured annotation interface (Figure 6 ), ensuring accurate mathematical notation and eliminating LaTeX artifacts. We then applied multi-stage difficulty screening: each candidate was evaluated with fixed snapshots of earlier state-of-the-art models (e.g., GPT-4.1, Gemini-2.0-Pro ( Comanici et al. 2025 ) , Qwen2.5-72B ( Yang et al. 2024 ) , Llama-3.1-70B ( Grattafiori et al. 2024 ) ). Using prior SOTA systems, rather than the contemporary models under evaluation, reduces the risk of adversarial tailoring and avoids unintended penalties to specific model families. We retained only problems that remained consistently challenging across diverse architectures. This process yields a final set of 44 problems combining high mathematical complexity with precisely rendered content. The size is comparable to recent math benchmarks, such as AIME 2024/2025, which offer two 15-question rounds (A and B) per year for a total of 30 questions. 3.2 Dataset Description Linear Algebra For a positive integer n , let A be an n\times n real matrix with n distinct positive eigenvalues. Find the number of real matrices B satisfying B^{2016}=A . Answer: 2^{n} Geometry & Analysis Compute the area of the quadrilateral with vertices (0,0,0) , (1,2,3) , (3,1,2) , (7,4,7) . Answer: \frac{15\sqrt{3}}{2} Table 2: Example question–answer blocks. Each block lists a category, a representative question, and its answer. Figure 2: KAIO benchmark results showing model accuracy with standard deviation across three runs. GPT-5 leads at 62.9%, with clear performance tiers emerging across model families. Error bars represent standard deviation from three independent evaluations. Table 2 shows examples of questions included in KAIO. The KAIO dataset covers a wide range of problem types, including linear algebra, analysis, probability and statistics, optimization, and logical reasoning. In

Read the complete text