HRET: A Self-Evolving LLM Evaluation Toolkit for Korean

Hanwool Lee, Sooyong Kim, Dasol Choi, Sangwon Baek, Seunghyeok Hong, Ilgyun Jeong, Naeun Lee, Inseon Hwang, Guijin Son · 2025

Read HRET: A Self-Evolving LLM Evaluation Toolkit for Korean by Hanwool Lee, Sooyong Kim, Dasol Choi, Sangwon Baek, Seunghyeok Hong, Ilgyun Jeong, Naeun Lee, Inseon Hwang, Guijin Son (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22968.txt.

Introduction

Hanwool Lee and co-authors introduced HRET in 2025, an open-source toolkit that unifies how Korean language models are evaluated. Korean models have multiplied, and so have benchmarks, yet clashing protocols leave score gaps of up to ten points between institutions for the same model. The answer, they argue, is not one forced test, but a framework sturdy enough to hold many methods.

HRET gathers major Korean benchmarks, several inference backends, and multiple scoring methods in a registry design. New data sets, methods, and backends can be added fast, and a consistency check enforces genuine Korean output. The head title frames the work as redefining evaluation standards, and its byline order differs from the catalog list. Readers get infrastructure for fair Korean model comparison.

Catalog record

CREATORHanwool Lee, Sooyong Kim, Dasol Choi, Sangwon Baek, Seunghyeok Hong, Ilgyun Jeong, Naeun Lee, Inseon Hwang, Guijin Son
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRET: A Self-Evolving LLM Evaluation Toolkit for Korean Hanwool Lee, Sooyong Kim, Dasol Choi, Sangwon Baek, Seunghyeok Hong, Ilgyun Jeong, Naeun Lee, Inseon Hwang, Guijin Son Abstract Recent advancements in Korean large language models (LLMs) have spurred numerous benchmarks and evaluation methodologies, yet the lack of a standardized evaluation framework has led to inconsistent results and limited comparability. To address this, we introduce HRET ( Haerae Evaluation Toolkit ), an open-source, self-evolving evaluation framework tailored specifically for Korean LLMs. HRET unifies diverse evaluation methods, including logit-based scoring, exact-match, language-inconsistency penalization, and LLM-as-a-Judge assessments. Its modular, registry-based architecture integrates major benchmarks (HAE-RAE Bench, KMMLU, KUDGE, HRM8K) and multiple inference backends (LiteLLM, HuggingFace, OpenAI-compatible endpoints). With automated pipelines for continuous evolution, HRET provides a robust foundation for reproducible, fair, and transparent Korean NLP research. Figure 1: Architecture overview of HRET, showing the registry-based modular design that integrates datasets, models, inference strategies, and evaluation methods through a unified pipeline. 1 Introduction Mature evaluation practices are essential for accurately tracking advancements in the capabilities of large language models (LLMs). Biased or skewed assessments can mislead the research community and hinder shared goals of technological progress. In this line, the English NLP community has long pursued credible, standardized benchmarks—starting with the GLUE and SuperGLUE leaderboards ( Wang et al. 2018 ; Wang et al. 2019 ) , followed by LM-Eval Harness ( Gao et al. 2024 ) , HELM ( Liang et al. 2023 ) , and Chatbot Arena—all aimed at providing fair, comparable evaluations under consistent conditions. Now, as Korean NLP research increases in volume, they face similar challenges: Technical reports for foundational models often yield inconsistent metrics, complicating direct comparisons. This underscores the need for a unified evaluation framework tailored to Koreans. Korean benchmarks vary considerably in their evaluation setups—HAE-RAE Bench utilizes a log-probability-based evaluation, while KMMLU employs exact-match criteria. Such methodological discrepancies often compel researchers and practitioners to customize evaluations, complicating direct comparisons and compromising interpretability. For example, when evaluating identical models like Gemma2-27B on the same benchmarks, different research groups report significantly different results (as shown in Table 1 ), highlighting the urgent need for standardization. Additionally, existing evaluations frequently overlook key linguistic nuances unique to Korean, such as intricate morphology, context-dependent word segmentation, and extensive honorific systems. These language-specific challenges require specialized consideration in evaluation methodologies to ensure accurate performance assessment of Korean LLMs. To address these challenges, we introduce HRET (Haerae Evaluation Toolkit), a standardized, open-source toolkit specifically tailored for evaluating Korean LLMs. HRET makes several key contributions: • Unified and reproducible evaluation : HRET standardizes dataset handling, evaluation methods, and benchmarking procedures, ensuring transparency and consistent outcomes. • Self-Evolving Mechanism : Its modular registry-based design allows HRET to continuously integrate new benchmarks, evaluation methodologies, and inference backends with minimal effort. • Multi-method evaluation : Supports multiple complementary evaluation methods—including logit-based scoring, exact-match evaluation, and LLM-as-a-judge—to comprehensively assess model performance. • Ease of Use : Intuitive API and streamlined command-line interfaces simplify evaluation workflows, making it accessible for both researchers and industry practitioners. By unifying prominent benchmarks and diverse inference backends—including HuggingFace, vLLM, and OpenAI-compatible APIs—HRET significantly improves reproducibility, comparability, and accessibility in Korean NLP research. The toolkit and comprehensive documentation will be publicly released under the Apache License 2.0 upon acceptance to encourage widespread adoption and community contributions. Report Model Datasets GSM8K MMLU KMMLU HumanEval MBPP EXAONE 3.5 Report Research et al. 2024 Qwen2.5-32B 92.00 81.40 62.10 89.00 88.90 Gemma2-27B 84.20 74.80 53.80 79.30 80.70 Kanana Report Bak et al. 2025 Qwen2.5-32B 82.41 83.10 63.15 75.16 73.40 Gemma2-27B 74.37 75.45 51.16 51.22 64.60 Table 1: Performance discrepancies reported for the Gemma-2-27B model between EXAONE 3.5 and Kanana technical reports, highlighting reproducibility challenges in Korean NLP evaluations. 2 Related Work and Background 2.1 LLM Benchmarks and Leaderboards The English LLM ecosystem has established robust evaluation frameworks like Evalverse Kim et al. 2024b and deepeval AI 2025 . The lm-evaluation-harness Gao et al. 2024 by Upstage has been particularly influential, powering their Open LLM Leaderboard which has accelerated global LLM development. Its success highlights how easy-to-use evaluation tools can democratize model development - lowering barriers for smaller research labs and startups to participate in advancing the field. Similarly, Chinese frameworks like FlagEval Team 2025 have adopted this accessibility-first approach to systematically advance their language-specific ecosystems. 2.2 Reproducibility Challenges Current Korean NLP evaluations exhibit considerable reproducibility challenges. A notable example is the inconsistent results reported for widely used models such as Gemma2-27B Team et al. 2024 and Qwen2.5-32B Yang et al. 2024 across various technical reports from major Korean organizations. As illustrated in Table 1 , evaluations of identical benchmarks using these same models yield significantly different outcomes depending on the evaluating institution. For instance, the EXAONE 3.5 report Research et al. 2024 and Kanana report Bak et al. 2025 report performance discrepancies ranging from around 1–10 percentage points across multiple tasks, including GSM8K, KMMLU, HumanEval, and MBPP. These discrepancies stem from variations in inference settings, prompt engineering, and evaluation implementations. The situation is further complicated by Korean’s linguistic features—complex morphology, context-dependent segmentation, and honorific systems—which are handled inconsistently across evaluations. Existing benchmarks lack standardized guidelines addressing these Korean-specific challenges, resulting in fragmented evaluation practices that impede meaningful progress in Korean NLP research. Feature lm-eval-harness Biderman et al. 2024 OpenAI Evals Liu et al. 2024 Evalverse Kim et al. 2024b DeepEval AI 2025 HRET (Ours) Primary Focus Academic benchmarks OpenAI model evaluation Unified evaluation hub Unit/regression testing Korean LLM evaluation Developer EleutherAI OpenAI UpstageAI Confident AI HAERAE Architecture Task/Metric modules Evaluation scripts Submodule integration Test case-based Registry-based plugins Built-in Benchmarks 60+ general benchmarks Several public evals Integrated external benchmarks Major benchmarks Korean-focused benchmarks Evaluation Metrics Traditional metrics Task-specific scripts Integrated metrics 14+ diverse metrics Traditional + LLM-as-judge User Interface CLI only CLI + Dashboard Slack bot + Web CLI + Web dashboard CLI + API Automation Script-based API-based No-code automation CI/CD integration Pipeline Runner Unique Strength High reproducibility OpenAI optimization Multi-tool integration Testing-focused Registry-based extensibility Table 2: Feature comparison of HRET with major existing LLM evaluation frameworks, highlighting its registry-based extensibility and Korean-specific benchmarks. 2.3 Comparison with Existing Systems To position HRET within the landscape of LLM evaluation frameworks, Table 2 compares its features with other systems. This comparison highlights HRET’s unique contributions to the field, particularly its focus on Korean language evaluation and its registry-based modular architecture. HRET differentiates itself through its specific focus on Korean language evaluation challenges, its registry-based architecture enabling easy extension, and its integrated pipeline approach that combines dataset handling, model inference, scaling, and evaluation into a unified framework. 3 HRET Framework HRET ( Haerae Evaluation Toolkit ) addresses critical reproducibility and comparability challenges faced in evaluating Korean large language models (LLMs). By providing a standardized yet flexible evaluation environment, HRET fosters consistency, transparency, and accessibility in Korean NLP research. Figure 1 illustrates its modular and registry-based architecture, seamlessly integrating datasets, evaluation methods, inference strategies, and model backends into a coherent evaluation pipeline. HRET is explicitly designed for: (1) researchers who require standardized evaluation methods for benchmarking Korean language models; (2) developers looking for robust tools to streamline model assessment and selection; and (3) industry practitioners aiming to reliably deploy Korean LLMs in production. 3.1 Novelty

Read the complete text