RecKon : Large-scale Re ference-based Effi c ient K nowledge Evaluati on for Large Language Model
Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua Xiao · 2025
Read RecKon : Large-scale Re ference-based Effi c ient K nowledge Evaluati on for Large Language Model by Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua Xiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00756.txt.
Introduction
Zhang, Gu, Shi, Feng, and Xiao wrote in twenty twenty-five about testing what large language models know, quickly and at scale. Standard benchmarks cost much to run and can lose information. Their method, called RECKON, judges models directly against reference data instead. It splits loose data into manageable units. For each cluster of units it builds targeted questions to probe knowledge.
Tests showed the method cut resource use by more than half. Accuracy stayed above ninety-seven percent across varied domains. Those domains included world knowledge, code, law, and biomedicine. The authors release their code for public use. Faster checks matter as models change often. A reference-based method can keep evaluation up with that pace.
Catalog record
| CREATOR | Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua Xiao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RecKon : Large-scale Re ference-based Effi c ient K nowledge Evaluati on for Large Language Model Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua Xiao Abstract As large language models (LLMs) advance, efficient knowledge evaluation becomes crucial to verifying their capabilities. Traditional methods, relying on benchmarks, face limitations such as high resource costs and information loss. We propose the Large-scale Re ference-based Effi c ient K nowledge Evaluati on for Large Language Model ( RecKon ), which directly uses reference data to evaluate models. RecKon organizes unstructured data into manageable units and generates targeted questions for each cluster, improving evaluation accuracy and efficiency. Experimental results show that RecKon reduces resource consumption by 56.5% compared to traditional methods while achieving over 97% accuracy across various domains, including world knowledge, code, legal, and biomedical datasets. Code is available at https://github.com/MikeGu721/reckon . 1 Introduction As large language models (LLMs) continue to advance rapidly, knowledge evaluation has become an essential component for verifying their capabilities and driving continuous improvements. The fundamental objective of evaluation is to assess and ensure the LLM’s alignment with reference data. Thorough and accurate knowledge evaluation plays a critical role in ensuring reliable model performance across different fields, especially in crucial areas such as healthcare, education, and law, where strong evaluation methods are necessary to reduce risks from errors or outdated information. With the rapid growth of LLM applications, the requirements for knowledge evaluation have become increasingly extensive. Modern evaluation approaches must meet growing demands, including wider coverage of different fields, more complex analytical tasks, and the handling of much larger datasets. These requirements pose significant challenges to traditional evaluation methods. Figure 1: The comparison of different evaluation method. Traditional knowledge evaluation methodologies primarily rely on benchmarks, encompassing various assessment formats such as multiple-choice questions Gu et al. 2024 , fill-in-the-blanks exercises Hendrycks et al. 2020 , and marking tasks Dimitrov et al. 2024 . These benchmarks fundamentally serve as condensed or refined representations of reference data, carefully designed to facilitate manageable and systematic evaluation processes. Despite widespread adoption across the field, these benchmark-based approaches exhibit several inherent limitations. First, the creation and verification of comprehensive benchmarks demand substantial resources, necessitating significant manual and computational expenses. Second, the process of distilling complex, multifaceted information into standardized questions inevitably results in information loss, potentially introducing systematic biases and compromising the validity of knowledge assessment. Finally, individual benchmarks face considerable challenges in fully capturing the intricate complexity and expansive diversity of knowledge domains, thereby constraining their adaptability and broader applicability. As illustrated in Fig. 1 , directly leveraging rich and dynamic reference data is an alternative approach, which presents distinct advantages over traditional benchmark-based methods. This direct reference approach not only preserves the nuanced complexity of the original information but also enables more comprehensive and authentic evaluation scenarios. Furthermore, it facilitates dynamic updating of evaluation criteria as new knowledge emerges, ensuring continued relevance and accuracy. The approach also demonstrates superior scalability across different knowledge domains and evaluation contexts, offering enhanced flexibility in evaluation. However, this direct reference-based evaluation introduces its own set of significant challenges: First, reference datasets typically exist in large-scale, unstructured formats, presenting substantial difficulties in effective organization and utilization during evaluation processes. Second, the absence of predefined questions or standardized answer formats complicates the establishment of consistent and equitable alignment between model outputs and reference data. To address these challenges, we propose a Re ference-based Effi c ient K nowledge Evaluati on framework, called RecKon , which directly and efficiently evaluates LLMs using reference data. In response to the difficulty of organizing and utilizing large and unstructured reference datasets, RecKon decomposes the reference data into discrete, manageable knowledge units. These units are then grouped into thematic clusters, forming an organized knowledge structure that streamlines the evaluation process. To solve the problem of ensuring consistent and fair alignment without predefined questions or fixed answers, RecKon generates targeted questions for each knowledge cluster. The evaluation focuses on assessing whether the model’s responses adequately cover all relevant knowledge units, ensuring comprehensiveness and accuracy. Experimental results highlight the superiority of RecKon as a comprehensive, efficient, and adaptable framework for knowledge evaluation. Across various common-sense and anti-common-sense tasks, RecKon effectively adapts references to reduce the inherent biases of LLMs. Notably, RecKon reduced resource consumption by 56.5% compared to “full reference input evaluation” without compromising evaluation accuracy. Additionally, RecKon demonstrated exceptional adaptability across diverse domains, including world knowledge, code, legal, and biomedical datasets, achieving an accuracy exceeding 97%. 2 Related work Open-ended question answering benchmarks for models include MMLU Hendrycks et al. 2020 , C-eval Huang et al. 2024 , Xiezhi Gu et al. 2024 , etc. These benchmarks contain a series of questions that are described in natural language and require the model to give an open-ended answer. However, these benchmarks rely on a large number of manual annotations and cannot be updated with the latest knowledge. Currently, some methods propose using LLMs to automatically build updatable benchmarks, such as LM-as-an-Examiner Bai et al. 2024 and TreeEval Li et al. 2024 . In these methods, when no benchmark is available, LLMs generate questions based on their own knowledge. However, the knowledge in LLM cannot be inherently complete, and the language model is biased, which will lead to incomplete questions and deviation of questions to a certain extent. Traditional open evaluation metrics are based on n-grams to measure semantic similarity between texts, including BLEU Papineni et al. 2002 , ROUGE Lin 2004 , METEOR Banerjee and Lavie 2005 , etc. These methods enable the automated evaluation of natural language texts. However, these methods lack expressive power and cannot semantically distinguish key information, such as negative words. To address these problems, BERTScore Zhang et al. 2019 , GPTScore Fu et al. 2023 , and other methods use language models to derive similarities between candidate answers and reference answers. In the latest evaluation work on LLM, LLM-as-Judge (a paradigm that uses LLM to evaluate open results) Zheng et al. 2024 ; Chan et al. 2023 is proposed. This method is recognized for its strong interpretability and scalability, making it widely adopted. These methods still rely on either artificially constructed reference answers or the LLM’s own knowledge, leaving the issue of knowledge limitations unresolved. RecKon distinguishes itself from traditional benchmarks by using a dynamic, reference-based approach rather than static question-answer pairs. This design makes it more adaptable, scalable, and effective at evaluating nuanced knowledge across diverse domains. Unlike traditional benchmarks, which are limited in coverage, flexibility, and efficiency, RecKon leverages rich reference data to align with evolving knowledge. Moreover, RecKon distinguishes itself from other LLM-related evaluation methods, such as LLM-as-Judge, by grounding evaluations in external references rather than relying on the model’s subjective judgment. This ensures a more objective, transparent, and consistent evaluation process, effectively mitigating biases and variability inherent in LLM-as-Judge approaches. 3 RecKon Framework Figure 2: RecKon consists of five main components that form a cycle of assessment. As shown in Fig. 2 , RecKon consists of five main components, and the description of these components is as follows: Step 1 Preparing Knowledge Unit Candidates: The RecKon framework begins by preparing knowledge units that will be utilized throughout the evaluation process. These units are sourced