MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks

Jaime Raldua Veuthey, Zainab Ali Majid, Suhas Hariharan, Jacob Haimes · 2025

Read MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks by Jaime Raldua Veuthey, Zainab Ali Majid, Suhas Hariharan, Jacob Haimes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14039.txt.

Introduction

Jaime Veuthey and three co-authors build a way to grade the graders: the test sets that grade language models. Tests grow fast, yet no set way grades the worth of a test itself. Their 2025 frame, MEQA, gives standard marks, scores in counts, and fair side-by-side readings of question-and-answer test sets. Human and model judges both run in it.

Cyber test sets serve as the field case, picked since models there cut both ways, as shield and as threat. The runs show the strengths and soft spots of each set in plain grades. Set scores let builders pick tests by proof, not by fame. A shared yardstick for tests is the lack the frame fills, they hold.

Catalog record

CREATORJaime Raldua Veuthey, Zainab Ali Majid, Suhas Hariharan, Jacob Haimes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14039

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks Jaime Raldua Veuthey, Zainab Ali Majid, Suhas Hariharan, Jacob Haimes Abstract As Large Language Models (LLMs) advance, their potential for widespread societal impact grows simultaneously. Hence, rigorous LLM evaluations are both a technical necessity and social imperative. While numerous evaluation benchmarks have been developed, there remains a critical gap in meta-evaluation: effectively assessing benchmarks’ quality. We propose MEQA, a framework for the meta-evaluation of question and answer (QA) benchmarks, to provide standardized assessments, quantifiable scores, and enable meaningful intra-benchmark comparisons. We demonstrate this approach on cybersecurity benchmarks, using human and LLM evaluators, highlighting the benchmarks’ strengths and weaknesses. We motivate our choice of test domain by AI models’ dual nature as powerful defensive tools and security threats. 1 Introduction Robust LLM evaluations are challenging ( Biderman et al., 2024 ) and past meta-evaluation work has focused on a limited number of specific metrics ( Biderman et al., 2024 ; Ren et al., 2024 ) . We propose MEQA, a holistic approach that synthesizes knowledge from previous work ( Biderman et al., 2024 ; Ren et al., 2024 ; Sclar et al., 2024 ; Stureborg et al., 2024 ; Subramonian et al., 2023 ) into eight key criteria, described in Table 1 . In order to ensure scoring is intuitive and specific, we break down these criteria into 44 sub-criteria, detailed in Appendix A . Evaluators then give scores between 1 and 5 for each sub-criterion. Table 1: Meta-evaluation criteria Criteria Description Memorization Robustness Tests for abilities of models beyond memorizing information ( Yu et al., 2023 ; Zhu et al., 2024 ) . Publicly available questions, for example, would be insufficient. Prompt Robustness Tests for models’ consistency in responding to various prompt formats ( Sclar et al., 2024 ) ; multi-prompt evaluations ( Mizrahi et al., 2024 ) , for example, are effective. Evaluation Design Separates evaluation criteria (e.g., technical accuracy, clarity, relevance) and uses appropriate scoring granularity for nuanced assessment ( Stureborg et al., 2024 ; Zheng et al., 2023 ; Saphra et al., 2024 ) . Evaluator Design Provides clear evaluation guidelines including considerations on multi-attribute judgment design, annotator disagreement, and ensuring evaluator expertise ( Fluri et al., 2023 ; Stureborg et al., 2024 ) . Reproducibility Ensure features such as full transparency and availability of evaluation code. Comparability Facilitate consistent and fair comparisons through factors such as standardized task implementations and prompt formats ( Bach et al., 2022 ) . Validity Features such as ensuring representative data and real-world applicability ( Subramonian et al., 2023 ; Jacobs & Wallach, 2019 ) . Reliability Demonstrates statistical rigor through output sharing, hyperparameter reporting, multiple runs, confidence intervals, standard errors, and reliability metrics ( Xiao et al., 2023 ) . 2 Cybersecurity benchmarks 2.1 Methodology We test MEQA using both human and LLM evaluators to score cybersecurity benchmarks. We had three independent human evaluators; their scores agreed exactly on the majority (over 80%) of the sub-criteria, and disagreements were no more than 1 point apart. Additionally, we used GPT-4o to assess benchmarks by prompting with sub-criteria definitions (code is available anonymously and compute details are present in Appendix F . The automated scoring tended to match human evaluators, particularly when producing extreme scores (1 or 5). Please see Appendix D for full limitations. This LLM-aided approach undoubtedly saved time and is crucial for scalable, automated meta-evaluations, needed to keep pace with new benchmarks across domains. 2.2 Results Figure 1: Scores of cybersecurity benchmarks per criterion. N/A indicates inapplicable criteria (e.g. SECURE uses pre-defined correct answers; evaluator design does not apply). Our scores in Figure 1 highlight cybersecurity benchmarks’ strengths. Most benchmarks score highly in terms of reproducibility and comparability. SecEval stands out regarding evaluation and evaluator design. HarmBench-Cyber and WMDP-Cyber scored well across several categories and highest scores (Table 2 ), indicating relatively high quality. However, key weaknesses were also brought to light. Most benchmarks obtain low scores for prompt robustness and reliability. The high standard deviations shown in Table 2 reveal significant variability, suggesting that each benchmark’s performance fluctuates considerably across different sub-criteria. Scores can be contextualised further using sub-criteria detailed in Appendix A and results in Appendices B and E . 3 Conclusions Table 2: Benchmark scores Benchmark Mean (Std) HarmBench-Cyber ( Mazeika et al., 2024 ) 3.6 (1.3) WMDP-Cyber ( Li et al., 2024 ) 3.5 (1.6) CyberSecEval 2 ( Bhatt et al., 2024 ) 3.4 (1.5) CyberSecEval 1 ( Bhatt et al., 2023 ) 3.2 (1.4) SecEval ( Li et al., 2023 ) 3.2 (1.4) SEvenLLM-Bench ( Ji et al., 2024 ) 2.9 (1.0) CyberMetric ( Tihanyi et al., 2024 ) 2.8 (1.2) SECURE ( Bhusal et al., 2024 ) 2.7 (1.1) SecQA ( Liu, 2023 ) 2.7 (1.2) We propose a novel meta-evaluation framework, MEQA, and test it on cybersecurity QA benchmarks. While many bechmarks perform well in terms of reproducibility and comparability, there is a clear need for improvement in prompt robustness and reliability. Additionally, our tests using LLM evaluators show promise in improving scalability. We aim to further refine our evaluation criteria and test additional benchmarks. We hope this framework can serve as a blueprint for benchmark developers and enable effective gap-analysis. Impact Statement Developing meta-evaluation frameworks for LLM benchmarks requires thoughtful consideration and ongoing refinement. It is crucial to carefully select and regularly update the criteria used. The rigor of these meta-evaluations is of utmost importance, as they effectively establish the standards for both LLM benchmarks and, by extension, the LLMs themselves. By maintaining high-quality meta-evaluation frameworks, we can drive the development of more robust, relevant, and reliable benchmarks, which in turn will lead to the creation of more capable and trustworthy language models. This process of continuous improvement and careful curation of evaluation criteria is essential for advancing the field of AI in a responsible and effective manner. Appendices Appendix A Criteria and sub-criteria We outline our meta-evaluation criteria in Table 1 . The below tables add additional detail to this framework, focusing on the 44 sub-criteria and our scoring rubric for each one. A.1 Memorization robustness The benchmark should be designed so that LLM capabilities being measured go beyond the capacity of LLMs to memorise information. This error mode can happen if the questions in the benchmark are designed using data on which the LLM was originally trained on. Sub-criteria are detailed in Table 3 . Table 3: Memorization Robustness sub-criteria Sub-criteria Detail Training cut-off date Purpose: The benchmark should include questions that are unlikely to be in the model’s training data. Low score: Uses questions from popular certification tests (likely in LLM training data). High score: Uses new information sources published after the LLM training cut-off date; questions are curated by domain experts based on this new information. Public/private data Purpose: Check if questions are taken from open public data or from private sources. Low score: Questions are from very easily accessible public data (e.g. Wikipedia). Medium score: Questions are from less accessible public sources (e.g. behind paywall). High score: Questions are from private, proprietary data or non-digitalized data. Question Formulation Purpose: Look at how questions are formulated: copied directly from the original source, paraphrased, or newly curated by

Read the complete text