CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering

Sher Badshah, Moamen Moustafa, Hassan Sajjad · 2025

Read CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering by Sher Badshah, Moamen Moustafa, Hassan Sajjad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08542.txt.

Introduction

Badshah, Moustafa, and Sajjad tackle a grading problem: free-form question answering has no single right answer to match, so automatic scoring fails, in a 2025 paper. Using large language models as judges works better but costs computing power. Their system, CLEV, asks two models to judge each answer and calls in a third only when the first two disagree.

Experiments, including checks by human raters, show the voting scheme gives consistent judgments while sparing the expense of a third opinion on easy cases. The authors present it as a scalable, resource-light way to evaluate language models on open-ended questions.

Catalog record

CREATORSher Badshah, Moamen Moustafa, Hassan Sajjad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08542

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering Sher Badshah, Moamen Moustafa, Hassan Sajjad Abstract Evaluating free-form Question-Answering (QA) remains a challenge due to its diverse and open-ended nature. Traditional automatic metrics fail to capture semantic equivalence or accommodate the variability of open-ended responses. Leveraging Large Language Models (LLMs) as evaluators offers a promising alternative due to their strong language understanding and instruction-following capabilities. We propose the Consensus via Lightweight Efficient Voting (CLEV), which employs two primary LLMs as judges and engages a third judge only in cases of disagreement. This approach prioritizes evaluation reliability while reducing unnecessary computational demands. Through experiments, including human evaluation, we demonstrate CLEV’s ability to provide consistent, scalable, and resource-efficient assessments, establishing it as a robust framework for evaluating LLMs on free-form QA. 1 Introduction One of the practical limitations in evaluating free‑form Question‑Answering (QA) is the lexical–semantic mismatch between Large Language Model (LLM) outputs and the reference answers. For the query “Who wrote 1984?” , a dataset may list simply “George Orwell” , while a helpful model replies: “It was penned by the British author Eric Arthur Blair.” Although both refer to the same person, there is no surface token overlap, so lexical-matching and even embedding-based metrics assign it an unduly low score. For instance, Exact Match (EM) requires strict lexical alignment (e.g., failing to equate “nuclear weapon” and “atomic bomb”) and ignores semantic equivalence ( Doostmohammadi et al., 2024 ) . A reference-aware LLM-as-a-judge can instead reason over meaning, recognize that the candidate entails the gold fact, and deliver a verdict, thereby overcoming this lexical–semantic gap ( Zheng et al., 2024 ; Verga et al., 2024 ) . Existing studies using LLM as judges primarily focus on subjective pairwise comparison ( Wang et al., 2023b ; Vu et al., 2024 ) and single-answer scoring ( Chiang and Lee, 2023 ; Hu et al., 2024a ; Liu et al., 2023 ) . However, objective evaluation using LLM judges, particularly for free-form QA, has received limited attention. Furthermore, LLM-based judging also lies on a cost–quality spectrum. Querying a single judge is efficient but is less reliable due to the known limitations, such as prompt sensitivity, inconsistency, and bias ( Ye et al., 2024 ) , as no individual model captures the full diversity of reasoning styles, long-tail knowledge, and user values ( Feng et al., 2025 ) . To improve robustness, some studies employ fixed ensembles of three or more LLM judges and aggregate their decisions via majority voting ( Badshah and Sajjad, 2025 ; Verga et al., 2024 ) . While this improves reliability, it increases computational cost and latency ( Badshah and Sajjad, 2024 ) , limiting scalability for large-scale evaluation ( Jung et al., 2024 ; Adlakha et al., 2024 ) . To address these trade-offs, we propose the Consensus via Lightweight Efficient Voting (CLEV) that balances the reliability and efficiency of using LLMs as judges for free-form QA. CLEV employs two primary judges for initial assessments and invokes a third judge only when disagreements occur. By minimizing redundant calls in the fixed three-judge majority vote setting, CLEV reduces computational overhead by roughly 80 to 95% (varying by task) while achieving substantial to perfect agreement. Our key contributions include: 1) establishing a principled LLM-as-a-judge evaluation setup for free-form QA that moves beyond token-level string matching metrics, 2) proposing CLEV, a lightweight consensus-based evaluation method that preserves the reliability of multi-judge voting while significantly reducing computational cost, 3) empirically validating CLEV across diverse QA datasets and multiple state-of-the-art LLMs, and 4) systematic analysis of failure cases. Figure 1: Our proposed Consensus via Lightweight Efficient Voting. 2 Methodology This section outlines the core components of our evaluation framework. Given a question and a candidate model’s answer, the evaluation task is to determine whether the answer is factually correct with respect to a reference. Because free-form QA requires an objective correctness decision rather than scalar scoring, we prompt LLM judges to return a binary verdict along with a brief rationale explaining their judgment. The verdict serves as the evaluation signal, while the rationale provides interpretability and supports later analysis. 2.1 Consensus via Lightweight Efficient Voting (CLEV) In traditional human evaluation settings, when two annotators disagree on a judgment, a third expert is often called upon to resolve the dispute. Drawing inspiration from this efficient practice, we propose CLEV. As illustrated in Figure 1 , rather than immediately employing three LLMs, CLEV adopts an efficient approach by beginning with two models as primary judges. When these judges reach a consensus, no further evaluation is needed. Only in cases of disagreement, the third LLM is engaged, whose decision then creates a majority verdict. 2.2 Judges inclusion and exclusion criteria To systematically select suitable judges for CLEV, we evaluate various LLMs (see Figure 2 ) using 100 random instances from HotpotQA. For each model, we compare binary verdicts against human annotations and compute Cohen’s Kappa ( \kappa ) and Macro F1. We interpret \kappa following the commonly used guideline where values between 0.61 – 0.80 indicate substantial agreement , and values above 0.80 indicate near-perfect agreement ( McHugh, 2012 ) . However, since \kappa is known to be sensitive to class imbalance ( Cicchetti and Feinstein, 1990 ) , we together consider Macro F1 to ensure balanced evaluation across both classes: \text{status}=\begin{cases}\text{primary judges},&\kappa\geq 0.6\ \text{and}\ \text{F1}\geq 0.85,\\[4.0pt] \text{third judge},&\kappa\geq 0.8\ \text{and}\ \text{F1}\geq 0.9,\\[4.0pt] \text{excluded},&\text{otherwise}.\end{cases} Figure 2: Judges selection based on the defined criteria in Section 2.2 . 3 Experiments We utilize the following settings to evaluate CLEV: Models Our candidate models includes Llama-3.1 70B ( Meta AI, 2024 ) , GPT-3.5-turbo ( Brown et al., 2020 ) , Mistral 7B ( Jiang et al., 2023 ) , and Mixtral 8x7B ( Jiang et al., 2024 ) . For judges, Mistral‑7B met our inclusion criteria ( \kappa\geq 0.6 , \mathrm{F1}\geq 0.85 ) while offering low cost and fast inference. Llama‑3.1‑70B, though more expensive, provided substantial agreement. We therefore use both as primary judges to capture a range of capabilities ( Feng et al., 2025 ; Liang et al., 2024 ; Sun et al., 2024 ) . GPT‑3.5‑turbo serves as the third judge because it exceeds the required thresholds ( \kappa\geq 0.8 , \mathrm{F1}\geq 0.9 ). All models are run with temperature 0 to ensure reproducibility, as higher temperatures degrade evaluator reliability ( Hada et al., 2024 ) . Datasets We utilize four datasets: AmbigQA ( Min et al., 2020 ) , HotpotQA ( Yang et al., 2018 ) , Natural Questions ( Kwiatkowski et al., 2019 ) , TriviaQA ( Joshi et al., 2017 ) . We only utilize FreshQA to evaluate the judge’s ability to detect outdated information. See Appendix A for details. Prompts We design minimal zero-shot role-playing prompts ( Kong et al., 2024 ) for both candidate and judge LLMs. The judge LLM is instructed to return a binary verdict with a brief explanation, which reduces subjectivity and simplifies automatic evaluation (see Appendix F ). Baselines We compare individual LLM judges and CLEV against EM, BERTScore ( Zhang et al., 2020 ) computed with microsoft/deberta-xlarge-mnli and thresholded at \tau=0.5 to yield binary decisions and majority voting, which always invokes a fixed setup of LLM

Read the complete text