EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta

Raymond Bernard, Shaina Raza PhD, Subhabrata Das, PhD, Rahul Murugan · 2025

Read EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta by Raymond Bernard, Shaina Raza PhD, Subhabrata Das, PhD, Rahul Murugan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00257.txt.

Introduction

Raymond Bernard, Shaina Raza, Subhabrata Das, and Rahul Murugan build a rule-based grading frame for open questions put to text models. Pick-one tests grade polish, not truth or depth of thought, and people must still mark most open work by hand. Their frame, EQUATOR, shown as 2024 in the text, pairs open questions with hand-marked answers in a vector store. Scores run by fixed rules keyed to truth and quality of reasoning.

The frame cuts the need for hand marking hard and scales where hand sets cannot. It beat pick-one grades in their runs while it held tight truth bars. A small local model runs the grading loop on site. Truth on open questions, they hold, can be graded by rule, not just by eye.

Catalog record

CREATORRaymond Bernard, Shaina Raza PhD, Subhabrata Das, PhD, Rahul Murugan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta Raymond Bernard, Shaina Raza PhD, Subhabrata Das, PhD, Rahul Murugan Abstract Despite the remarkable coherence of Large Language Models (LLMs), existing evaluation methods often suffer from fluency bias and rely heavily on multiple-choice formats, making it difficult to assess factual accuracy and complex reasoning effectively. LLMs thus frequently generate factually inaccurate responses, especially in complex reasoning tasks, highlighting two prominent challenges: (1) the inadequacy of existing methods to evaluate reasoning and factual accuracy effectively, and (2) the reliance on human evaluators for nuanced judgment, as illustrated by Williams and Huckle (2024) [ 1 ] , who found manual grading indispensable despite automated grading advancements. To address evaluation gaps in open-ended reasoning tasks, we introduce the EQUATOR Evaluator (Evaluation of Question Answering Thoroughness in Open-ended Reasoning). This framework combines deterministic scoring with a focus on factual accuracy and robust reasoning assessment. Using a vector database, EQUATOR pairs open-ended questions with human-evaluated answers, enabling more precise and scalable evaluations. In practice, EQUATOR significantly reduces reliance on human evaluators for scoring and improves scalability compared to Williams and Huckle’s (2004) [ 1 ] methods. Our results demonstrate that this framework significantly outperforms traditional multiple-choice evaluations while maintaining high accuracy standards. Additionally, we introduce an automated evaluation process leveraging smaller, locally hosted LLMs. We used LLaMA 3.2B, running on the Ollama binaries to streamline our assessments. This work establishes a new paradigm for evaluating LLM performance, emphasizing factual accuracy and reasoning ability, and provides a robust methodological foundation for future research. 1 Introduction Can a deterministic scoring framework, specifically the EQUATOR Evaluator, provide a more objective, reliable, and scalable method for evaluating the factual accuracy and reasoning capabilities of LLMs in response to open-ended questions compared to traditional evaluation methods? The paper challenges the reliance on traditional evaluation methods, such as multiple-choice formats, which are often susceptible to biases like fluency bias and lack a thorough assessment of reasoning abilities. The paper explores the hypothesis that a deterministic scoring framework, utilizing a vector database of human-evaluated answers, can mitigate these limitations and offer a more precise evaluation of LLM performance, particularly in complex reasoning tasks where factual accuracy is paramount. As Large Language Models (LLMs) become increasingly integrated into various applications, evaluating their responses accurately is critical. Traditional evaluation methods often fail to assess model performance, mainly because LLMs generate fluent and logically coherent responses that are stochastic and hallucinate, which means they are usually factually incorrect [ 2 ] . Human and automated evaluators also assign higher scores based on the fluency of reactions, the position of the response, and frequency rather than their factual accuracy. The challenges in evaluating LLM responses and biases have motivated researchers to develop more deterministic approaches. Some key factors that have driven this need include likelihood bias (overrate sentences with higher likelihoods while underrating those with lower likelihoods) [ 3 ] , position bias (prioritize content based on its position within the given context) [ 4 ] , lack of reference text [ 5 ] , limitations in semantic understanding, and the need for explain-ability [ 6 ] . LLMs can also exhibit various types of biases [ 7 ] , including demographic, cultural, linguistic, and temporal biases, which arise from factors such as training data, algorithmic choices, and model architecture. Williams and Huckle (2024) [ 1 ] highlighted that LLMs frequently fail at open-ended reasoning and logic questions, even when these questions are straightforward for humans. This issue underscores the necessity for a more rigorous evaluation system that prioritizes factual accuracy over persuasive ability. Our work addresses this gap by introducing a deterministic scoring framework emphasizing factual correctness in LLM outputs. [ 8 ] . We leverage a vector database constructed from questions and corresponding human-evaluated answers; we provide a reliable source of truth that focuses on the logical and factual content of responses. Our work differentiates itself from previous works, such as [ 9 , 10 ] by introducing a deterministic scoring framework that leverages a vector database. This framework addresses the limitations of traditional methods, which often rely on multiple-choice formats or human-evaluated answers, by providing a scalable and flexible source of truth. Unlike prior approaches, we incorporate smaller large language models (LLMs) as automated evaluators, combining computational efficiency with advanced language understanding to emphasize factual accuracy over linguistic fluency. While prior research has explored semantic similarity and embeddings for answer evaluation, these methods are often influenced by the fluency and persuasiveness of responses, potentially introducing bias. By contrast, our approach mitigates such biases by prioritizing objective correctness, ensuring reliable and scalable evaluations. The real-world consequences of hallucinations and reasoning failures in large language models (LLMs) are profound, particularly in high-stakes applications such as healthcare, finance, legal systems, and governance. LLM hallucinations—when models generate false or fabricated information—can lead to decisions based on incorrect or misleading data, with potentially catastrophic outcomes. In healthcare, for instance, an LLM providing inaccurate advice on medical treatments or diagnostics could jeopardize patient safety. Similarly, in the financial sector, reasoning failures in tasks such as risk assessment, portfolio management, or fraud detection can result in significant monetary losses, regulatory violations, or even systemic instability. The stakes are equally high in legal systems, where flawed reasoning could misguide legal professionals or decision-makers, undermining justice and fairness. A deterministic framework like EQUATOR is important because of its ability to systematically evaluate LLM reasoning under open-ended and context-dependent scenarios, where the risk of errors is amplified. By identifying the limitations of LLMs in reasoning and hallucination-prone areas, EQUATOR helps mitigate the risks associated with deploying these models in real-world, high-stakes environments. For instance, the framework’s structured evaluation ensures that models meet rigorous benchmarks before applying in sensitive domains. This not only improves the reliability of AI systems but also fosters trust among stakeholders who rely on them for critical decision-making. EQUATOR’s deterministic and reproducible approach to evaluation is therefore a vital step in bridging the gap between LLM capabilities and the demands of real-world applications, ensuring that their deployment aligns with safety, accuracy, and accountability standards. A key challenge would be designing practical and scalable techniques. For example, how can we develop differentially private model training approaches that are applicable for billions or trillions of parameters in generative AI models? How can we ensure privacy of end users when leveraging inputs from end users as part of retraining of LLMs (using, say, PATE-like approaches)? Considering the importance of high quality datasets for evaluating LLMs for truthfulness, bias, robustness, safety, and related dimensions, and the challenges with obtaining such datasets in highly sensitive domains such as healthcare, how do we develop practical and feasible approaches for differentially private synthetic data generation, potentially leveraging a combination of sensitive datasets (e.g., patient health records and clinical notes) and publicly available datasets along with the ability to generate data by querying powerful LLMs? The specific contributions of this work are: 1. We employ a deterministic scoring system using a vector database to enhance scalability and flexibility for evaluating diverse, open-ended responses. 2. We achieve efficient and consistent assessments by integrating smaller LLMs as automated evaluators. This approach reduces computational demands and mitigates biases

Read the complete text